Building a Real-Time Event-Driven App with Node.js and Kafka on Heroku

|

Real-time event-driven applications have become a crucial part of modern web architecture. These applications react to events and process data in real-time, providing timely and dynamic user experiences. Node.js, known for its event-driven architecture, and Apache Kafka, a distributed streaming platform, are powerful tools for building such applications. Hosting this setup on Heroku, a cloud platform as a service (PaaS), simplifies deployment and scaling.

This article will guide you through building a real-time event-driven app using Node.js and Kafka, hosted on Heroku. We will cover the setup, coding examples, and best practices.

Setting Up the Environment

Before diving into the code, let’s set up our development environment.

  1. Install Node.js: Ensure you have Node.js installed. Download and install it from Node.js official website.
  2. Install Apache Kafka: Download and set up Kafka on your local machine or use a managed Kafka service like Confluent Cloud or Amazon MSK.
  3. Heroku Account: Create a Heroku account if you don’t have one. Install the Heroku CLI from Heroku CLI documentation.
  4. Create a Heroku App: Log in to Heroku and create a new app:

    bash

    heroku login
    heroku create my-event-driven-app

Initializing the Node.js Project

Initialize a new Node.js project and install the necessary dependencies.

  1. Initialize Project:

    bash

    mkdir event-driven-app
    cd event-driven-app
    npm init -y
  2. Install Dependencies:

    bash

    npm install express kafka-node
  • express: A minimal and flexible Node.js web application framework.
  • kafka-node: A Kafka client for Node.js.

Building the Producer

A producer sends messages to a Kafka topic. Let’s create a simple producer using Kafka-node.

  1. Create a Producer Script: Create a file named producer.js.

    javascript

    const kafka = require('kafka-node');
    const Producer = kafka.Producer;
    const client = new kafka.KafkaClient({kafkaHost: 'localhost:9092'});
    const producer = new Producer(client);    const payloads = [
    { topic: ‘test-topic’, messages: ‘Hello Kafka’, partition: 0 }
    ];producer.on(‘ready’, () => {
    producer.send(payloads, (err, data) => {
    if (err) {
    console.error(‘Error sending message:’, err);
    } else {
    console.log(‘Message sent successfully:’, data);
    }
    });
    });producer.on(‘error’, (err) => {
    console.error(‘Error in Kafka producer:’, err);
    });
  2. Run the Producer:

    bash

    node producer.js

This script initializes a Kafka producer that sends a “Hello Kafka” message to the test-topic.

Building the Consumer

A consumer listens for messages from a Kafka topic. Let’s create a simple consumer.

  1. Create a Consumer Script: Create a file named consumer.js.

    javascript

    const kafka = require('kafka-node');
    const Consumer = kafka.Consumer;
    const client = new kafka.KafkaClient({kafkaHost: 'localhost:9092'});
    const consumer = new Consumer(
    client,
    [{ topic: 'test-topic', partition: 0 }],
    { autoCommit: true }
    );    consumer.on(‘message’, (message) => {
    console.log(‘Received message:’, message);
    });consumer.on(‘error’, (err) => {
    console.error(‘Error in Kafka consumer:’, err);
    });
  2. Run the Consumer:

    bash

    node consumer.js

This script initializes a Kafka consumer that listens to the test-topic and logs any received messages.

Integrating with Express

Next, we’ll integrate Kafka producer and consumer with an Express application to create a simple web interface for sending and receiving messages.

  1. Create an Express App: Create a file named app.js.

    javascript

    const express = require('express');
    const kafka = require('kafka-node');
    const app = express();
    const port = process.env.PORT || 3000;    // Kafka setup
    const Producer = kafka.Producer;
    const Consumer = kafka.Consumer;
    const client = new kafka.KafkaClient({kafkaHost: ‘localhost:9092’});
    const producer = new Producer(client);
    const consumer = new Consumer(
    client,
    [{ topic: ‘test-topic’, partition: 0 }],
    { autoCommit: true }
    );app.use(express.json());// Endpoint to send messages
    app.post(‘/send’, (req, res) => {
    const payloads = [
    { topic: ‘test-topic’, messages: req.body.message, partition: 0 }
    ];
    producer.send(payloads, (err, data) => {
    if (err) {
    return res.status(500).send(‘Error sending message’);
    }
    res.send(‘Message sent successfully’);
    });
    });// WebSocket to receive messages
    const http = require(‘http’).Server(app);
    const io = require(‘socket.io’)(http);consumer.on(‘message’, (message) => {
    io.emit(‘message’, message);
    });

    http.listen(port, () => {
    console.log(`Server running at http://localhost:${port}/`);
    });

  2. Run the Express App:

    bash

    node app.js

This script sets up an Express server with a /send endpoint for sending messages and a WebSocket to broadcast received messages to connected clients.

Deploying to Heroku

Finally, we’ll deploy our application to Heroku.

  1. Create a Procfile: Create a Procfile in the root directory with the following content:

    makefile

    web: node app.js
  2. Create a .gitignore: Create a .gitignore file to ignore node_modules and other unnecessary files:
    node_modules
  3. Initialize Git and Deploy:

    bash

    git init
    git add .
    git commit -m "Initial commit"
    heroku git:remote -a my-event-driven-app
    git push heroku master

Heroku will build and deploy your application.

Conclusion

Building a real-time event-driven application with Node.js and Kafka offers a powerful solution for applications requiring high throughput and low latency. Node.js provides a non-blocking, event-driven architecture ideal for handling real-time data streams, while Kafka ensures reliable and scalable message brokering.

In this guide, we walked through setting up a Node.js project, creating Kafka producers and consumers, integrating with an Express server, and deploying the application on Heroku. This setup allows you to leverage the scalability of Kafka and the simplicity of Heroku for hosting your applications.

Key takeaways:

  • Node.js and Kafka: A robust combination for real-time applications.
  • Express Integration: Simplifies building a web interface for interacting with Kafka.
  • Heroku Deployment: Streamlines deployment and scaling.

By following these steps, you can build and deploy scalable, real-time event-driven applications, ensuring a responsive and dynamic user experience. This foundation can be extended with more complex features, such as advanced message processing, error handling, and security measures, to build production-ready applications.