Apache Kafka

Kafka is message broker which can be connected to any real-time framework available on the market. In this book, we will use Kafka often for all types of examples. We will use Kafka as a data source which keeps data from files in queues for further processing. Download Kafka from https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.1/kafka_2.11-0.10.1.1.tgz to your local machine. Once the kafka_2.11-0.10.1.1.tgz file is downloaded, extract the files using the following command:

    cp kafka_2.11-0.10.1.1.tgz /home/ubuntu/demo/kafka
    cd /home/ubuntu/demo/kafka
    tar -xvf kafka_2.11-0.10.1.1.tgz

The following files and folders are extracted as seen in the following screenshot:

To start Kafka use the following commands:

    /bin/zookeeper-server-start.sh config/zookeeper.properties
    /bin/kafka-server-start.sh config/server.properties

Kafka will start on your local machine. Topics will be created later on as per the need. Let's move on to the NiFi setup and example.