How to Install and Configure Apache Kafka on Windows 11 (Step-by-Step Guide)

INTRODUCTION

Real-time analytics, event-driven apps, and high-performance data pipelines are all made possible by Apache Kafka, a powerful distributed event streaming platform. You may install and set it up on Windows 11, even though it is mainly optimized for Linux systems. You will find a clear explanation of the installation and configuration procedure in this guide.

Why Pick Kafka?

Before we get started, let's examine the factors that contributed to Kafka's enormous appeal:

• Real-time Data Processing: Kafka excels at handling enormous volumes of data instantly.

• Scalability: It can easily increase to meet increasing demands for data.

• Fault Tolerance: Kafka's robust architecture guarantees the dependability of your data.

• Versatility: From log aggregation to real-time analytics, it finds use in a variety of contexts.

Requirements

Make sure you have the following before starting the Kafka installation:

• JDK, or the Java Development Kit: You must have a JDK installed because Kafka runs on Java. Using JDK 8 or a later version is advised.

• Windows 11 OS: Users of Windows 11 are the target audience for this article.

• Basic PowerShell/Command Prompt Knowledge: Using the command line is necessary for some procedures.

• Apache Zookeeper (an essential Kafka component)

Step 1: Installing Java (JDK)

• Get the JDK from the OpenJDK or Oracle official websites.

• To install the JDK, adhere to the on-screen directions.

• Set up the environment variable JAVA_HOME:

• Press Win + R, type "sysdm.cpl", then press Enter to open the System Properties.

• Next, choose Environment Variables under the Advanced tab.

• Click on New in the System Variables area and enter the following information:

  • Variable Name: JAVA_HOME
  • Variable Value: Path to the JDK installation directory (e.g., C:\Program Files\Java\jdk-11)

• Click OK to save your changes after that.

• Click "Edit..." after finding the "Path" variable in the System Variables section.

• Click "New" and add %JAVA_HOME%\bin.

• Click "OK" in each of the open windows to confirm.

• On the Command Prompt, run the following command to ensure everything is configured properly.

Java -version

Step 2: Install Apache Kafka.

1. Go to the download page for Apache Kafka.

2. Get the most recent binary version that is accessible.

3. Use a program like 7-Zip or WinRAR to extract the ZIP file you downloaded to the area of your choice (for example, C:\kafka).

Step 3: Set up Kafka.

1. Launch PowerShell or Command Prompt, then navigate to the config directory in your Kafka installation.

2. Modify the file server.properties:

3. In a text editor, open server properties

4. Provide a directory for Kafka to store its data in the log. dirs field (for example, log.dirs=C:/kafka/kafka-logs). If this directory doesn't already exist, be careful to create it.

5. Verify that the value zookeeper.connect=localhost:2181 is set.

6. Save your modifications.

Step 4: Installing and configuring Apache Zookeeper

Zookeeper is used by Kafka to manage distributed setups. To get it operating, take these actions:

1. Navigate to C:\kafka, the extracted Kafka directory.

2. Locate the zookeeper.properties file in the config folder.

3. In Notepad, open the zookeeper.properties file and make the following changes:

dataDir=C:\kafka\data\zookeeper

4. Create manually the C:\kafka\data\zookeeper directory.

5. Open the Kafka directory on the Command Prompt:

cd C:\kafka 

6. Launch Zookeeper by running:

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Step 5: Start Kafka and ZooKeeper

Start ZooKeeper

• Launch ZooKeeper by opening a new PowerShell or Command Prompt window.

• Go to the location where Kafka is installed.

• Run the following command:

bin\windows\zookeeper-server-start.bat config\zookeeper.properties 

• Don't close this window.

Let's launch Apache Kafka now:

• Launch a second window of the Command Prompt.

• Navigate to C:\kafka, the Kafka directory.

• Run Kafka to get it started:

.\bin\windows\kafka-server-start.bat .\config\server.properties

• Kafka ought to be operational at this point.

Step 6: Create a Kafka Topic

• Launch a fresh instance of the Command Prompt.

• Create a topic and name it test-topic or any name of your choice:

.\bin\windows\kafka-topics.bat --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

• Verify that the subject was generated successfully:

.\bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092

Step 7: Message Sending and Receiving

To Communicate:

1. Launch a fresh PowerShell or Command Prompt window.

2. Go to the directory where Kafka is installed:

 .\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic

3. To send your message, type it and press Enter.

To Read a Message:

1. Launch a new PowerShell or Command Prompt window.

2. Go to the directory where Kafka is installed:

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --from-beginning 

3. You should see the messages that were sent earlier

CONCLUSION:

Apache Kafka has been successfully installed and configured on Windows 11. You may now begin investigating its potent features for creating applications that stream data in real time.