How to Install Apache Kafka with CMAK on CentOS

Apache Kafka is a free and open-source streaming platform used for streaming analytics, data integration, and mission-critical applications. Compared to ActiveMQ and RabbitMQ, Kafka has better throughput, built-in partitioning, replication, and fault-tolerance. Kafka is a distributed message agent that can deal with huge volumes of real-time information effectively.

CMAK also known as a "Cluster Manager" is used for managing Kafka cluster developed by Yahoo. Using CMAK, you can manage multiple clusters and inspect cluster state including, topics, consumers, offsets, brokers, replica distribution, partition distribution and many more.

In this tutorial, we will show how to install Apache Kafka streaming platform with CMAK on CentOS 8 server.

Prerequisites

  • A server running CentOS 8.
  • A root password is configured the server.

Getting Started

First, updating your system packages to the latest version is recommended. You can update all with the following command:

dnf update -y

Once all the packages are updated, install other required dependencies with the following command:

dnf install git unzip -y

Once you are finished, you can proceed to the next step.

Install Java

Kafka is based on Java, so Java must be installed in your server. If not installed, you can install if with the following command:

dnf install java-11-openjdk-devel -y

Once the Java is installed, you can verify the installed version of Java using the following command:

java --version

You should get the following output:

openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-CentOS-0centos8)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-CentOS-0centos8, mixed mode, sharing)

Once you are finished, you can proceed to the next step.

Download Kafka

Before downloading Kafka, create a directory to store Kafka. You can create it with the following command:

mkdir /usr/local/kafka-server

Next, change the directory to kafka-server and download the latest version of Kafka using the following command:

cd /usr/local/kafka-server
wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz

Once the download is completed, extract the downloaded file with the following command:

tar -xvzf kafka_2.13-2.6.0.tgz --strip 1

Once you are finished, you can proceed to the next step.

Create a Systemd File for Kafka and Zookeeper

Next, you will need to create a systemd service file to manage the Zookeeper and Kafka service.

First, create a Zookeeper service file using the following command:

nano /etc/systemd/system/zookeeper.service

Add the following lines:

[Unit]
Description=Apache Zookeeper Server
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka-server/bin/zookeeper-server-start.sh /usr/local/kafka-server/config/zookeeper.properties
ExecStop=/usr/local/kafka-server/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and close the file when you are finished.

Next, create a Kafka service file with the following command:

nano /etc/systemd/system/kafka.service

Add the following lines:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties
ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and close the file then reload the systemd daemon with the following command:

systemctl daemon-reload

Next, start both services and enable them to start at boot with the following command:

systemctl enable --now zookeeper
systemctl enable --now kafka
systemctl start zookeeper
systemctl start kafka

Next, verify the status of both services using the following command:

systemctl status zookeeper kafka

You should see the following output:

? zookeeper.service - Apache Zookeeper Server
   Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-08-29 08:48:42 EDT; 30s ago
 Main PID: 26722 (java)
    Tasks: 31 (limit: 12523)
   Memory: 68.8M
   CGroup: /system.slice/zookeeper.service
           ??26722 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCIn>

Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,326] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,327] INFO Created server with tickTime 3000 minSessionTimeout 6>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,344] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactor>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,348] INFO Configuring NIO connection handler with 10s sessionle>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,355] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zook>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,391] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zooke>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,397] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapsho>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,401] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapsho>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,431] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.a>
Aug 29 08:48:53 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:53,030] INFO Creating new log file: log.1 (org.apache.zookeeper.se>

? kafka.service - Apache Kafka Server
   Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-08-29 08:48:50 EDT; 22s ago
     Docs: http://kafka.apache.org/documentation.html
 Main PID: 27100 (java)
    Tasks: 66 (limit: 12523)
   Memory: 306.4M
   CGroup: /system.slice/kafka.service
           ??27100 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancy>

Aug 29 08:48:56 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:56,733] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.c>
Aug 29 08:48:56 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:56,747] INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.De>
Aug 29 08:48:56 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:56,846] INFO [/config/changes-event-process-thread]: Starting (kafka.c>
Aug 29 08:48:56 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:56,877] INFO [SocketServer brokerId=0] Starting socket server acceptor>
Aug 29 08:48:57 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:57,236] INFO [SocketServer brokerId=0] Started data-plane acceptor and>
Aug 29 08:48:57 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:57,237] INFO [SocketServer brokerId=0] Started socket server acceptors>
Aug 29 08:48:57 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:57,246] INFO Kafka version: 2.6.0 (org.apache.kafka.common.utils.AppIn>
Aug 29 08:48:57 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:57,246] INFO Kafka commitId: 62abe01bee039651 (org.apache.kafka.common>

Install and Configure CMAK

Next, you will need to install CMAK for managing Apache Kafka cluster. You can download it from the Git repository with the following command:

cd /root
git clone https://github.com/yahoo/CMAK.git

Once downloaded, you will need to edit the application.conf file and define the zookeeper hosts.

nano ~/CMAK/conf/application.conf

Change the following lines:

cmak.zkhosts="localhost:2181"

Save and close the file when you are finished. Then, create a zip file to deploy the application.

cd ~/CMAK/
./sbt clean dist

You should see the following output:

model contains 640 documentable templates
[info] Main Scala API documentation successful.
[info] LESS compiling on 1 source(s)
[success] All package validations passed
[info] Your package is ready in /root/CMAK/target/universal/cmak-3.0.0.5.zip
[success] Total time: 150 s (02:30), completed 29-Aug-2020, 8:55:40 AM

Next, unzip the zip file which is created in the previous step with the following command:

cd ~/CMAK/target/universal
unzip cmak-3.0.0.5.zip

Once you are finished, you can proceed to the next step.

Start CMAK Services

Now, change the directory to ~/CMAK/target/universal/cmak-3.0.0.5 and start the CMAK service using the following command:

cd ~/CMAK/target/universal/cmak-3.0.0.5
bin/cmak

At this point, CMAK is started and listening on port 9000.

Configure SELinux and Firewall

By default, SELinux is enable in CentOS 8. So you will need to configure SELinux for CMAK. You can configure it with the following command:

chcon -t httpd_sys_rw_content_t ~/CMAK/target/universal/cmak-3.0.0.5 -R
setsebool -P httpd_can_network_connect 1

Next, you will need to allow port 9000 through firewalld. You can do it with the following command:

firewall-cmd --permanent --zone public --add-port 9000/tcp
firewall-cmd --reload

Once you are finished, you can proceed to the next step.

Access CMAK Web UI

Now, open your web browser and access the CMAK using the URL http://your-server-ip:9000. You should see the following page:

CMAK Clusters

In the above screen, you should see that there is no cluster available. So you will need to create a new cluster first.

Click on the Cluster => Add CLuster. You should see the following screen:

Add Cluster

Cluster settings

Provide your cluster name, zookeeper host, other required information and click on the Save button. You should see the following screen:

Successfully addded a cluster

Once you are done, you can proceed to the next step.

Add Your First Topic on Kafka

Next, you will need to create a topic on Kafka to test the functionality. To do so, open another terminal, change the directoru to kafka-server and create a new topic by specifying a topic name and replication factor by running the following command:

cd /usr/local/kafka-server
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic MyNewTopic

Now, go to the CMAK dashboard and click on the Topic. You should see the Topic which you have created earlier in the following screen:

Add Apache Kafka topic

Click on the Topic number. You should see your newly created topic in the following screen:

Operations and topics

Now, click on your Topic name. You should see all the information of your topic in the following screen:

Kafka Topic

Conclusion

Congratulations! you have successfully installed Apache Kafka with CMAK on CentOS 8 server. You can now start exploring CMAK for more functionality. Feel free to ask me if you have any questions.

Share this page:

0 Comment(s)