Kafka
Apache Kafka standalone installation and configuration.
Zookeeper and Kafka requires Java 8 or Java 11. If Java is not already installed on your server, please follow steps on Prerequisites page.

Zookeeper

Zookeeper installation

Download the latest stable version (follow this link to latest stable : http://apache.mediamirrors.org/zookeeper/stable/):
1
cd /opt
2
wget http://apache.mirrors.hoobly.com/zookeeper/zookeeper-3.6.1/apache-zookeeper-3.6.1-bin.tar.gz
3
tar zxf apache-zookeeper-3.6.1-bin.tar.gz
4
rm apache-zookeeper-3.6.1-bin.tar.gz
Copied!
Create a symbolic link 'zookeeper' (Simpliest path and easier for upgrade):
1
ln -s /opt/apache-zookeeper-3.6.1-bin/ /opt/zookeeper
Copied!
Create zk service account for Zookeeper and Kafka:
1
groupadd -r zk
2
useradd -r -g zk -M -s /sbin/nologin -c "Kafka and Zookeeper service user" zk
Copied!
Create folders to store Zookeeper data and logs:
1
mkdir /opt/zookeeper/data
2
mkdir /opt/zookeeper/logs
3
chown -R zk:zk /opt/zookeeper/logs/
4
chown -R zk:zk /opt/zookeeper/data/
Copied!
Create Zookeeper systemd script:
1
sudo vi/etc/systemd/system/zookeeper.service
2
3
# Add the following lines
4
[Unit]
5
Description=Zookeeper Daemon
6
Documentation=http://zookeeper.apache.org
7
Requires=network.target
8
After=network.target
9
10
[Service]
11
Type=forking
12
WorkingDirectory=/opt/zookeeper
13
User=zk
14
Group=zk
15
ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/server.configuration
16
ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/server.configuration
17
ExecReload=/opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/conf/server.configuration
18
TimeoutSec=30
19
Restart=on-failure
20
21
[Install]
22
WantedBy=default.target
23
Copied!
Enable and start Zookeeper:
1
# Enable and start after configuration
2
systemctl enable zookeeper
3
systemctl start zookeeper
Copied!

Zookeeper configuration

Create a configuration file and add the following lines:
1
sudo vi /opt/zookeeper/conf/server.configuration
2
3
tickTime=2000
4
dataDir=/opt/zookeeper/data
5
dataLogDir/opt/zookeeper/logs
6
clientPort=2181
7
clientPortAddress=localhost # Server IP can also be specified
Copied!

Kafka

Kafka installation

Download the latest stable version:
1
cd /opt
2
wget http://apache.mirrors.hoobly.com/kafka/2.5.0/kafka_2.12-2.5.0.tgz
3
tar zxf kafka_2.12-2.5.0.tgz
4
rm tar zxf kafka_2.12-2.5.0.tgz
Copied!
Create a symbolic link 'kafka' (Simpliest path and easier for upgrade) :
1
ln -s /opt/kafka_2.12-2.5.0/ /opt/kafka
Copied!
Create Kafka systemd script :
1
sudo vi /etc/systemd/system/kafka.service
2
3
# Add the following lines and replace JAVA_HOME
4
[Unit]
5
Description=Apache Kafka server (broker)
6
Documentation=http://kafka.apache.org/documentation.html
7
Requires=network.target remote-fs.target
8
After=network.target remote-fs.target zookeeper.service
9
10
[Service]
11
Type=simple
12
User=zk
13
Group=zk
14
Environment=JAVA_HOME=$JAVA_HOME
15
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
16
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
17
18
[Install]
19
WantedBy=multi-user.target
20
Copied!
Enable and start Kafka :
1
# Enable and start after configuration
2
systemctl enable kafka
3
systemctl start kafka
Copied!

Kafka configuration

Kafka listeners can be configured to send and receive data in plaintext, encrypted or both in same time.
Nybble support SSL configuration. SSL can be configured to be used by Beats shippers producers and by Nybble consumer (Nybble is a Kafka client in this case).

Plaintext configuration

Edit Kafka configuration and add/edit following lines :
1
sudo vi /opt/kafka/config/server.properties
2
3
#Each brokers have a unique ID
4
broker.id=0
5
#Listen on localhost on port 9092 in plaintext
6
listeners=PLAINTEXT://$your_server_ip:9092
7
#Logs are stored during 7 days in Kafka Topics
8
log.retention.hours=168
9
#Zookeeper server address and port (Standalone Zookeeper installation)
10
zookeeper.connect=localhost:2181 # Replace localhost by your server IP if you modified it in previous configuration
11
zookeeper.connection.timeout.ms=6000
Copied!

SSL configuration

This part contains a summary of steps from Vertica TLS/SSL tutorial.
This example is based on self-signed certificate. You may already have a PKI infrastructure, in this case your client certificates will be signed by your CA.
SSL Configuration consists in creation of Root CA, then a Client certificate signed by the Root CA for Beats shippers and finally truststore & keystore containing the Root CA which will be used in Kafka configuration.
Root CA
Generate a private key named nybble-root.key:
1
sudo openssl genrsa -out nybble-root.key
Copied!
Generate a self-signed root CA named nybble-ca.crt:
1
sudo openssl req -new -x509 -key nybble-root.key -out nybble-ca.crt
Copied!
set "Common Name" field to a wildcard value for your domain: *.yourdomain.tld
Change permissions on key and certificate to prevent modification:
1
sudo chmod 600 nybble-root.key
2
sudo chmod 644 nybble-ca.crt
Copied!
Client certificate (For Beats shippers)
Generate a private key for the client certificate:
1
sudo openssl genrsa -out nybble-siem-client.key
Copied!
Generate a certificate request for the client certificate:
1
sudo openssl req -new -key nybble-siem-client.key -out nybble-siem-client-reqout.txt
Copied!
set "Common Name" field to a wildcard value for your domain: *.yourdomain.tld
Sign the client certificate request with the root CA:
1
sudo openssl x509 -req -in nybble-siem-client-reqout.txt -days 3650 -sha1 -CAcreateserial -CA nybble-ca.crt -CAkey nybble-root.key -out nybble-siem-client.crt
2
sudo rm nybble-siem-client-reqout.txt
Copied!
Truststore & Keystore
Create a truststore which will be used by all your Kafka brokers:
1
sudo keytool -keystore nybble.truststore.jks -alias CARoot -import -file nybble-ca.crt
Copied!
Truststore password will be used in Kafka configuration.
Create a keystore file for each Kafka brokers:
1
sudo keytool -keystore siem-broker01.keystore.jks -storetype JKS -alias localhost -validity 3650 -genkey -keyalg RSA -ext SAN=DNS:siem-broker01.nybble.local
Copied!
Use your own broker FQDN for the Subject Alternative Name (SAN).
Use your own broker FQDN for response to "What is your first and last name?" prompt.
Keystore password will be used in Kafka configuration.
Key password will be used in Kafka configuration.
Export the Kafka broker's certificate to signed it with the root CA:
1
sudo keytool -keystore siem-broker01.keystore.jks -alias localhost -certreq -file siem-broker01.unsigned.crt
Copied!
Sign the Kafka broker's certificate with the Root CA:
1
sudo openssl x509 -req -CA nybble-ca.crt -CAkey nybble-root.key -in siem-borker01.unsigned.crt -out siem-borker01.signed.crt -days 3650 -CAcreateserial
Copied!
Import the Root CA certificate in the Kafka broker's keystore:
1
sudo keytool -keystore siem-broker01.keystore.jks -alias CARoot -import -file nybble-ca.crt
Copied!
Import the signed Kafka broker's certificate in the Kafka broker's keystore:
1
sudo keytool -keystore siem-broker01.keystore.jks -alias localhost -import -file siem-borker01.signed.crt
Copied!
Copy the truststore and the Kafka broker keystore in the Kafka configuration folder and set zk service account as owner:
1
sudo cp nybble.truststore.jks /opt/kafka/config/
2
sudo cp siem-broker01.keystore.jks /opt/kafka/config/
3
4
sudo chown zk:zk /opt/kafka/config/nybble.truststore.jks
5
sudo chown zk:zk /opt/kafka/config/siem-broker01.keystore.jks
Copied!
Edit Kafka configuration and add/edit following lines :
1
sudo vi /opt/kafka/config/server.properties
2
3
4
#Each brokers have a unique ID
5
broker.id=0
6
#Listen on localhost on port 9092 in plaintext
7
listeners=PLAINTEXT://$server_name:9092,SSL://$server_name:9093
8
# Under SSL/TLS Configuration part
9
ssl.keystore.location=/opt/kafka/config/siem-broker01.keystore.jks
10
ssl.keystore.password=$Your_Keystore_Password
11
ssl.key.password=$Your_Key_Password
12
ssl.truststore.location=/opt/kafka/config/nybble.truststore.jks
13
ssl.truststore.password=$Your_Truststore_Password
14
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
15
ssl.client.auth=required
16
17
#Logs are stored during 7 days in Kafka Topics
18
log.retention.hours=168
19
#Zookeeper server address and port (Standalone Zookeeper installation)
20
zookeeper.connect=localhost:2181 # Replace localhost by your server IP if you modified it in previous configuration
21
zookeeper.connection.timeout.ms=6000
Copied!
Last modified 1yr ago