My big data Tour - Kafka environment construction


Environmental preparation:

Three CentOS virtual machines with JDK and Zookeeper installed


Environment building


1) download the compressed package

2) decompression:

[feng@hadoop129 software]$ ls
[feng@hadoop129 software]$ tar -zxf kafka_2.11-2.2.0.tgz
[feng@hadoop129 software]$ ls
kafka_2.11-2.2.0  kafka_2.11-2.2.0.tgz
[feng@hadoop129 software]$

3) move to the specified directory:

[feng@hadoop129 software]$ mv kafka_2.11-2.2.0 /opt/module/

4) create the logs directory under / opt / module / Kafka ﹤ 2.11-2.2.0:

[feng@hadoop129 kafka_2.11-2.2.0]$ mkdir logs
[feng@hadoop129 kafka_2.11-2.2.0]$ ls
bin  config  libs  LICENSE  logs  NOTICE  site-docs

5) modify the configuration file:

[feng@hadoop129 config]$ vim /opt/module/kafka_2.11-2.2.0/config/

The modification and addition of the file are as follows:


#Newly added
#Delete topic function enable

#Modify the path of kafka operation log

#Modify the configuration connection Zookeeper cluster address

6) configure environment variables:

[feng@hadoop129 kafka_2.11-2.2.0]$ sudo vim /etc/profile

To add a new configuration, execute the command source /etc/profile:

export KAFAKA_HOME=/opt/module/kafka_2.11-2.2.0
export PATH=$PATH:${KAFAKA_HOME}/bin

7) synchronize the entire folder kafaka ﹣ 2.11-2.2.0 to the other two machines: Hadoop 130 / Hadoop 131 and modify the in the configuration file

Hadoop 130 machine: = 130

Hadoop 131 machine: = 131

8) start zookeeper first, then kafka

[feng@hadoop129 kafka_2.11-2.2.0]$ pwd
[feng@hadoop129 kafka_2.11-2.2.0]$ bin/ config/ &

[feng@hadoop130 kafka_2.11-2.2.0]$ pwd
[feng@hadoop130 kafka_2.11-2.2.0]$ bin/ config/ &

[feng@hadoop131 kafka_2.11-2.2.0]$ pwd
[feng@hadoop131 kafka_2.11-2.2.0]$ bin/ config/ &

9) create and view topic s

[feng@hadoop129 kafka_2.11-2.2.0]$ bin/ --create --bootstrap-server hadoop129:9092 --replication-factor 1 --partitions 1 --topic kafka-first
[feng@hadoop129 kafka_2.11-2.2.0]$ bin/ --list --bootstrap-server hadoop129:9092

10) sending information

[feng@hadoop129 kafka_2.11-2.2.0]$ bin/ --broker-list hadoop129:9092 --topic kafka-first
>Hello,Kafka ;-)
>Learning makes you happy

11) view messages

[feng@hadoop129 kafka_2.11-2.2.0]$ bin/ --bootstrap-server hadoop129:9092 --topic kafka-first --from-beginning
Hello,Kafka ;-)
Learning makes you happy

12) view the details of a Topic

[feng@hadoop129 kafka_2.11-2.2.0]$ bin/ --describe --bootstrap-server hadoop129:9092 --topic kafka-first
Topic:kafka-first       PartitionCount:1        ReplicationFactor:1     Configs:segment.bytes=1073741824
        Topic: kafka-first      Partition: 0    Leader: 129     Replicas: 129   Isr: 129
[feng@hadoop129 kafka_2.11-2.2.0]$

The article refers to the Quick start kafka video course of monk school.



































Tags: kafka Zookeeper Hadoop vim

Posted on Tue, 05 Nov 2019 10:31:51 -0800 by Xorandnotor