Spark Streaming + Kafka

1. Summarize some concepts:

Install zookeeper 3.4.6

cp zoo_sample.cfg zoo.cfg
vim zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt /software/zookeeper-3.4.6/data
dataLogDir=/opt/software/zookeeper-3.4.6/logs
clientPort=2181
server.1=pc.apache:2888:3888

Add zookeeper to the environment variable;
In the mkdir data directory, insert vim myid 1, mkdir logs

zkServer.sh start







 

Install Kafka 0.10.2

vim server.propertiesbroker.id=0port=9092host.name=pc.apachelog.dirs=/opt/software/kafka_2.11-0.10 .2.0/datanum.network.threads=3num.io.threads=8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600zookeeper.connect=pc.apache:2181num.partitions =3num.recovery.threads.per.data.dir=1log.retention.hours=168log.segment.byt es=1073741824 log.retention.check.interval.ms=300000zookeeper.connection.timeout.ms=6000 Add kafka to the environment variable and start the service: nohup kafka-server-start.sh /opt/software/kafka_2.11-0.10. 2.0/config/server.properties> /opt/software/kfk.out & create partition kafka-topics.sh --create --zookeeper pc.apache:2181 --replication-factor 1 --partitions 3 --topic k1 settings Test producer: vim producer.propertiesbootstrap.servers=pc.apache:9092 Set test consumer: vim consumer.propertieszookeeper.connect=pc.apache:2181kafka-console-producer.sh --broker-list pc.apache: 9092 --topic k1kafka-console-consumer.sh --bootstrap-server pc.apache:9092 --topic k1 --from-beginning

cp zoo_sample.cfg zoo.cfg
vim zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir =/opt/software/zookeeper-3.4.6/data
dataLogDir=/opt/software/zookeeper-3.4.6/logs
clientPort=2181
server.1=pc.apache:2888:3888

Add zookeeper to the environment variable;
In the mkdir data directory, insert vim myid 1, mkd ir logs

zkServer.sh start







vim server.propertiesbroker.id=0port=9092host.name=pc.apachelog.dirs=/opt/software/kafka_2.11-0.10.2.0/datanum.network.threads= 3num.io.threads=8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600zookeeper.connect=pc.apache:2181num.partitions=3num.recovery.threads.per. data.dir=1log.retention.hours=168log.segment.bytes=1073741824log.retention.check.interval.ms=300000zookeeper.connection.timeout.ms=6000 Add kafka to the environment variable and start the service: nohup kafka-server- start.sh /opt/software/kafka_2.11-0.10.2.0/config/server.properties> /opt/software/kfk.out & create partition kafka-topics.sh --create --zookeeper pc.apache:2181- -replication-factor 1 --partitions 3 --topic k1 Set test producer: vim producer.propertiesbootstrap.servers=pc.apache:9092 Set test consumer: vim consumer.propertieszookeeper.connect=pc.apache:2181kafka- console-producer.sh --broker-list pc.apache:9092 --topic k1kafka -console-consumer.sh --bootstrap-server pc.apache:9092 --topic k1 --from-beginning

Leave a Comment

Your email address will not be published.