基于docker搭建zookeeper集群、kafka集群,Docker中搭建zookeeper集群
zookeeper集群搭建
https://www.cnblogs.com/znicy/p/7717426.html #Docker中搭建zookeeper集群，昵称：zni.feng
https://www.cnblogs.com/luotianshuai/p/5206662.html #真机搭建参数参考
SRE实战 互联网时代守护先锋，助力企业售后服务体系运筹帷幄！一键直达领取阿里云限量特价优惠。-------------------------------------------------------------------------------------------
docker inspect docker.io/zookeeper
"Cmd": [ "zkServer.sh", "start-foreground" ], "Volumes": { "/data": {}, "/datalog": {} }, "WorkingDir": "/zookeeper-3.4.10", "Entrypoint": [ "/docker-entrypoint.sh" ], #由镜像信息看出，容器启动会运行这个脚本
此脚本内容：https://www.cnblogs.com/znicy/p/7717426.html
由以上内容，创建启动zookeeper集群容器脚本：
#!/bin/bash #Get zookeeper image zkimage=`docker images | grep zookeeper | awk {'print $1'}` if [ -n "$zkimage" ] then echo 'The zookeeper image is already existed.' else echo 'Pull the latest zookeeper image.' docker pull zookeeper fi #Create network for zookeeper containers zknet=`docker network ls | grep yapi_net | awk {'print $2'}` if [ -n "$zknet" ] then echo 'The zknetwork is already existed.' else echo 'Create zknetwork.' docker network create --subnet 172.30.0.0/16 yapi_net fi #Start zookeeper cluster echo 'Start 3 zookeeper servers.' rm -rf /opt/zookeeper_1/data /opt/zookeeper_1/datalog /var/log/zookeeper_1/log rm -rf /opt/zookeeper_2/data /opt/zookeeper_2/datalog /var/log/zookeeper_2/log rm -rf /opt/zookeeper_3/data /opt/zookeeper_3/datalog /var/log/zookeeper_3/log mkdir -p /opt/zookeeper_1/data /opt/zookeeper_1/datalog /var/log/zookeeper_1/log mkdir -p /opt/zookeeper_2/data /opt/zookeeper_2/datalog /var/log/zookeeper_2/log mkdir -p /opt/zookeeper_3/data /opt/zookeeper_3/datalog /var/log/zookeeper_3/log ZOO_SERVERS="server.1=zookeeper_1:2888:3888 server.2=zookeeper_2:2888:3888 server.3=zookeeper_3:2888:3888" #docker run --network yapi_net --ip 172.30.0.10 -d --name yapi_mongodb --restart always -v /opt/yapi_mongo/config:/data/configdb -v /opt/yapi_mongo/mongodb:/data/db docker.io/mongo docker run --network yapi_net --ip 172.30.0.31 -d --restart always -v /opt/zookeeper_1/data:/data -v /opt/zookeeper_1/datalog:/datalog -v /var/log/zookeeper_1/log:/logs -e ZOO_SERVERS="$ZOO_SERVERS" -e ZOO_MY_ID=1 --name zookeeper_1 -p 2182:2181 docker.io/zookeeper docker run --network yapi_net --ip 172.30.0.32 -d --restart always -v /opt/zookeeper_2/data:/data -v /opt/zookeeper_2/datalog:/datalog -v /var/log/zookeeper_2/log:/logs -e ZOO_SERVERS="$ZOO_SERVERS" -e ZOO_MY_ID=2 --name zookeeper_2 -p 2183:2181 docker.io/zookeeper docker run --network yapi_net --ip 172.30.0.33 -d --restart always -v /opt/zookeeper_3/data:/data -v /opt/zookeeper_3/datalog:/datalog -v /var/log/zookeeper_3/log:/logs -e ZOO_SERVERS="$ZOO_SERVERS" -e ZOO_MY_ID=3 --name zookeeper_3 -p 2184:2181 docker.io/zookeeper
执行代码即可
kafka集群
配置事项：
分区多的优点，理论上整个集群所能达到的吞吐量就越大。
分区不是越多越好，一、客户端/服务器端需要使用的内存就越多，二、文件句柄的开销（ulimit -n）三、降低高可用性，因为要进行分区副本leader的选举
分区数 = Tt / max(Tp, Tc),（创建一个只有1个分区的topic，然后测试这个topic的producer吞吐量和consumer吞吐量。假设它们的值分别是Tp和Tc，单位可以是MB/s。然后假设总的目标吞吐量是Tt）
consumer线程数最好=分区数
listeners和advertised.listeners的区别：https://www.colabug.com/6020170.html http://www.devtalking.com/articles/kafka-practice-16/
由开机启动脚本，编写代码
#!/bin/bash #Get kafka image kfkimage=`docker images | grep 'docker.io/wurstmeister/kafka' | awk {'print $1'}` if [ -n "$kfkimage" ] then echo 'The docker.io/wurstmeister/kafka is already existed.' else echo 'Pull the image.' docker pull docker.io/wurstmeister/kafka fi #Create network for zookeeper containers kfknet=`docker network ls | grep yapi_net | awk {'print $2'}` if [ -n "$kfknet" ] then echo 'The kfknetwork is already existed.' else echo 'Create kfknetwork.' docker network create --subnet 172.30.0.0/16 yapi_net fi #Start 3 zookeeper cluster echo 'Start 3 kafka servers.' rm -rf /opt/kafka_1/logdata rm -rf /opt/kafka_2/logdata rm -rf /opt/kafka_3/logdata mkdir -p /opt/kafka_1/logdata mkdir -p /opt/kafka_2/logdata mkdir -p /opt/kafka_3/logdata #kafka ip kfk_1_ip='172.30.0.41' kfk_2_ip='172.30.0.42' kfk_3_ip='172.30.0.43' zk_jiqun_ip='172.30.0.33:2181' #docker run --restart always -d --name kafka_1 --network yapi_net --ip ${kfk_1_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='41' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://192.168.181.130:9092' -v /opt/kafka_1/logdata:/kafka -p 9092:9092 docker.io/wurstmeister/kafka #docker run --restart always -d --name kafka_2 --network yapi_net --ip ${kfk_2_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9093' -e KAFKA_BROKER_ID='42' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://192.168.181.130:9093' -v /opt/kafka_2/logdata:/kafka -p 9093:9092 docker.io/wurstmeister/kafka #docker run --restart always -d --name kafka_3 --network yapi_net --ip ${kfk_3_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9094' -e KAFKA_BROKER_ID='43' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://192.168.181.130:9094' -v /opt/kafka_3/logdata:/kafka -p 9094:9092 docker.io/wurstmeister/kafka #docker run --restart always -d --name kafka_1 --network yapi_net --ip ${kfk_1_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='41' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://172.30.0.41:9092' -v /opt/kafka_1/logdata:/kafka -p 9092:9092 05cef8845b3d #docker run --restart always -d --name kafka_2 --network yapi_net --ip ${kfk_2_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9093' -e KAFKA_BROKER_ID='42' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://172.30.0.42:9092' -v /opt/kafka_2/logdata:/kafka -p 9093:9092 05cef8845b3d #docker run --restart always -d --name kafka_3 --network yapi_net --ip ${kfk_3_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9094' -e KAFKA_BROKER_ID='43' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://172.30.0.43:9092' -v /opt/kafka_3/logdata:/kafka -p 9094:9092 05cef8845b3d docker run --restart always -d --name kafka_1 --network yapi_net --ip ${kfk_1_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='41' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://'${kfk_1_ip}':9092' -v /opt/kafka_1/logdata:/kafka -p 9092:9092 docker.io/wurstmeister/kafka docker run --restart always -d --name kafka_2 --network yapi_net --ip ${kfk_2_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='42' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://'${kfk_2_ip}':9092' -v /opt/kafka_2/logdata:/kafka -p 9093:9092 docker.io/wurstmeister/kafka docker run --restart always -d --name kafka_3 --network yapi_net --ip ${kfk_3_ip} -e KAFKA_ZOOKEEPER_CONNECT=${zk_jiqun_ip} -e KAFKA_LISTENERS='PLAINTEXT://0.0.0.0:9092' -e KAFKA_BROKER_ID='43' -e KAFKA_ADVERTISED_LISTENERS='PLAINTEXT://'${kfk_3_ip}':9092' -v /opt/kafka_3/logdata:/kafka -p 9094:9092 docker.io/wurstmeister/kafka
执行代码即可
更多精彩
