Kafka概念

目录

环境搭建

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.0/kafka_2.12-2.3.0.tgz

tar xf kafka_2.12-2.3.0.tgz && cd kafka_2.12-2.3.0

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

cp config/server.properties config/server0.properties
# broker.id=0
# listeners=PLAINTEXT://:9092
# log.dirs=/tmp/kafka-logs0
bin/kafka-server-start.sh -daemon config/server0.properties

cp config/server.properties config/server1.properties
# broker.id=1
# listeners=PLAINTEXT://:9093
# log.dirs=/tmp/kafka-logs1
bin/kafka-server-start.sh -daemon config/server1.properties

cp config/server.properties config/server2.properties
# broker.id=2
# listeners=PLAINTEXT://:9094
# log.dirs=/tmp/kafka-logs2
bin/kafka-server-start.sh -daemon config/server2.properties

jps
1
2
3
4
5
6
7
8
9
10
11
12
13
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic topic-demo --partitions 3 --replication-factor 3

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 -topic topic-demo
# hello kafka

bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --topic topic-demo
# >hello kafka

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic topic-demo
# Topic: topic-demo PartitionCount:3 ReplicationFactor:3 Configs:
# Topic: topic-demo Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
# Topic: topic-demo Partition: 1 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
# Topic: topic-demo Partition: 2 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2

水平扩展

Producer

  • 扩展Topic => Multi Partition

kafka-concepts-01.png

  • Broker Count MUST >= Partition Factor

Comsumer

  • 扩展Consume => Comsumer Group

kafka-concepts-02.png

  • Consumer Count SHOULD <= Partition Factor

高可用性

ZooKeeper

1
2
3
4
5
Server failure detection

Data partitioning

In-sync data replication

Replicas

kafka-concepts-03.png

数据一致

kafka-concepts-04.png

  • AR(所有副本) = ISR(和leader同步的副本) + OSR(未和leader同步的副本)

  • 不是完全同步 => 因为性能较差 & 也不是单纯异步 => 因为一致性较差

参考