[FIXED] How to create multiple kafka consumer groups in same topic

Issue

Below is required scenario.

Topic-1 has 6 partitions, now I want to create 3 consumer groups cg1,cg2 and cg3 and map it like this (cg1 - 0,1 ; cg2 - 2,3 ; cg3 - 4,5). How can i create it using kafka-console-consumer.sh or kafka-consumer-groups.sh

Even Kafka documentation explained about this scenario but nowhere mentioned how to do it. Any help is appreciated !!!

Solution

Kafka Consumer Group is a collection of consumers who shared the same group id. Consumer Group distributes processing by sharing partitions across consumers.

The diagram below shows a single topic with three partitions and a consumer group with two members. Each partition in the topic is assigned to exactly one member of the group.

Note: topic with n partition can at max consume by n consumer of Consumer Group with 1 partition per consumer.

In your case, if you use a consumer group on a topic means all partitions will get assigned to that Consumer group.

But if you are not interested in Consumer Group you can directly assign a partition to each consumer group in that case rebalance will not come in the picture

I am using Kafka Confluent-kafka 2.6.0-5.1.2:

sh kafka-console-consumer --bootstrap-server localhost:9092  --partition 0 --topic abc --group cg1
sh kafka-console-consumer --bootstrap-server localhost:9092  --partition 1 --topic abc --group cg1

--partition <Integer: partition> : The partition to consume from Consumption starts from the end of the partition unless '--offset' is specified.

Using consumer group you can describe consumer details

sh kafka-consumer-groups --bootstrap-server  localhost:9020 --describe --group a
TOPIC    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
abc         0          123           678            0                 -               -               -
abc         1          234           345            0                 -               -               -

You can also manually assigned partition through Java as below

List<TopicPartition> partitions = new ArrayList<>();
partitions.add(new TopicPartition("abc", 0));
partitions.add(new TopicPartition("abc", 1));
......
new KafkaConsumer<>(consumerProperties).assign(partitions);

Note that it isn't possible to mix manual partition assignment (i.e. using assign) with dynamic partition assignment through topic subscription (i.e. using subscribe).

Ref: here

There are below alternate approaches:

1. Use 3 separate topics to consume messages using a separate consumer group.
2. Programmatically filter partitions while consuming messages.

Answered By - Nitin

Answer Checked By - Willingham (PHPFixing Volunteer)