PHPFixing
  • Privacy Policy
  • TOS
  • Ask Question
  • Contact Us
  • Home
  • PHP
  • Programming
  • SQL Injection
  • Web3.0

Monday, September 19, 2022

[FIXED] What may cause huge load in Kafka `__consumer_offsets` topic?

 September 19, 2022     apache-kafka, consumer, internal, load, performance     No comments   

Issue

I have simple observation in my Kafka cluster (Kafka 0.11.0.0).

According to jmx information the __consumer_offsets topic constantly is loaded with 10 times more messages than sum of all messages in all other topics. I had also conected console consumer to this topic and I can measure similar values.

  • What could be the reason?
  • How can I check what is Kafka broker doing and generate such a load on its own?

Solution

to read __consumer_offsets topic:

bin/kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server brokers --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --new-consumer --consumer.config consumer.conf

for kafka 11 use formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"

where consumer.conf has one line

exclude.internal.topics=false



Answered By - Natalia
Answer Checked By - Timothy Miller (PHPFixing Admin)
  • Share This:  
  •  Facebook
  •  Twitter
  •  Stumble
  •  Digg
Newer Post Older Post Home

0 Comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Total Pageviews

Featured Post

Why Learn PHP Programming

Why Learn PHP Programming A widely-used open source scripting language PHP is one of the most popular programming languages in the world. It...

Subscribe To

Posts
Atom
Posts
Comments
Atom
Comments

Copyright © PHPFixing