Consumer sitting idle not consuming kafka messages from topic

Pekko based application pods are idle and not consuming kafka messages from all partitions during initial startup. This happens when first time input topic is created and the applications starts (first time consumer group is created from application itself) **Workaround**: This start to work when we create the same consumer group from kafka cli scripts before starting the application. **Evidence Logs**: ``` ./kafka-topics.sh --bootstrap-server b-1.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-3.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-2.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098 --topic df-tmf628-dp-input-real-hxd1 --create --partitions 5 --command-config client.properties Created topic df-tmf628-dp-input-real-hxd1. ./kafka-topics.sh --bootstrap-server b-1.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-3.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-2.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098 --topic df-tmf628-dp-input-real-hxd1 --describe --command-config client.properties Topic: df-tmf628-dp-input-real-hxd1 TopicId: ApzaR2gSQzGl5fg4YHwcdQ PartitionCount: 5 ReplicationFactor: 3 Configs: min.insync.replicas=2,message.format.version=2.8-IV1,unclean.leader.election.enable=true Topic: df-tmf628-dp-input-real-hxd1 Partition: 0 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1 Topic: df-tmf628-dp-input-real-hxd1 Partition: 1 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3 Topic: df-tmf628-dp-input-real-hxd1 Partition: 2 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: df-tmf628-dp-input-real-hxd1 Partition: 3 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: df-tmf628-dp-input-real-hxd1 Partition: 4 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 ``` ``` ./kafka-console-consumer.sh --bootstrap-server "b-1.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-3.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-2.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098" --group df-tmf628-dp-input-real-hxd1-Tmf628Normalizer --topic df-tmf628-dp-input-real-hxd1 --consumer.config client.properties ^CProcessed a total of 0 messages hdahiya@IN-HDAHIYA-T14:~/tool/kafka_2.12-3.4.0/bin$ ./kafka-consumer-groups.sh --bootstrap-server "b-1.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-3.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-2.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098" --group df-tmf628-dp-input-real-hxd1-Tmf628Normalizer --command-config client.properties --timeout 10000 --describe ``` ``` Consumer group 'df-tmf628-dp-input-real-hxd1-Tmf628Normalizer' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 4 0 0 0 - - - df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 2 0 0 0 - - - df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 1 0 0 0 - - - df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 3 0 0 0 - - - df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 0 0 0 0 ``` ` ./kafka-consumer-groups.sh --bootstrap-server "b-1.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-3.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-2.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098" --group df-tmf628-dp-input-real-hxd1-Tmf628Normalizer --command-config client.properties --timeout 10000 --describe` ``` GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 4 0 0 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-92ff0fa9-4a8c-4763-b2af-732faed37bcd /10.101.3.225 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 2 0 0 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-92ff0fa9-4a8c-4763-b2af-732faed37bcd /10.101.3.225 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 1 0 0 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-92ff0fa9-4a8c-4763-b2af-732faed37bcd /10.101.3.225 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 3 0 0 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-92ff0fa9-4a8c-4763-b2af-732faed37bcd /10.101.3.225 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 0 0 0 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-92ff0fa9-4a8c-4763-b2af-732faed37bcd /10.101.3.225 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 hdahiya@IN-HDAHIYA-T14:~/tool/kafka_2.12-3.4.0/bin$ in$ ./kafka-umer-groups.sh --bootstrap-server "b-1.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-3.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-2.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098" --group df-tmf628-dp-input-real-hxd1-Tmf628Normalizer --command-config client.properties --timeout 10000 --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 0 18 18 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-193b09fe-e4ec-4184-a403-fcbb60a2dbd4 /10.101.23.206 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 1 0 2 2 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-193b09fe-e4ec-4184-a403-fcbb60a2dbd4 /10.101.23.206 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 2 3 3 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-193b09fe-e4ec-4184-a403-fcbb60a2dbd4 /10.101.23.206 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 3 8 8 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-92ff0fa9-4a8c-4763-b2af-732faed37bcd /10.101.3.225 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 4 4 4 0 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1-92ff0fa9-4a8c-4763-b2af-732faed37bcd /10.101.3.225 consumer-df-tmf628-dp-input-real-hxd1-Tmf628Normalizer-1 ``` Root Cause: The difference between 2 steps i mean when application create the consumer group by itself vs we create manually consumer group of application is CURRENT-OFFSET is - when application create the consumer group CURRENT-OFFSET is 0 when we create the consumer group manually for the application Reference below: ``` ~/tool/kafka_2.12-3.4.0/bin$ ./kafka-consumer-groups.sh --bootstrap-server "b-1.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-3.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098,b-2.bdamskcluster.4rpefn.c4.kafka.eu-west-1.amazonaws.com:9098" --group df-tmf628-dp-input-real-hxd1-Tmf628Normalizer --command-config client.properties --timeout 10000 --describe Consumer group 'df-tmf628-dp-input-real-hxd1-Tmf628Normalizer' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 4 0 0 0 - - - df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 2 0 0 0 - - - df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 1 0 0 0 - - - df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 3 0 0 0 - - - df-tmf628-dp-input-real-hxd1-Tmf628Normalizer df-tmf628-dp-input-real-hxd1 0 0 0 0 additionally , The consumer group created by application has 2 additionally properties defined i.e. .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") .withProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false") ``` More Info: we are using apache pekko scala library for streaming applications Status: We have work around for this solution but we donot know why this problem occured when application creates the consumer group by itself