I tried this numerous times (regardless of PLAINTEXT/SSL connnections). The setup is to do with a single node (1 Zeek, 1 Broker) as mentioned in the startup docs on Kafka site.
1) Follow kafka documentation to initialise all Zk and Kafka props. 2) Enable automatic topic creation with a minimum partition 1 in broker properties. 3) Start Zk and Kafka. 4) Use the standard shipped producer.properties file with bootstrap.servers, key.serializer, and value.serializer properties values. 5) Use the standard shipped consumer.properties file with the following settings: max.poll.interval.ms=21414111 #consumer group id group.id=group1 key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer enable.auto.commit=true auto.offset.reset=earliest fetch.min.bytes=1 session.timeout.ms=20000 Since we are using console consumer, I kept the enable.auto.commit=true (i think every 5 secs is the default interval for committing auto) 4) Run following command: kafka-console-producer.bat --topic test --broker-list localhost:9092 --producer.config ..\..\config\producer.properties > HELLO KAFKA!!! > FINISHING > ^ C (keyboard close producer) 5) Run the following command kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --new-consumer --consumer.config ..\..\config\consumer.properties --from-beginning HELLO KAFKA!!! FINISHING (wait for 5 seconds or more) ^C (keyboard close consumer) 6) Now repeat #4 with some more messages and close producer 7) Repeat #5.- confirm that all messages (old + new ones) are being displayed >From the help documentation for console-consumer. > --from-beginning If the conumer doesn not already have an established > offset to consume from, start from the earliest message present in the log > rather than the latest message. The consumer group is present so any offset committed correctly should have been used by a newly added consumer to retrieve offset details. `auto.offset.reset=earliest` but that's not relevant since offsets have been committed and should have been found. When I did the following: kafka-consumer-groups.bat --bootstrap.server localhost:9092 --describe --group group1 --new-consumer I see "Error group1 does not exist". I also tried to use kafka-consumer-offset-checker - but ran into the error " org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/group1/offsets/test/0" Which seems to be a bug as mentioned by - https://stackoverflow.com/questions/22563401/rebalancing-issue-while-reading-messages-in-kafka My question is - why does the console consumer fail to retrieve the offset information and loses data? Or is this expected? I didn't have any consumer offset expiry default configs altered. So any info. is helpful. I don't believe this is relevant to Windows/Linux but if anyone has this example (with the same setup) working in Linux ONLY - it's good to know. I can confirm that I can see all the consumer offsets created in the specific log directory in broker prop file. Kindest Regards,
