While at the Kafka Summit I was asked to write up a list of challenges and confusions my team encountered using Kafka. We are using 0.9.0.1 and use the new Java KakfaConsumer.
1. The new Java KafkaConsumer doesn’t have a method to return the high watermark (last offset in the topic/partition's log. 2. Can’t connect using the Java client to just check status on topics (committed offset for different consumer groups, high watermark, etc) 3. kafka-consumer-groups.sh requires a member of the consumer group to be connected and consuming or offset values won't be displayed (artificial prerequisite) 4. Default config for tracking committed offsets is poor (commits should be very permanent shouldn’t age out after 24 hours). 5. It should not be possible to set an offset.retention time < log.retention time. 6. Consumer group rebalances affect all consumers across all topics within the consumer group including topics without a new subscriber. 7. Changing the broker config requires a 1-at-a-time roll of all the cluster, a service kafka reload would be nice. 8. Console consumer still uses “old” consumer style configuration options (--zookeeper). This is a bit strange for anyone who has started using Kafka with version 0.9 or later, since the cli options don’t correspond to what you expect the consumer to need. 9. Heartbeat only on poll() causes problems when we have gaps in consuming before committing (such as when we publish files and don’t want to commit until the publish is complete). Supposedly position() will perform a heartbeat too in addition to poll() (I haven’t verified this but heard it at the Kafka Summit), but it does add extra complexity to the application. Thanks for listening, Cliff Rhyne -- Cliff Rhyne Software Engineering Manager e: [email protected] signal.co ________________________ Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2016 Signal. All rights reserved.
