Re: group protocol/metadata documentation

2015-11-27 Thread Jason Gustafson
Hey Dana, The intention of the UserData field is to allow custom partition assignments to leverage member-specific metadata. For example, it might include the rack name of the host for a rack-aware assignment strategy or the number of cpus for a resource-based assignment strategy. The strategies s

Re: group protocol/metadata documentation

2015-11-25 Thread Dana Powers
Thanks, Jason. I see the range and roundrobin assignment strategies documented in the source. I don't see userdata used by either -- is that correct (I may be misreading)? The notes suggest userdata for something more detailed in the future, like rack-aware placements? One other question: in what

Re: group protocol/metadata documentation

2015-11-24 Thread Jason Gustafson
Hey Dana, Have a look at this wiki, which has more detail on the consumer's embedded protocol: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal . At the moment, the group protocol supports consumer groups and kafka connect groups. Kafka tooling currently dep

group protocol/metadata documentation

2015-11-24 Thread Dana Powers
Hi all - I've been reading through the wiki docs and mailing list threads for the new JoinGroup/SyncGroup/Heartbeat APIs, hoping to add functionality to the python driver. It appears that there is a shared notion of group "protocols" (client publishes supported protocols, coordinator picks protocol