Your consumers/producers will push/pull messages directly from kafka. More specifically, to/from the broker that is responsibile for the partition assigned to the consumer or corresponding to the published message key.
Others should correct me if I am wrong, but zookeeper was used to store information about topics, consumers, and offsets (version 0.8.*-ish). In newer versions of kafka (0.9.* and 0.10.*) those values are instead written to internal topics for Kafka to manage consumers, topics, and offsets. So much (possibly all?) of the functionality that zookeeper provided is being deprecated in favor of storing that info in kafka. Another thing to consider is your serialization strategy and contract between producers and consumers. You should consider predefining schemas for your topics. Confluent.io <http://docs.confluent.io/2.0.1/quickstart.html> provides steps to start using a schema registry <http://docs.confluent.io/2.0.0/schema-registry/docs/intro.html> to enforce producers. There are also libraries that leverage this schema registry to provide fast and efficient serialization and deserialization. Check out Apache Avro <https://cwiki.apache.org/confluence/display/AVRO/Index>. Also checkout https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem On Tue, Aug 30, 2016 at 7:16 PM, Jesse Whitham <jesse.whit...@loyalty.co.nz> wrote: > Hi Everyone, > > > I am trying to get some information around what my consumer should connect > to. According to the docs sending messages should be directly to the Kafka > broker and consuming messages should be from Zookeeper? I am really after a > reason this is the case as I am aware that you can just consume directly > from the Kafka broker. > > > Thanks in advance for any help! > > > Jesse Whitham > > Devops Engineer > > e. jesse.whit...@loyalty.co.nz<mailto:%20jesse.whit...@loyalty.co.nz> > > w. loyalty.co.nz<http://www.loyalty.co.nz/> > > [Loyalty NZ] > > Loyalty New Zealand Ltd. PO Box 3451, New Zealand 6011 > > Level 3, NZX Building, 11 Cable Street, Wellington > -- *David Esposito* | Product Developer | m. 678-653-1225 3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 <http://www.bettercloud.com> "We could do that better. We’ve got the right technology." <https://www.bettercloud.com/monitor/meet-brandon-williams/?utm_source=bettercloud_email&utm_medium=email_signature&utm_campaign=workplace_innovators> - Meet Brandon Williams, our newest Modern Workplace Innovator
