Consumers can be split up based on partitions. So, you can tell a consumer group to listen to several topics and it will divvy up the work. Your use case sounds very canonical. I would take a look at Kafka connect (if you’re using the confluent stack).
-Daivd http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html On 9/11/17, 4:48 PM, "Nishanth S" <nishanth.2...@gmail.com> wrote: All, I am very new to kafka . We have a case where we need to ingest multiple avro record types . These avro record types vary vastly in volume and size and I am thinking of sending each of these message types to a different topic and creating partitions based on volume and through put needed. What the kafka consumer has to do is read the record of from partitions and write to different hdfs locations based on record type . I am guessing we should at least start with one consumer per topic . Is this understanding correct or is there a better way to look at it?
