Could you just use a topic for each stage? The nice thing about topics is you don't even have to tell the Kafka cluster about the topics beforehand, so if you a new stage appears, just start writing to that topic.
Kafka is great. :-) Phiilp On Thu, Jun 27, 2013 at 7:27 AM, Hussain Pirosha <hussain.piro...@impetus.co.in> wrote: > A source produces data in kafka at different stages of execution. Topic is > partitioned by number of execution stages. > Partition number shall give us that information. > > Eg. 0 specifies data received by source, #last partition specifies processed > data delivered by source etc. > > Thanks, > Hussain > > -----Original Message----- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Thursday, June 27, 2013 7:47 PM > To: users@kafka.apache.org > Subject: Re: How to get partition information from Kafka Stream? > > Currently, partition # is not exposed in MessageAndMetaData. We do plan to > expose such information in a future release. How do you plan to use this info? > > Thanks, > > Jun > > > On Thu, Jun 27, 2013 at 7:13 AM, Hussain Pirosha < > hussain.piro...@impetus.co.in> wrote: > >> After retrieving a kafka stream or kafka message how to get the >> corresponding partition number to which it belongs ? I am using kafka >> version 0.7.2. >> More specifically kafka.consumer.KafkaStream and >> kafka.message.MessageAndMetaData classes, does not provide API to >> retrieve partition number. Are there any other API's to get the partition >> number ? >> >> Thanks, >> Hussain >> >> >> ________________________________ >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited >> when received in error. Impetus does not represent, warrant and/or >> guarantee, that the integrity of this communication has been >> maintained nor that the communication is free of errors, virus, interception >> or interference. >> > > ________________________________ > > > > > > > NOTE: This message may contain information that is confidential, proprietary, > privileged or otherwise protected by law. The message is intended solely for > the named addressee. If received in error, please destroy and notify the > sender. Any use of this email is prohibited when received in error. Impetus > does not represent, warrant and/or guarantee, that the integrity of this > communication has been maintained nor that the communication is free of > errors, virus, interception or interference.
