Jay, For the consumer: - Separation of the consumer logic from the main logic - Making it easier to build the consumer for different versions of Scala (say 2.10) - Make it easier to read from any offset you want, while being able to keep partition management features - Better support for Akka and other non-blocking / event-based frameworks (instead of a timeout, implement true hasNext functionality, for example)
thanks, Evan On Mon, Jan 21, 2013 at 9:27 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > It's worth mentioning that we are interested in exploring potential > generalizations of the producer and consumer API, but as a practical matter > most of the committers are working on getting a stable 0.8 release out the > door. So an improved consumer and producer api would be a 0.9 feature. > > If you have a concrete thing you are trying to do now that is awkward it > would be great to hear about the use case. > > Possible goals of improving the apis and client impls would include the > following: > > Producer: > 1. Include the offset in the information returned to the producer > 2. Pipeline producer requests to improve throughput for synchronous > production > > Consumer > 1. Simplify api while supporting various advanced use cases like > multi-stream consumption > 2. Make partition assignment optional and server-side (this is currently > the difference between the zk consumer and the simple consumer) > 3. Make offset management optional > 4. Remove threading from the consumer > 5. Simplify consumer memory management > > -Jay > > > > > On Mon, Jan 21, 2013 at 8:05 AM, Jun Rao <jun...@gmail.com> wrote: > > > No, but you can implement it in your application. > > > > Thanks, > > > > Jun > > > > On Sun, Jan 20, 2013 at 11:02 PM, navneet sharma < > > navneetsharma0...@gmail.com> wrote: > > > > > Is there any property to make consumer work for lets say only 10 mins > (ie > > > some kind of timer) > > > So, i want to close the consumer after 10 mins reading from broker.. > > > > > > Thanks, > > > Navneet Sharma > > > > > > > > > On Mon, Jan 21, 2013 at 11:48 AM, Ryan LeCompte <lecom...@gmail.com> > > > wrote: > > > > > > > Perfect. Thanks Jun! > > > > > > > > Ryan > > > > > > > > > > > > On Sun, Jan 20, 2013 at 10:16 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > > > Yes, see consumer.timeout.ms in > > > > http://kafka.apache.org/configuration.html > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Sun, Jan 20, 2013 at 7:14 PM, Ryan LeCompte <lecom...@gmail.com > > > > > > wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > > Does Kafka still only support blocking stream iterstors? It would > > be > > > > > great > > > > > > to pass a timeout or have a poll() operation for fetching items. > > > Right > > > > > now > > > > > > I'm always blocking in this call: > > > > > > for (m <- stream) ... > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Ryan > > > > > > > > > > > > > > > -- -- *Evan Chan* Senior Software Engineer | e...@ooyala.com | (650) 996-4600 www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala<http://www.twitter.com/ooyala>
