Re: Architecture: amount of partitions

2014-08-08 Thread Guozhang Wang
Kane, The in-built offset management is already in master branch, and will be included in 0.8.2. For now you can give the current trunk a spin. Guozhang On Fri, Aug 8, 2014 at 1:42 PM, Kane Kane wrote: > Hello Guozhang, > > Is storing offsets in kafka topic already in master branch? > We woul

Re: Architecture: amount of partitions

2014-08-08 Thread Kane Kane
Hello Guozhang, Is storing offsets in kafka topic already in master branch? We would like to use that feature, when do you plan to release 0.8.2? Can we use master branch meanwhile (i.e. is it stable enough). Thanks. On Fri, Aug 8, 2014 at 1:38 PM, Guozhang Wang wrote: > Hi Roman, > > Current K

Re: Architecture: amount of partitions

2014-08-08 Thread Guozhang Wang
Hi Roman, Current Kafka messaging guarantee is at-least once, and we are working on transactional messaging features to make it exactly once. We are expecting it to be used as synchronization/replication layer for some storage systems as your use case after that. As for your design, since you wil

Re: Architecture: amount of partitions

2014-08-08 Thread Jonathan Weeks
The approach may well depend on your deploy horizon. Currently the offset tracking of each partition is done in Zookeeper, which places an upper limit on the number of topic/partitions you want to have and operate with any kind of efficiency. In 0.8.2 hopefully coming in the next month or two,

Architecture: amount of partitions

2014-08-08 Thread Roman Iakovlev
Dear all, I'm new to Kafka, and I'm considering using it for a maybe not very usual purpose. I want it to be a backend for data synchronization between a magnitude of devices, which are not always online (mobile and embedded devices). All the synchronized information belong to some user, and ca