Re: at-least-once guarantee?

Interesting. .. wouldn't the producer sequence grow without bounds, in the first case, even with the simpler non-ha of key assumption, to provide a strict exactly once semantics? In other words, wouldn't you need to store the entire set of keys that the broker has ever seen to ensure that a potent

Re: reprocessing messages in kafka

It is possible to consume the same message more than once with the same consumer. However WHAT you actually do with the message (such as idempotent writes) is the tricker part. Regards Milind On Wed, Jul 31, 2013 at 8:22 AM, Oleg Ruchovets wrote: > Hi , > > I just don't know which mail list is

Re: Truncated messages in fetch response

Jay is correct. ..it manifests itself when the size threshold condition is violated. For erlang client, this is fixed in 0.4.6 of mps. Github.com/milindparikh/mps Regards Milind On Jun 27, 2013 3:30 PM, "Bob Potter" wrote: > Vadim, > > I don't know under exactly what conditions it happens but

Re: Using Kafka for "data" messages

You might want to take a look at mps (github.com/milindparikh/mps). As I was thinking about blogging about it, your use-case magically surfaced on the mailing list. Since one of the supposed benefits of having a framework is to enable quick building of stuff on top of it, I took a crack at your use

Re: mps, an eventing framework for kafka

ot; wrote: > Milind, > > Thanks for sharing. How did you consume Kafka data in Erlang? > > Jun > > > On Fri, Jun 14, 2013 at 10:23 PM, Milind Parikh >wrote: > > > There is a lot of useful discussion on the user group about data messages > > and eventin

mps, an eventing framework for kafka

There is a lot of useful discussion on the user group about data messages and eventing, I thought that I would introduce mps, an eventing framework for kafka. It is written in Erlang and available on githubof at github.com/milindparikh/mps. The README on github explains the philosophy with which

Re: Partitioning and scale

Number of files to manage by os, I suppose. Why wouldn't you use consistent hashing with deliberately engineered collisions to generate a limited number of topics / partitions and filter at the consumer level? Regards Milind On May 23, 2013 4:22 PM, "Timothy Chen" wrote: > Hi Neha, > > Not sure

Kafka support

If a F500 company wants commercial support for Kafka, who would they turn to? It appears that there seems to be natural fit with real time processing schemes aka storm&trident. I am sure that someone in the community must have come across this issue. Thanks Milind

Re: Keeping logs forever

Forever is a long time. The definition of replay and navigating through different versions of kafka would be key. Example: If you are storing market data into kafka and have a cep engine running on top and would like replay "transactions" to be fed back to ensure replayability, then you would prob

Re: 0.8 producer -- many questions

I am in the process of updating my erlang producer/consumer client to 0.8. Being lazy, I don't really want to go to Scala code. So I am just going through https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol.