Re: resilient producer

On Tue, Jan 15, 2013 at 3:12 PM, Corbin Hoenes wrote: > +1 how about posting yours to GitHub? > Sounds like a good contrib project. > There is nothing to post at the moment as we're currently in the requirements gathering phase :) Potentially, we might have a contrib project along the lines of

Re: resilient producer

Jay, Thanks for your insight! More comments are below. On Tue, Jan 15, 2013 at 3:18 PM, Jay Kreps wrote: > I can't speak for all users, but at LinkedIn we don't do this. We just run > Kafka as a high-availability system (i.e. something not allowed to be > down). These kind of systems require

Re: resilient producer

I can't speak for all users, but at LinkedIn we don't do this. We just run Kafka as a high-availability system (i.e. something not allowed to be down). These kind of systems require more care, but we already have a number of such data systems. We chose this approach because local queuing leads to d

Re: resilient producer

+1 how about posting yours to GitHub? Sounds like a good contrib project. Sent from my iPhone On Jan 15, 2013, at 12:29 PM, Stan Rosenberg wrote: > Hi, > > In out current data ingestion system, producers are resilient in the sense > that if data cannot be reliably published (e.g., network is

Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis

>When the producer >tries to send to the old broker (which is either dead, or a slave now), >the broker will either not respond, or the response will contain an error >code. In this case, the broker sends a response with an error code to the producer, and then the producer retries the metadata r

Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis

Hey Guys, Correct me if I'm wrong, but I believe in 0.8, the producer uses a metadata request to get topic/partition mappings from the broker. The broker then interacts with ZK (rather than having the producer do it using zk.connect). In the event that the master for a topic/partition fails, a ne

Re: The future of 0.7

Broadly, the strategy stays the same, except that the 0.8 broker will not be able to serve requests coming from a pre-0.8 client. So, the upgrade path will be and we will document this in detail soon - 1. Start a 0.8 Kafka cluster overlaid on the 0.6/0.7 hardware or on separate hardware if you can

Re: The future of 0.7

Will it be possible to upgrade from 0.6 directly to 0.8, and what would the migration strategy be like? I understand for 0.6 to 0.7, the strategy was to first upgrade your consumers, then the broker, then the producers. -Evan On Tue, Jan 15, 2013 at 9:44 AM, Neha Narkhede wrote: > To answer y

Re: hadoop-consumer code in contrib package

Please read the Kafka design paper . It may look a little long, but it's as short as it can be. Kafka differs from other messaging system in a couple of ways, and it's important to understand the fundamental design choices that were made in order to understand

Re: The future of 0.7

To answer your question, even if the team is working on 0.8, we understand that it will take some time for 0.8 to be completely stable and for users to build some confidence in moving to 0.8 since it is backwards incompatible. Keeping that in mind, we will try our best to patch the 0.7 branch for b

Re: The future of 0.7

Our current plan to to get 0.8 out asap and only patch blocker issues in 0.7. If the issues that you found apply to 0.8, please feel free to open a jira and/or attach a patch. Thanks, Jun On Tue, Jan 15, 2013 at 9:07 AM, Wouter de Bie wrote: > Hi all, > > We're currently using Kafka 0.7 and he

The future of 0.7

Hi all, We're currently using Kafka 0.7 and here and there we're running in a few bugs here and there. Since 0.8 is around the corner, it might not make too much sense in sending the patches for 0.8 upstream, unless 0.7 will live a (long) life next to 0.8. What's your take on the future of the

Re: hadoop-consumer code in contrib package

Thanks Felix for sharing your work. Contrib hadoop-consumer looks like the same way. I think i need to really understand this offset stuff. So far i have used only high level consumer.When consumer is done reading all the messages, i used to kill the process(because it won't on its own). Again i

Re: kafka-0.7.2 sbt and remote installs help

It should work on at least all versions of java 6 and beyond. Thanks, Jun On Tue, Jan 15, 2013 at 8:24 AM, Joseph Crotty wrote: > Jun, > > Will it matter if the versions of Java are different between the two > machines? > > Joe > > > On Mon, Jan 14, 2013 at 4:04 PM, Jun Rao wrote: > > > You ne

Re: kafka-0.7.2 sbt and remote installs help

Jun, Will it matter if the versions of Java are different between the two machines? Joe On Mon, Jan 14, 2013 at 4:04 PM, Jun Rao wrote: > You need to do "./sbt update " and "./sbt package" on your local machine > with internet access. Then, you can copy the whole dir to your remote > machine.

Re: Number of Partitions Per Broker

At LinkedIn, some of the high volume topics are configured with more than 1 partition per broker. Having more partitions increases I/O parallelism for writes and also increases the degree of parallelism for consumers (since partition is the unit for distributing data to consumers). On the other ha

Re: SyncProducer vs Producer

That code is probably outdated. It should use Producer. Could you file a jira? Thanks, Jun On Mon, Jan 14, 2013 at 11:50 PM, navneet sharma < navneetsharma0...@gmail.com> wrote: > If that is the case: > "Producer is actually the client api that everyone > should be using." > > Then why contrib/