Re: Kafka on yarn

Steve - yes I have been monitoring YARN advances in this area particularly YARN-1051 which seems to have most of what long running services with hard node locality requirements need and is based on MS's Rayon framework (https://issues.apache.org/jira/secure/attachment/12628143/curino_MSR-TR-2013

Re: Kafka on yarn

Jay - good points on rolling broker upgrades/config changes and the challenges of having an app master coordinate this type of thing. Not really specific to kafka but something you would hope an app master managing these types of services would take care off.  I also thought an app master should

Re: Kafka on yarn

Hi Gwen Your recommendations in the field to partition off non-cluster nodes and reserve them for kafka brokers totally make sense given current YARN limitations. I'm familiar with the llama hacks - effectively reserving containers with dummy processes that just sit there and then running the '

ConsumerConnector not processing partitions on a particular kafka broker.

Hello all Some background. I have a 3 kafka brokers A,B and C, there is a kafka topic called topic with 20 partitions (no replicas). Everything has been working fine for about a week when suddenly all the data sent to partitions belonging to broker C are not seen by the Consumer the consumer is

Re: ConsumerConnector not processing partitions on a particular kafka broker.

Hi Pablo, During the period did you see any exception/errors on Broker C's logs and the consumer logs also? Guozhang On Thu, Jul 24, 2014 at 6:23 AM, Pablo Picko wrote: > Hello all > > Some background. > > I have a 3 kafka brokers A,B and C, there is a kafka topic called topic > with 20 parti

Re: ConsumerConnector not processing partitions on a particular kafka broker.

Hey Guozhang Thanks for the reply, No nothing at all in the logs to suggest anything went wrong. Its really puzzling as to what's happened. When I restarted the consumer everything worked again. Prior to the restart I even stopped the producer for a bit. However any messages that got assigned to

Re: how to ensure strong consistency with reasonable availability

About re-publishing m2. it seems it's better to let the producer choose whether to do this or not. There is another known bug KAFKA-1211 that's not fixed yet. The situation when this can happen is relatively rare and the fix is slightly involved. So, it may not be addressed in 0.8.2. Thanks, Jun

Durability

Hi, I have come back to looking at Kafka after a while. Is it really the case that messages can be lost if the producer is disconnected from the broker, as described in KAFKA-789 , and touched on with some elaboration in KAFKA-156

Re: how to ensure strong consistency with reasonable availability

Thanks for the Jira info. Just to clarify, in the case we are outlining above, would the producer would have received an ack on m2 (with acks = -1) or not? If not, then I have no concerns, if so, then how would the producer know to re-publish? On Thu, Jul 24, 2014 at 9:38 AM, Jun Rao wrote: >

Re: ConsumerConnector not processing partitions on a particular kafka broker.

Pablo, Do you see any rebalance related logs in consumers? Guozhang On Thu, Jul 24, 2014 at 9:02 AM, Pablo Picko wrote: > Hey Guozhang > > Thanks for the reply, No nothing at all in the logs to suggest anything > went wrong. > > Its really puzzling as to what's happened. When I restarted the

Re: ConsumerConnector not processing partitions on a particular kafka broker.

Guozhang I didn't no. I did spot other people with similar symptoms to my problem mentioning your suggestion too but I don't see anything in the log to suggest it rebalanced. It could very well be the reason but I can't see anything suggesting it is yet. Thanks Pablo On 24 Jul 2014 17:57, "Guozha

Re: how to ensure strong consistency with reasonable availability

With ack=-1, the producer is guaranteed to receive an error when publishing m2 in the above case. Thanks, Jun On Thu, Jul 24, 2014 at 9:46 AM, Scott Clasen wrote: > Thanks for the Jira info. > > Just to clarify, in the case we are outlining above, would the producer > would have received an a

Serious Bug? Segment getting deleted as soon as it is rolled over

We just noticed that one of our topics has been horribly misbehaving. *retention.ms * for the topic is set to 120960 ms However, segments are getting schedule for deletetion as soon as a new one is rolled over. And naturally consumers are running into a kafka.common.Offse

Re: ConsumerConnector not processing partitions on a particular kafka broker.

Hmm, that is a bit wired. Did you make sure the consumer logs are turned at least on the INFO level? Guozhang On Thu, Jul 24, 2014 at 10:05 AM, Pablo Picko wrote: > Guozhang > > I didn't no. I did spot other people with similar symptoms to my problem > mentioning your suggestion too but I don'

Re: Serious Bug? Segment getting deleted as soon as it is rolled over

Relevant parameters from server.properties log.dir=/var/lib/fk-3p-kafka/logs log.flush.interval.messages=1 log.flush.interval.ms=1000 log.retention.hours=168 log.segment.bytes=536870912 log.cleanup.interval.mins=1 log.retention.hours=336 On Thu, Jul 24, 2014 at 10:34 PM, Kashyap Paidimarri

Re: ConsumerConnector not processing partitions on a particular kafka broker.

What is the value for what you are setting for your number of streams when calling createMessageStreamsByFilter or if using createMessageStreams for the TopicCount ( topic -> numberOfStreams )? How are you threading the iterator on each stream? /*** Joe St

Re: ConsumerConnector not processing partitions on a particular kafka broker.

Hey guys.. I have my my log level set to info, saying that I am not seeing much logs at all for kafka on startup i see detail about the serializer.class my producer uses but very little consumer related logs is there anything I should always see if my log config is correct for the info level In r

Re: ConsumerConnector not processing partitions on a particular kafka broker.

For the consumer you should see logs like "Connecting to zookeeper instance at " + config.zkConnect "begin registering consumer " + consumerIdString + " in ZK consumerThreadId + " successfully owned partition " + partition + " for topic " + topic "starting auto committer every " + config.autoCommi

Re: ConsumerConnector not processing partitions on a particular kafka broker.

Pablo, if you see this again, can you take a thread-dump of your consumer and verify that the fetchers to all the brokers are still alive as well as the corresponding iterator threads? It could be that your consumer ran into some decoder error or some other exception (although in general that shoul

Lost messages during leader election

Hi, I have a test that continuously sends messages to one broker, brings up another broker, and adds it as a replica for all partitions, with it being the preferred replica for some. I have auto.leader.rebalance.enable=true, so replica election gets triggered. Data is being pumped to the old broke

Re: Lost messages during leader election

Hi Jad, Thanks for bring this up. It seems to be a valid issue: in the current auto leader rebalancer thread's logic, if the imbalance ratio threshold is violated, then it will trigger the preferred leader election whether or not the preferred leader is in ISR or not. Guozhang On Thu, Jul 24, 2

Re: Lost messages during leader election

Hi Guozhang, Isn't it also possible to lose messages even if the preferred leader is in the ISR, when the current leader is ahead by a few messages, but the preferred leader still has not caught up? Thanks, Jad. On Thu, Jul 24, 2014 at 4:59 PM, Guozhang Wang wrote: > Hi Jad, > > Thanks for b

Re: Lost messages during leader election

With ack=-1 all messages produced to leader must have been acked by all replicas to respond. So that will not cause data loss. On Thu, Jul 24, 2014 at 6:07 PM, Jad Naous wrote: > Hi Guozhang, > > Isn't it also possible to lose messages even if the preferred leader is in > the ISR, when the curr

Re: Lost messages during leader election

Ah yes. OK, thanks! So it seems like we should only manually trigger re-election after seeing that all replicas are in the ISR. Is there a bug to follow this up? Thanks, Jad. On Thu, Jul 24, 2014 at 6:27 PM, Guozhang Wang wrote: > With ack=-1 all messages produced to leader must have been acke

Re: Lost messages during leader election

Actually, is the following scenario possible? - We start off with only 1 replica (the leader) - the producer continuously sends messages - a new replica (the preferred one) comes online - it becomes an ISR just after an ack is sent to the producer - the new replica gets elected as the new leader, b

Re: Lost messages during leader election

I'm still not sure I understand after his reply - http://qnalist.com/questions/5034216/lost-messages-during-leader-election - I really need a tutorial on Kafka. I don't understand why they made it so complicated when Cassandra and Hbase are similar but simpler. * Ashwin Jayaprakash* | Engineeri

Re: Serious Bug? Segment getting deleted as soon as it is rolled over

Have you configured log.retention.bytes? Thanks, Jun On Thu, Jul 24, 2014 at 10:04 AM, Kashyap Paidimarri wrote: > We just noticed that one of our topics has been horribly misbehaving. > > *retention.ms * for the topic is set to 120960 ms > > However, segments are get

Re: Serious Bug? Segment getting deleted as soon as it is rolled over

No, we haven't configured that. We have a few hundred topics but this seems to be the only one affected (I did a quick check, not thorough). The relevant config params that we have set in server.properties. log.dir=/var/lib/fk-3p-kafka/logs log.flush.interval.messages=1 log.flush.interval.ms=

Re: Serious Bug? Segment getting deleted as soon as it is rolled over

Attached a transcript that explains what I'm seeing On Fri, Jul 25, 2014 at 10:52 AM, Kashyap Paidimarri wrote: > No, we haven't configured that. We have a few hundred topics but this > seems to be the only one affected (I did a quick check, not thorough). > > The relevant config params that we