Re: New kafka client for Go (golang)

How does it compare to Sarama? On Mon, Mar 30, 2015 at 3:09 PM, Piotr Husiatyński wrote: > Hi, > I wanted to share new client library for Go language that I'm developing at > Optiopay. Library is MIT licensed and provides API close to latest kafka > request/response wire format with few higher l

Re: Which version works for kafka 0.8.2 as consumer?

Is there a reason the incomplete version was included in the 0.8.2.1 release? On Wed, Apr 1, 2015 at 1:02 PM, Mayuresh Gharat wrote: > What do you mean by offset syncing? > > Thanks, > > Mayuresh > > On Wed, Apr 1, 2015 at 9:59 AM, pushkar priyadarshi < > priyadarshi.push...@gmail.com> wrote: >

Re: How to consume from a specific topic, as well as a wildcard of topics?

Yeah, I think you need to have 2 consumer connectors (I routinely have multiple consumer connectors co-existing in the same app). That error message about the ephemeral node is really annoying, by the way. It happens under lots of scenarios (at least it did under 0.8.1.1), where it simply never r

Re: Side-by-side migration for 0.7 to 0.8

Hi Patrick, When we went through this, we 'shaded' the old kafka jar, so the 2 could co-exist in the same app. We use maven, and there's a maven 'shade plugin', etc. In our case, it was intractable to try to update all producers and consumers "in one go" as you suggest, so we had to have a way i

Side-by-side migration for 0.7 to 0.8

Hello, Does anyone have tips or advice for side-by-side migration from 0.7 to 0.8 clusters? We're in the process of migrating, and are bringing up our 0.8 cluster and first producer & consumer now. We're running into issues with getting the 0.7 and 0.8 jar's to play nice, as it seems like there's n

How to consume from a specific topic, as well as a wildcard of topics?

Hi, I want to consume from both a specific topic "a_topic" as well as all topics that match a certain prefix "prefix.*". When I do that using a single instance of a ConsumerConnector, I get a hang when creating the 2nd set of message streams. Code: ConsumerConnector consumer = Consume

question about Kafka

Hello, group I am a newbie to Kafka. I am researching on how to commit a new appended log message (e.g. apache access log) to Kafka. Could you please share some ideas/solutions? Thanks Liang

RE: Problem with node after restart no partitions?

Likewise, I was not "at the wheel" when this was happening, and there very well could have been a similar situation of not waiting for a controlled shutdown to complete successfully. Fortunately we did not end up in exactly your situation where the entire cluster went down, but I can say I know

Re: delete.retention.ms in 0.8.1

Hello I am using 0.8.1.1 only . I configured log.retention.minutes=2 and log.retention.check.interval.ms=1000 and configured delete.retention.ms=6000 . This help's me to delete the log from kafka's topic folder every 2 minutes. thanks gaurav On Fri, Apr 3, 2015 at 5:50 PM, Madhukar Bharti wrot

Re: Problem with node after restart no partitions?

I will provide what I can (we don't have separate logs for controller, etc., it's all integrated in a single log with log4j for us, we embed Kafka in a java container, etc). Are there specific log classes you'd be interested in seeing? (I can look at the default log4j configs to see what's set up

Re: Problem with node after restart no partitions?

This sounds a very serious issueŠ Could you provide the controller log and the log for the first broker on which you tried controlled shutdown and upgrade? On 4/3/15, 8:57 AM, "Jason Rosenberg" wrote: >I'm preparing a longer post here, but we recently ran into a similar >scenario. Not sure yet

Re: delete.retention.ms in 0.8.1

Another thing is that the active log segment would not be deleted, so if there is only one log segment, it will not honor the retention policy. You may config log.roll.ms to make sure you have a rolled over log segment. On 4/3/15, 5:20 AM, "Madhukar Bharti" wrote: >Hi Gaurav, > >What is your "lo

partition reassignment

I'm in the process of reassigning partitions away from failing machines and it appears to be stuck. One thought is because our machines are failing at a very high rate and so some partitions no longer have any live replicas at all. At this point I don't care about the data, I just want to get

Re: partition size

Yes it is based on the machines capacity. In practice we try to limit partitions to about 50GB to ensure the data is evenly spread across machines and that the recovery time of a failure is minimized. -Clark Sent from my iPhone > On Apr 3, 2015, at 6:17 AM, Nirmal ram wrote: > > Hi, > > Wh

Re: Problem with node after restart no partitions?

I'm preparing a longer post here, but we recently ran into a similar scenario. Not sure yet if it's the same thing you saw (but it feels similar). We were also doing a rolling upgrade from 0.8.1.1 to 0.8.2.1, and during the controlled shutdown of the first node (of a 4 node cluster), the controll

partition size

Hi, What is the maximum size a partition can have? Does it depends on the machines storage capacity?

Re: delete.retention.ms in 0.8.1

Hi Gaurav, What is your "log.retention.check.interval.ms" ? There might be a chance it will be high so it is not able to delete in specified interval. And also in Kafka 0.8.1 it will be "retention.ms". Please check this Regards, Madh

delete.retention.ms in 0.8.1

hello group, I have created a topic with the delete retention ms time 1000 and send and consume message across. Nothing happened after that , i checked the log also , message also not deleted as well. Please help me to come to know what is the need