Unclaimed partitions

Hi, I am trying to consume a 24-partition topic across 12 processes. Each process is using the simple consumer API, and each is being assigned two consumer threads. I have noticed when starting these processes that sometimes some of my processes are not being assigned any partitions, and no reba

Re: Unclaimed partitions

you are assigning partitions in your application code. Did >you mean to say that you are using the high level consumer API? > >Thanks, >Aditya > >____ >From: Dave Hamilton [dhamil...@nanigans.com] >Sent: Tuesday, April 28, 2015 7:58 AM &g

Re: Unclaimed partitions

signing partitions in your >>application code. Did you mean to say that you are using the high level >>consumer API? >> >>Thanks, >>Aditya >> >> >>From: Dave Hamilton [dhamil...@nanigans.com] >>Sent: Tuesday, April 28, 2015 7:58 AM &g

Re: Unclaimed partitions

Hi, would anyone be able to help me with this issue? Thanks. - Dave On Tue, Apr 28, 2015 at 1:32 PM -0700, "Dave Hamilton" mailto:dhamil...@nanigans.com>> wrote: 1. We’re using version 0.8.1.1. 2. No failures in the consumer logs 3. We’re using the ConsumerOffsetChe

Re: Unclaimed partitions

e anything to indicate that the rebalance >process on those hosts even started? Does this happen occasionally or every >time you start the consumer group? Can you paste the output of >ConsumerOffsetChecker and describe topic? > >Thanks, >Aditya >_

Re: Kafka connector throughput reduction upon avro schema change

Hi, does anyone have advice on how to deal with this issue? Is it possible that changing a schema compatibility setting could correct it? Thanks, Dave On 5/26/17, 1:44 PM, "Dave Hamilton" wrote: We are currently using the Kafka S3 connector to ship Avro data to S3. We made a

Re: Kafka connector throughput reduction upon avro schema change

Bumping this. Has anyone here observed this in their Kafka connect deployments? Thanks, Dave On 5/26/17, 1:44 PM, "Dave Hamilton" wrote: We are currently using the Kafka S3 connector to ship Avro data to S3. We made a change to one of our Avro schemas and have noticed consumer

Error finding consumer coordinators after restart

Hello, we ran into a memory issue on a Kafka 0.10.0.1 broker we are running that required a system restart. Since bringing Kafka back up it seems the consumers are having issues finding their coordinators. Here are some errors we’ve seen in our server logs after restarting: [2017-01-12 19:02:10

Re: Error finding consumer coordinators after restart

Just wanted to close the loop on this. It seems the consumer offset logs might have been corrupted by the system restart. Deleting the topic logs and restarting the Kafka service cleared up the problem. Thanks, Dave On 1/12/17, 2:29 PM, "Dave Hamilton" wrote: Hello, we

Kafka log.message.format.version and consumer client versions

Hi, I have a question about the performance implications of upgrading the Kafka message format relating to the following from the upgrade documentation: The message format in 0.10.0 includes a new timestamp field and uses relative offsets for compressed messages. The on disk message format can b

Kafka connector throughput reduction upon avro schema change

We are currently using the Kafka S3 connector to ship Avro data to S3. We made a change to one of our Avro schemas and have noticed consumer throughput on the Kafka connector drop considerably. I am wondering if there is anything we can do to avoid such issues when we update schemas in the futur