Re: security: encryption at rest and key rotation idea

2016-01-24 Thread Jens Rantil
Josh, Could you maybe post your final solution here in this thread? I'm curious to hear about which way you finally went with this. Thanks, Jens On Fri, Jan 22, 2016 at 12:47 AM, Josh Wo wrote: > Thanks Jim. I like the fact that the offset management will not require us > to customize kafka. I

Re: Missing ISR On Some Partitions

2016-01-24 Thread Raju Bairishetti
Partitions on 3rd nide was not replicated to other nodes. Those partitions are having only one copy. If the node goes down then it will be ended of having data loss. This is not an expected behavior.I think you may need to check replica fetcher threads to see why partitions are not getting r

Dealing with diverse consumption speeds

2016-01-24 Thread Jens Rantil
Hi, How are you dealing with a slow consumer in Kafka? In the best of world, each consumer will have the exact same specs and the exact same workload. But unfortunately that's rarely true: Virtual machines share hardware with other VMs, some Kafka tasks takes longer to process, some partition keys

subscribe kafka

2016-01-24 Thread Richard He

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-24 Thread Krzysztof Ciesielski
Yes, it's exactly 5 seconds on every machine. Sure, I'll open a JIRA On Jan 22, 2016 7:24 AM, "Jason Gustafson" wrote: > Hi Krzysztof, > > This is definitely weird. I see the data in the broker's send queue, but > there's a delay of 5 seconds before it's sent to the client. Can you create > a JIR

Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements

2016-01-24 Thread Ismael Juma
Hi, On Sun, Jan 24, 2016 at 7:17 PM, Krzysztof Ciesielski < krzysztof.ciesiel...@softwaremill.com> wrote: > Yes, it's exactly 5 seconds on every machine. Sure, I'll open a JIRA > Jason already did so: https://issues.apache.org/jira/browse/KAFKA-3135 Feel free to add relevant information there.

Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message

2016-01-24 Thread Dave Peterson
On Wed, Jan 20, 2016 at 11:43 PM, Joel Koshy wrote: > Yes - compaction is a topic-level property. You can use --describe to > verify that the topic is compacted or not and if you didn't intend it to be > compacted you can alter the configuration. > I tried using the --describe option with kafka-