Re: New to Kafka

Hi Christian/Gaurav, I am looking for the user group meetings which is listed in the page https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations . User group meetings: - 1st Kafka user group meeting at LinkedIn, Jun. 14, 2012. video (part 1)

Producer connection leakage situation in 0.8.1.1

System configuration: OS: Ubuntu 14.04LTS java version "1.7.0_80", Java(TM) SE Runtime Environment (build 1.7.0_80-b15), Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) kafka version: kafka_2.10-0.8.1.1 No. of brokers: 4 All, We're experiencing a weird tcp connection leakage situ

Re: seekToBeginning doesn't work without auto.offset.reset

Is there a KIP or Jira related to " working on improving these cases with improved APIs " ? I saw that there was some discussion of it in KIP-41, but that seemed to have been resolved in favor of keeping everything inside of poll() On Fri, Mar 11, 2016 at 11:17 AM, Guozhang Wang wrote: > Cody, M

Re: Kafka Streams

Hey David, The commit always happens at a "safe point", when the local portion of the processing topology has fully processed a set of inputs. The frequency is controlled by the property commit.interval.ms. -Jay On Fri, Mar 11, 2016 at 9:28 AM, David Buschman wrote: > @Jay, I currently use rea

Re: Kafka Streams

@Jay, I currently use reactive-kaka for my Kafka sources and sinks in my stream processing apps. I was interested to see if this new stream API would make that setup easier/simpler/better in the future when it becomes available. How does the Streams API handle the commit offsets? Since you are p

Re: seekToBeginning doesn't work without auto.offset.reset

Cody, Mansi: All good points! Let me try to answer them one-by-one. About this specific issue, as I suggested in the JIRA we can separate the case about resetting offset upon initializing a partition to fetch, from the case that fetching offset out-of-range in the auto.offset.reset config. These

Re: New to Kafka

What exactly is outdated? Are you not able to grok Kafka with the current presentations? On Fri, Mar 11, 2016 at 1:01 AM, prabhu v wrote: > Hi, > > Can anyone please help me with the video presentations from Kafka experts? > > Seems the link provided in Kafka home page > > https://cwiki.apache.o

Re: Kafka broker decommission steps

I think the "scripts for operations" should be APIs that provides higher level operations. It would be simpler and more powerful. On Thu, Mar 10, 2016 at 10:54 AM Guozhang Wang wrote: > Thanks Alexis, these are good points. > > We are aware of the partition assignment json problem and trying to

Re: New to Kafka

What u need prabhu from presentation, go to YouTube u will get presentation or search Kafka example u will get . On Mar 11, 2016 9:12 PM, "prabhu v" wrote: > Hi, > > Can anyone please help me with the video presentations from Kafka experts? > > Seems the link provided in Kafka home page > > https

Re: Kafka topic deletion still not working well

Didn't finish sentence, wanted to change font but managed to hit Send button instead... As you can see from [1] replica server.log fragment, replica had one state in cache, while actual in ZK was different (node didn't exist anymore). I'm not sure why /brokers/topics/foo/partitions/0/state node d

Re: Kafka topic deletion still not working well

Sorry, I do not know if this is related to KAFKA-2937, I let you judge and decide whether ticket needs to be reopened or new one created. I'm still missing full picture what was happening but here are more pieces. 1) Topic foo was created (likely with Kafka 0.8.2.x). Topic foo was never used (no

Re: Kafka topic deletion still not working well

Hi Stevo, Let me know if we want to open Kafka-2937 again. I can include the above finding in to the patch or you want to create a separate JIra for this. Thanks, Mayuresh On Fri, Mar 11, 2016 at 7:53 AM, Mayuresh Gharat wrote: > kafka-2937 is different from this I think. Kafka-2937 deals wit

Re: Kafka topic deletion still not working well

kafka-2937 is different from this I think. Kafka-2937 deals with the delete topic getting stuck because the LeaderAndISR in ZK was updated by a controller and then the controller dies and the new controller gets in to the exception and never completes deleting the topic. The topic existed in the cl

New to Kafka

Hi, Can anyone please help me with the video presentations from Kafka experts? Seems the link provided in Kafka home page https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations is outdated.. Thanks in advance..

Re: KafkaConsumer#poll not returning records for all partitions of topic in single call

The partitions enable scalability. Consumers and produces know which partition records belong in based on their key (or manual assignment), which makes it very easy to scale up your kafka cluster or a consuming cluster. The ordering problem is one that I have faced, and have a workaround. Just kee

Re: Kafka Streams

Nice - I've read topics on the idea of a database as the 'now' view of a stream of updates, it's a very powerful concept. Reminds me of Rich Hickeys talk on DAtomic, if anyone's seen that. On 10 March 2016 at 21:26, Jay Kreps wrote: > Hey all, > > Lot's of people have probably seen the ongoing

Re: Kafka topic deletion still not working well

Topic it seems would get deleted but request in ZK to delete topic would not get cleared even after restarting Kafka cluster. I'm still investigating why deletion did not complete in the first place without restarting any nodes. It seems something smelly happens when there is request to delete mor

Re: Retry Message Consumption On Database Failure

Thanks Christian, Sending a heartbeat without having to poll would also be useful when using a large max.partition.fetch.bytes. For now I'm just going to shut the consumer down and restart after x period of time. Thanks for your insights. Michael > On 10 Mar 201

New to Kafka

Hi, Can anyone please help me with the video presentations from Kafka experts? Seems the link provided in Kafka home page https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations is outdated.. Thanks in advance..