Re: Abort transaction semantics

2021-02-19 Thread Guozhang Wang
Hello Peter, Note that when you upgrade from 2.4 to later versions in Kafka, your error handling could be modified and simplified a bit as well. You can read the example code in KIP-691 as a reference: https://cwiki.apache.org/confluence/display/KAFKA/KIP-691%3A+Enhance+Transactional+Producer+Exce

Re: Window Store

2021-02-19 Thread Guozhang Wang
Hello Navneeth, I would agree with Liam that a session store seems a good fit for your case. But note that session stores would not expire a session themselves and it is still the processor node's job to find those already expired sessions and emit results / delete. You can take a look at the KStr

Re: Abort transaction semantics

2021-02-19 Thread Peter Cipov
This was really helpful. Thank you On Thu, Feb 18, 2021 at 8:08 PM Boyang Chen wrote: > Thanks for the question. I think Gary provided an excellent answer. > Additionally, you could check out the code example > < > https://urldefense.com/v3/__https://github.com/apache/kafka/blob/trunk/examples/

[Spark SQL] - Not able to consume Kafka topics

2021-02-19 Thread Rathore, Yashasvini
Hello, Issues : * I and my team are trying to consume some kafka topics based on the timestamps using startingOffsetsByTimestamps option, and the code works fine when we run via a Databricks notebook. * There is a need to setup the whole process in a local system (IntelliJ), but the sa