Stale data in KStream->KTable join

This is my third kafka streams application and I'd thought I had gotten to know the warts and how to use it correctly. But I'm beating my head against something that I just cannot explain. Values written to a table, when later read back in a join operation, are stale. Assume the following simplifi

Re: [kafka-clients] Re: [VOTE] 2.3.1 RC0

Do we have an ETA on when y'all think 2.3.1 will land? On Sat, Sep 28, 2019 at 1:55 PM Matthias J. Sax wrote: > There was a recent report about vulnerabilities of some dependent > libraries: https://issues.apache.org/jira/browse/KAFKA-8952 > > I think we should fix this for 2.3.1. > > Furthermor

Kstream to Kafka table

Hi, I have kstreams table that was created from select statement of ksql table and kstreams. Since kstream is process, how do I need to push the data into kafka topic from this stream? I also want the schema to be created in schema registry for this topic. Thanks, Asmath

Re: Kstream to Kafka table

Hey Asmath, just for the KStream question, feel free to checkout our official doc: https://kafka.apache.org/23/documentation/streams/developer-guide/dsl-api.html if you need to push to a output topic, in DSL there is a #to(String topic) function which will write the stream output to your intended

Re: Kstream to Kafka table

Thanks Chen. Can we do through commands instead of using programming library? Sent from my iPhone > On Oct 4, 2019, at 3:51 PM, Boyang Chen wrote: > > Hey Asmath, > > just for the KStream question, feel free to checkout our official doc: > https://kafka.apache.org/23/documentation/streams/deve

Re: Stale data in KStream->KTable join

Hey Trey, as I was reading, several suggestions I have are: 1. Could you revert 0ms commit interval to default? It will not help with the situation as you will try to commit on every poll() 2. I couldn't know how you actually write your code, but you could try something really simple as print st

Re: Kafka Streams changelog topic has 5 times higher out-traffic than in-traffic

Hey Xiyuan, to better understand the situation, we need to clarify the actual consumer of the changelog topic which contributes to the volume increase. You could attempt to expose some broker metrics to see the actual client, but normally there are two types of changelog consumers: 1. restore cons

Re: Kstream to Kafka table

I'm not KSQL expert, but if you want to write SQL queries as commands, you could check some instructions here: https://docs.confluent.io/current/ksql/docs/developer-guide/index.html which is built on top of Kafka Streams. On Fri, Oct 4, 2019 at 2:27 PM KhajaAsmath Mohammed wrote: > Thanks Chen.

Re: Stale data in KStream->KTable join

I think the issue here is that you're basically creating a cycle in your streams topology, which is generally supposed to be a DAG. If I understand correctly, rather than writing the new data to the underlying store you're sending it to the topic from which the table is built. Is that right? The p

[VOTE] 2.3.1 RC1

Hello all, we identified a few bugs and a dependency update we wanted to get fixed for 2.3.1. In particular, there was a problem with rolling upgrades of streams applications (KAFKA-8649). Check out the release notes for a complete list. https://home.apache.org/~davidarthur/kafka-2.3.1-rc1/RELEASE

Re: [kafka-clients] Re: [VOTE] 2.3.1 RC0

RC0 was cancelled and a new voting thread for RC1 was just sent out. Thanks! On Fri, Oct 4, 2019 at 11:06 AM Matt Farmer wrote: > Do we have an ETA on when y'all think 2.3.1 will land? > > On Sat, Sep 28, 2019 at 1:55 PM Matthias J. Sax > wrote: > > > There was a recent report about vulnerabil

Re: Stream branching and states

Hi All, I'm waiting for inputs to proceed with my POC and convert existing applications from flink to kafka streams. It would be really helpful if I can get some clarity on this. Thanks On Wed, Oct 2, 2019 at 7:05 PM Navneeth Krishnan wrote: > Hello All, > > Any inputs? > > Thanks > > On Tue,