Schema validation in JDBC Connector

2020-07-10 Thread vishnu murali
Hi Guys First I am creating a topic and set a schema first and then I am trying to take data from MySQL using JDBC source connector. In that time how can I validate the data from MySQL matches the schema I set in schema registry? Can any one have any idea about this??

help needed, kafka-streams 2.5.0 does not support Named on reduce?

2020-07-10 Thread Dumitru-Nicolae Marasoui
Hello kafka community, Hi, kafka-streams 2.5.0 does not support Named on reduce? I configured it but it does not take it..the grouped ones takes the named & see the repartitioned topics with that name but the reduced does not take that name; i would need in order to be able to demo nicely azkarra s

Re: help needed, kafka-streams 2.5.0 does not support Named on reduce?

2020-07-10 Thread Dumitru-Nicolae Marasoui
Hello kafka community, I understand, the Named argument i used is used to config used to name the processor in the topology, But is there any way I can give a custom, more readable name to an intermediate topic like kstream-aggregate-... ? Thanks On Fri, 10 Jul 2020 at 12:34, Dumitru-Nicolae Maras

Re: Schema validation in JDBC Connector

2020-07-10 Thread Steve Howard
Hi VIshnu, Is there a reason you wouldn't want to allow the connector to handle the schema registration and evolution over time? The assumption in most cases is that the database is the source of truth. If you register the schema yourself in the schema registry, you are introducing a tight coupl

Re: Detected out-of-order KTable update warnings

2020-07-10 Thread Richard Rossel
Thanks Matthias, it makes sense, now I need to find out why the topic is not sorted by timestamp. The topic I'm using to be loaded as globalKTable is partitioned by key, and the timestamp is being handled by kafka. I have multiple clients, different machines, pushing data to that topic, maybe the

Re: Detected out-of-order KTable update warnings

2020-07-10 Thread Matthias J. Sax
By default, timestamps are set by the producer. Thus, if different producers write to the same topic-partition, their writes are interleaved and even if each individual client sends data ordered, they might end up out-of-order in the topic. As you are only interested in per-key ordering, it might

Re: help needed, kafka-streams 2.5.0 does not support Named on reduce?

2020-07-10 Thread Matthias J. Sax
It should be possible to name all processors/stores/topics. However, it's not always clear how the API works -- in fact, some `Named` parameters "overlap". It's a know issue and not well documented. We need to fix this at some point. If you want to name a changelog topic, you can name the corresp

Re: Detected out-of-order KTable update warnings

2020-07-10 Thread Richard Rossel
Configuring the topic to let the brokers set up the timestamps should fix my problem. Thanks for the explanation and for sharing your talk. On Fri, Jul 10, 2020 at 12:28 PM Matthias J. Sax wrote: > By default, timestamps are set by the producer. > > Thus, if different producers write to the sam

Kafka Mirror Maker 2.0

2020-07-10 Thread Ananya Sen
I was exploring the Mirror maker 2.0. I read through this https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 documentation and I have a few questions. 1) For running mirror maker as a dedicated mirror maker cluster, the documentation specifies a config file and a st