Re: Mirror Maker 2.0 Queries

2020-07-13 Thread Ryanne Dolan
Ananya, yes the driver is distributed, but each worker only communicates via kafka. They do not listen on any ports. Ryanne On Sat, Jul 11, 2020, 11:28 AM Ananya Sen wrote: > Hi > > I was exploring the Mirror maker 2.0. I read through this > > https://cwiki.apache.org/confluence/display/KAFKA/K

Re: NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 3

2020-07-13 Thread Nag Y
Thanks Liam, so the phrase " current ISR set " in the warning refers to ISR set that is being shown as kafka-topics describe command ? And also, should the "maximum" value of " min.insync.replicas" be (replication factor - 1 ) - I mean min.insync.replicas should not be same as " replication factor

Kafka connect after restart doesn't read connectors configuration

2020-07-13 Thread Сергей Булавинцев
Hi, I have a question regarding kafka connect and how it operates in distributed mode. We use docker swarm to deploy kafka connect and only one instance of it in swarm. After node crashing with kafka connect and starting another instance on a new node it starts to read configuration topic from off

ktable - ktable join

2020-07-13 Thread Dumitru-Nicolae Marasoui
Hello kafka community, In a ktable-ktable join, assuming that kt1 has k1 and v1 that contains k2 and kt2 has k2 and v2, is it possible that the (k1, v1, k2, v2) pair is never emitted? I am trying to understand how it works and if any race condition would be possible. If race conditions would not be

3 kafka-streams or a single kafka streams

2020-07-13 Thread Dumitru-Nicolae Marasoui
Hello kafka community, I imagine the following behavior that I could code in 3 kafka-streams pipelines and wondering if it can be done in fewer kafka streams with the same guarantees: I have 3 compacted topics, t1, t2 and t3, where t2 is the link (many-many) between t1 & t3. The same about t4-t6.

Re: 3 kafka-streams or a single kafka streams

2020-07-13 Thread Dumitru-Nicolae Marasoui
Hello kafka community, Sorry, besides the 3 kafka streams that just merge topics into another common topic, there must be consumers from those merged topics, that keep local state, and that, once both t1 and t2 values for a particular t1 key exist, emit the pair, and will keep emitting pairs multip

Re: 3 kafka-streams or a single kafka streams

2020-07-13 Thread Dumitru-Nicolae Marasoui
Hello kafka community, Instead of the consumer, I can also have kafka-streams for stateful transformation (aggregation / fold) of both types of events & detection when both sides are present to emit a pair. In fact I thought a bit more and in my particular case, if we will agree that I do not need

kafka-streams merge + aggregate vs merge + to topic + from topic + aggregate

2020-07-13 Thread Dumitru-Nicolae Marasoui
Hi, I would like to understand the ordering guarantees if any at the merge operator level. I think unless I am writing into a topic, there can be no ordering guarantees, is that so? I would normally do a key transformation, merge, write to an output topic, and i know the partition (key) ordering g