Re: Deduplicating KStream-KStream join

2017-05-09 Thread Ofir Sharony
> >> > >> > >> -Matthias > >> > >> > >> On 5/5/17 6:54 AM, Adrian McCague wrote: > >>> Hi Matthias > >>> > >>> We have been thinking about this problem recently and thought, wouldn't > >> it be

Re: Deduplicating KStream-KStream join

2017-05-08 Thread Matthias J. Sax
;1 time', within the >> retention period of the join window. So if a join has occurred already on a >> particular key, further ones will be ignored for the remainder of the >> retention period (or maybe the retention is reset). Assuming I have not >> missed a feature on the cur

Re: Deduplicating KStream-KStream join

2017-05-06 Thread Ofir Sharony
erent from putting a > transform after the join as a second state store would be required? > > > > Adrian > > > > -Original Message- > > From: Matthias J. Sax [mailto:matth...@confluent.io] > > Sent: 04 May 2017 23:55 > > To: users@kafka.apache.or

Re: Deduplicating KStream-KStream join

2017-05-05 Thread Matthias J. Sax
:matth...@confluent.io] > Sent: 04 May 2017 23:55 > To: users@kafka.apache.org > Subject: Re: Deduplicating KStream-KStream join > > Hi, > > we don't believe in triggers ;) > https://www.confluent.io/blog/watermarks-tables-event-time-dataflow-model/ > > -> Thus, it

RE: Deduplicating KStream-KStream join

2017-05-05 Thread Adrian McCague
ssage- From: Matthias J. Sax [mailto:matth...@confluent.io] Sent: 04 May 2017 23:55 To: users@kafka.apache.org Subject: Re: Deduplicating KStream-KStream join Hi, we don't believe in triggers ;) https://www.confluent.io/blog/watermarks-tables-event-time-dataflow-model/ -> Thus, it&#

Re: Deduplicating KStream-KStream join

2017-05-04 Thread Matthias J. Sax
Hi, we don't believe in triggers ;) https://www.confluent.io/blog/watermarks-tables-event-time-dataflow-model/ -> Thus, it's a BigQuery flaw to not support updates... (IMHO) (We are also considering improving KStream-KStream join though, but that's of course no short term solution for you: https

Deduplicating KStream-KStream join

2017-05-04 Thread Ofir Sharony
Hi guys, I want to perform a join between two KStreams. An event may appear only on one of the streams (either one of them), so I can't use inner join (which emits only on a match) or left join (which emits only when the left input arrives). This leaves me with outer join. The problem with outer j