Re: question on doing deduplication with KafkaStreams

2018-02-27 Thread Bill Bejeck
From: Marasoiu, Nicu [nicu.maras...@metrosystems.net] > Sent: Tuesday, February 27, 2018 11:03 AM > To: users@kafka.apache.org > Subject: question on doing deduplication with KafkaStreams > > Hi, > From a programatic perspective, doing a groupByKey.reduce((val1, val2) -> > v

RE: question on doing deduplication with KafkaStreams

2018-02-27 Thread Marasoiu, Nicu
))) .reduce((val1, val2) -> val1) From: Marasoiu, Nicu [nicu.maras...@metrosystems.net] Sent: Tuesday, February 27, 2018 11:03 AM To: users@kafka.apache.org Subject: question on doing deduplication with KafkaStreams Hi, >From a progr

question on doing deduplication with KafkaStreams

2018-02-27 Thread Marasoiu, Nicu
Hi, >From a programatic perspective, doing a groupByKey.reduce((val1, val2) -> >val1) would deduplicate entries, but then I have a few questions: this state >would accumulate without limit, right? Should we do a windowing, to eliminate >old records be needed, right? Will the state accumulate jus