Jon,
Damian already answered your direct question, so my comment is a FYI:
There's a demo example at
https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/test/java/io/confluent/examples/streams/EventDeduplicationLambdaIntegrationTest.java
(this is for Confluent 3.2 / Kafka 0.10.2
Jon,
You don't need all the data for every topic as the data is partitioned by
key. Therefore each state-store instance is de-duplicating a subset of the
key set.
Thanks,
Damian
On Mon, 27 Mar 2017 at 13:47 Jon Yeargers wrote:
> Ive been (re)reading this document(
> http://docs.confluent.io/3.2.
Ive been (re)reading this document(
http://docs.confluent.io/3.2.0/streams/developer-guide.html#state-stores)
hoping to better understand StateStores. At the top of the section there is
a tantalizing note implying that one could do deduplication using a store.
At present we using Redis for this as
