I'd like to have the following: One large stream of content coming through a topic, with Kafka Stream filtering to identify records of interest. I can see how this would be sharded to allow scale out to handle a large stream of content.
I would like to have a 2nd, smaller, topic to define the areas of interest that would be used for the filtering. This topic should be available to all the stream processing filters. When a new filter comes up it should be able to recreate its state. As area of interest definitions change these changes should also go out to all the filtering applications. Can this be done directly with Kafka Streams? I can get the primary stream working with a static set of interests and the filtering works fine. But adding in a second input stream I'm having trouble. There doesn't seem to be a way to have the same topic/partition go to all the applications? Alternatively I could imagine broadcasting the interests to multiple partitions but don't see how that is done. Perhaps the area-of-interest topic should be done using a plain old Kafka producer, sending it to all partitions? Am I making sense? Happy New Year Matt
