That's great. Is there anything I can do to force that particular topic to start at the earliest message every time the job is started? On Mar 25, 2016 10:20 AM, "Yasuhiro Matsuda" <yasuhiro.mats...@gmail.com> wrote:
> It may not be ideal, but there is a way to prioritize particular topics. It > is to set the record timestamps to zero. This can be done by using a custom > TimestampExtractor. Kafka Streams tries to synchronize multiple streams > using the extracted timestamps. So, records with the timestamp 0 have > greater chance to be processed earlier than others. > > On Thu, Mar 24, 2016 at 6:57 PM, Greg Fodor <gfo...@gmail.com> wrote: > > > Really digging Kafka Streams so far, nice work all. I'm interested in > > being able to materialize one or more KTables in full before the rest > > of the topology begins processing messages. This seems fundamentally > > useful since it allows you to get your database tables replicated up > > off the change stream topics from Connect before the stream processing > > workload starts. > > > > In Samza we have bootstrap streams and stream prioritization to help > > facilitate this. What seems desirable for Kafka Streams is: > > > > - Per-source prioritization (by defaulting to >0, setting the stream > > priority to 0 effectively bootstraps it.) > > - Per-source initial offset settings (earliest or latest, default to > > latest) > > > > To solve the KTable materialization problem, you'd set priority to 0 > > for its source and the source offset setting to earliest. > > > > Right now it appears the only control you have for re-processing is > > AUTO_OFFSET_RESET_CONFIG, but I believe this is a global setting for > > the consumers, and hence, the entire job. Beyond that, I don't see any > > way to prioritize stream consumption at all, so your KTables will be > > getting materialized while the general stream processing work is > > running concurrently. > > > > I wanted to see if this case is actually supported already and I'm > > missing something, or if not, if these options make sense. If this > > seems reasonable and it's not too complicated, I could possibly try to > > get together a patch. If so, any tips on implementing this would be > > helpful as well. Thanks! > > > > -Greg > > >
