Re: Tracking progress for messages generated by a batch process

2019-03-26 Thread Tim Gent
h processing. Is it possible > there's another datastore that's better suited for your use case? > > On Fri, Mar 22, 2019 at 11:04 AM Matthias J. Sax > wrote: > > > Sounds reasonable to me. > > > > -Matthias > > > > On 3/22/19 9:50 AM,

Tracking progress for messages generated by a batch process

2019-03-22 Thread Tim Gent
Hi all, We have a data processing system where a daily batch process generates some data into a Kafka topic. This then goes through several other components that enrich the data, these are also integrated via Kafka. So overall we have something like: Batch job -> topic A -> streaming app 2 -> top