; Agreed@Henn. If work is batch-oriented, Spark or other batch processing
> system is a more suitable solution.
>
>
>
> ---Original---
> From: "Harper Henn"
> Date: Tue, Mar 26, 2019 00:36 AM
> To: "users";
> Subject: Re: Tracking progress for messages
s";
Subject: Re: Tracking progress for messages generated by a batch process
Assuming you know how many items are in a batch ahead of time, could you
just add a batch ID and position of a message within a batch to each
message you send to topic A? Then your end application (streaming app 3)
Assuming you know how many items are in a batch ahead of time, could you
just add a batch ID and position of a message within a batch to each
message you send to topic A? Then your end application (streaming app 3)
could check if every message in that batch has been processed, and trigger
events wh
Sounds reasonable to me.
-Matthias
On 3/22/19 9:50 AM, Tim Gent wrote:
> Hi all,
>
> We have a data processing system where a daily batch process generates
> some data into a Kafka topic. This then goes through several other
> components that enrich the data, these are also integrated via Kafka.
Hi all,
We have a data processing system where a daily batch process generates
some data into a Kafka topic. This then goes through several other
components that enrich the data, these are also integrated via Kafka.
So overall we have something like:
Batch job -> topic A -> streaming app 2 -> top
