gt; @Eduardo: Thanks for sharing your approach! Not sure if I understood it
>> correctly, but I think that the approach does not guarantee that all results
>> of a window are emitted before the end-of-window marker is written.
>> Since the sink operator and the single-task-opera
sure if I understood it
> correctly, but I think that the approach does not guarantee that all
> results of a window are emitted before the end-of-window marker is written.
> Since the sink operator and the single-task-operator are separate
> operators, the output records might get stuck (or be b
results of a window are emitted before the end-of-window marker is written.
Since the sink operator and the single-task-operator are separate
operators, the output records might get stuck (or be bufffered) in one of
the sink tasks and the single-task would still emit an end-of-window marker
record
Hi,
I'll chip in with an approach I'm trying at the moment that seems to work,
and I say seems because I'm only running this on a personal project.
Personally, I don't have anything against end-of-message markers per
partition, Padarn you seem to not prefer this option as it overloads the
meaning
Hi again Fabian,
Thanks for pointing this out to me. In my case there is no need for keyed
writing - but I do wonder if having each kafka task write only to a single
partition would significantly affect performance.
Actually now that I think about it, the approach to just wait for the first
recor
Hi Padarn,
Yes, this is quite tricky.
The "problem" with watermarks is that you need to consider how you write to
Kafka.
If your Kafka sink writes to keyed Kafka stream (each Kafka partition is
written by multiple producers), you need to broadcast the watermarks to
each partition, i.e., each parti
Hi Fabian, thanks for your input
Exactly. Actually my first instinct was to see if it was possible to
publish the watermarks somehow - my initial idea was to insert regular
watermark messages into each partition of the stream, but exposing this
seemed quite troublesome.
> In that case, you could
Hi Padarn,
What you describe is essentially publishing Flink's watermarks to an
outside system.
Flink processes time windows, by waiting for a watermark that's past the
window end time. When it receives such a WM it processes and emits all
ended windows and forwards the watermark.
When a sink rece
Hello Users,
I have a question that is perhaps not best solved within Flink: It has to
do with notifying a downstream application that a Flink window has
completed.
The (simplified) scenario is this:
- We have a Flink job that consumes from Kafka, does some preprocessing,
and then has a sliding w
