Hi everyone,
I created a FLIP and started a discussion around that topic [1].
Best,
Arvid
[1]
https://lists.apache.org/thread.html/r8357d64b9cfdf5a233c53a20d9ac62b75c07c925ce2c43e162f1e39c%40%3Cdev.flink.apache.org%3E
On Tue, Jun 8, 2021 at 11:46 PM Eron Wright
wrote:
> Thanks, the narrowed
Thanks, the narrowed FLIP-167 is fine for now. I'll re-activate the vote
process. Thanks!
On Tue, Jun 8, 2021 at 3:01 AM Till Rohrmann wrote:
> Hi everyone,
>
> I do agree that Flink's definition of idleness is not fully thought through
> yet. Consequently, I would feel a bit uneasy to make it
Hi everyone,
I do agree that Flink's definition of idleness is not fully thought through
yet. Consequently, I would feel a bit uneasy to make it part of Flink's API
right now. Instead, defining the proper semantics first and then exposing
it sounds like a good approach forward. Hence, +1 for optio
Hi Eron,
The FLIP-167 is narrow, but we recently discovered some problems with
current idleness semantics as Arvid explained. We are planning to present a
new proposal to redefine them. Probably as a part of it, we would need to
rename them. Given that, I think it doesn't make sense to expose idle
> > >> >
> > > > > > > >> > gist is a new `markIdle` method on the `StreamOperator`
> > > > interface,
> > > > > > > >> >
> > > > > > > >> > that
> > > > > > > >
> > > >> > Please let me know if you'd like me to proceed to update the
> > > FLIP
> > > > > > >> >
> > > > > > >> > with
> > > > > > >> >
> > > > > >
o
> > > > > >> >
> > > > > >> > be
> > > > > >> >
> > > > > >> > encoded.
> > > > > >> >
> > > > > >> > It seems like this point was asked, but not followe
].
> > > > >> > If you start it, I can already cast a binding vote and we just
> > > > >> >
> > > > >> > need 2
> > > > >> >
> > > > >> > more
> > > > >> >
> >
>> > wrote:
> > > >> >
> > > >> >
> > > >> > Arvid,
> > > >> > Thanks for the feedback. I investigated the japicmp
> > > >> >
> > > >> > configuration,
> > > >> >
> >> >
> > >> > 1. fair point. It still feels odd to have writeWatermark in the
> > >> > SinkFunction (it's supposed to be functional as you mentioned),
> > >> >
> > >> > but I
> > >> >
> > &
gt;> >
> >> > is really just context information of that particular record and
> >> >
> >> > I
> >> >
> >> > don't
> >> >
> >> > see any use beyond timestamp and watermark. For SinkFunction, I'd
> >> >
io
>> >
>> > .invalid>
>> > wrote:
>> >
>> >
>> > Arvid,
>> >
>> > 1. I assume that the method name `invoke` stems from
>> >
>> > considering
>> >
>> > the
>> >
>> > Sink
he same as for elements.
> >
> > Thanks,
> > Eron
> >
> > On Tue, May 25, 2021 at 12:42 PM Arvid Heise >
> > wrote:
> >
> > Hi Eron,
> >
> > I think the FLIP is crisp and mostly good to go. Some smaller
> > things/qu
14 (with this FLIP) or vice versa?
>5. What happens with sinks that use the new methods in
>
> applications
>
> that
>
>do not have watermarks (batch mode, processing time)? Does
>
> this
>
> also
>
> work
>with ingestion time sufficiently?
>
>>> programming model is more unified in that respect since 1.12
> > >>>>>> (FLIP-134).
> > >>>>>>>> 6. The failure behavior is the same as for elements.
> > >>>>>>>>
> > >>>>>&
go. Some smaller
> >>>>>>>>> things/questions:
> >>>>>>>>>
> >>>>>>>>>1. SinkFunction#writeWatermark could be named
> >>>>>>>>>SinkFunction#invokeWatermark or invokeOnWa
;>>>>>>>StreamingFileSink bucket policies. We may add that
>>> processing
>>>>> time
>>>>>>>> flag
>>>>>>>>>also to SinkWriter#Context in the future.
>>>>>>>>>3. Alter
tly once sinks deal with written watermarks
> in
> > > > case
> > > > > of
> > > > > > > >failure? I guess it's the same as normal records. (Either
> > > > rollback
> > > > > > of
> > > > > &
gt; > > > > > wrote:
> > > > > > >
> > > > > > > > Does anyone have further comment on FLIP-167?
> > > > > > > >
> > > > > > > >
> > > > > > >
>
> > > > > > Eron
> > > > > > >
> > > > > > >
> > > > > > > On Thu, May 20, 2021 at 5:02 PM Eron Wright <
> > > ewri...@streamnative.io
> > > > >
> > > > > > > wrote:
> &g
for Sink API:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API
> > > > > >
; >
> > > > > > I'd like to call a vote next week, is that reasonable?
> > > > > >
> > > > > >
> > > > > > On Wed, May 19, 2021 at 6:28 PM Zhou, Brian
> > wrote:
> > > > > >
> > > > > >
cussion and I read through Eron's pull request
> > and I
> > > > >> think this can benefit Pravega Flink connector as well.
> > > > >>
> > > > >> Here is some background. Pravega had the watermark concept through
> > the
> >
nted to propagate the Flink watermark to Pravega in the
> > > >> SinkFunction, and at that time we just used the existing Flink API
> > that
> > > we
> > > >> keep the last watermark in memory and check if watermark changes for
> > > each
>
existing Flink API
> that
> > we
> > >> keep the last watermark in memory and check if watermark changes for
> > each
> > >> event[2] which is not efficient. With such new interface, we can also
> > >> manage the watermark propagation much more
ich is not efficient. With such new interface, we can also
> >> manage the watermark propagation much more easily.
> >>
> >> [1] https://pravega.io/blog/2019/11/08/pravega-watermarking-support/
> >> [2]
> >>
> https://github.com/pravega/fl
/src/main/java/io/pravega/connectors/flink/FlinkPravegaWriter.java#L465
>>
>> -----Original Message-
>> From: Arvid Heise
>> Sent: Wednesday, May 19, 2021 16:06
>> To: dev
>> Subject: Re: [DISCUSS] Watermark propagation with Sink API
>>
>>
>> [
in/java/io/pravega/connectors/flink/FlinkPravegaWriter.java#L465
>
> -Original Message-
> From: Arvid Heise
> Sent: Wednesday, May 19, 2021 16:06
> To: dev
> Subject: Re: [DISCUSS] Watermark propagation with Sink API
>
>
> [EXTERNAL EMAIL]
>
> Hi Eron,
>
> Than
flink/FlinkPravegaWriter.java#L465
-Original Message-
From: Arvid Heise
Sent: Wednesday, May 19, 2021 16:06
To: dev
Subject: Re: [DISCUSS] Watermark propagation with Sink API
[EXTERNAL EMAIL]
Hi Eron,
Thanks for pushing that topic. I can now see that the benefit is even bigger
t
Hi Eron,
Thanks for pushing that topic. I can now see that the benefit is even
bigger than I initially thought. So it's worthwhile anyways to include that.
I also briefly thought about exposing watermarks to all UDFs, but here I
really have an issue to see specific use cases. Could you maybe take
Update: opened an issue and a PR.
https://issues.apache.org/jira/browse/FLINK-22700
https://github.com/apache/flink/pull/15950
On Tue, May 18, 2021 at 10:03 AM Eron Wright
wrote:
> Thanks Arvid and David for sharing your ideas on this subject. I'm glad
> to hear that you're seeing use cases f
Thanks Arvid and David for sharing your ideas on this subject. I'm glad to
hear that you're seeing use cases for watermark propagation via an enhanced
sink interface.
As you've guessed, my interest is in Pulsar and am exploring some options
for brokering watermarks across stream processing pipeli
Hi Eron,
Thanks for starting this discussion. I've been thinking about this recently
as we've run into "watermark related" issues, when chaining multiple
pipelines together. My to cents to the discussion:
How I like to think about the problem, is that there should an invariant
that holds for any
Hi Eron,
I think this is a useful addition for storage systems that act as
pass-through for Flink to reduce recovery time. It is only useful if you
combine it with regional fail-over as only a small part of the pipeline is
restarted.
A couple of thoughts on the implications:
1. Watermarks are clo
I would like to propose an enhancement to the Sink API, the ability to
receive upstream watermarks. I'm aware that the sink context provides the
current watermark for a given record. I'd like to be able to write a sink
function that is invoked whenever the watermark changes. Out of scope
would
35 matches
Mail list logo
