Hi, @Hongshun
Thank you for your participation in the discussion.
> I wonder whether to replace the older event-time watermark strategy in
this FLIP or a later job?
The generic watermark we are introducing will support event time watermark
on DataStream V2 in the next job, which I am currently w
Hi Xu Hang,
Sorry, it's late to join this. Thanks for your job, it make sense to me. I
have several questions:
> With this abstraction, the original event-time watermark can be seen as a
built-in use case of it.
I wonder whether to replace the older event-time watermark strategy in this
FLIP or
Hi Yunfeng, The watermark identifiers are case-sensitive, and we cannot
distinguish which connectors are internal. Therefore, the naming strategy
could be "CONNECTOR_KAFKA_IDLE." I have added the naming strategy to FLIP,
PTAL. Thanks again! Best, Xu Huang
Yunfeng Zhou 于2024年12月19日周四 14:30写道：
> H
Hi Xu Huang,
“INTERNAL_RUNTIME_BACKLOG” sounds good to me. But I’m not sure whether it is
proper to treat connector watermarks as internal, or which connectors should be
regarded as internal. I’m okay with it to add the naming strategy to the FLIP
now before we reached an agreement on the detai
Hi，@Yunfeng
> add to the FLIP a proposal to the naming convention of identifiers
Thank you for your suggestion! It's a great solution to remind users that
the identifier should be unique.
In my opinion, the Flink internal identifiers could be named as
"INTERNAL_MODULE_XXX," such as "INTERNAL_RUNT
The FLIP LGTM. Thanks Xu, Jeyhun and Weijie for preparing this.
+1 from my side
Best,
Xintong
On Wed, Dec 18, 2024 at 7:11 PM Yunfeng Zhou
wrote:
> Hi Xu Huang,
>
> I noticed your discussions mentioned that the identifier of watermarks
> needs to be global across the whole job. Therefore, w
Hi Xu Huang,
I noticed your discussions mentioned that the identifier of watermarks needs to
be global across the whole job. Therefore, would it be better to add to the
FLIP a proposal to the naming convention of identifiers? For example, the
identifiers are encouraged to be named with its hold
Thank you for participating in the discussion.
@jrlee@gmail.com
> If I have a large number of generalized watermarks that need to be
created, where should they be declared?
In the current design, the generalized watermark should be declared only
once in the first user-defined function or sou
Hi Xu Huang,
Thanks for the proposal!
I have a question: If I have a large number of generalized watermarks that
need to be created, where should they be declared? Should they be declared
only once in a single Source, or in all operators that need to send,
receive, and process them?
Best regards
Hi Devs,
Jeyhun Karimov, Weijie Guo and I would like to initiate a discussion about
FLIP-467: Introduce Generalized Watermarks [1].
Based on our findings, we recognize the need for specific events that
require propagation and alignment across streams, functioning similarly to
watermarks. An examp
Hi Xintong,
Thanks for your comments.
However, my major concern after reading this FLIP is that, the current
> design might be too complicated. It tries take all possible kinds of events
> (timestamp watermark, end-of-data, end-of-partition, internal watermark,
> and arbitrary user defined waterm
Hi Jeyhun,
Thanks for working on this FLIP.
In general, I think it's a good idea to generalize the concept of Watermark
to not only representing the advancing of event time, but general
indicators / events / signals that need to be passed along the data
streams. So +1 for working towards this dir
Hi devs,
I'd like to start a discussion about FLIP-467: Introduce Generalized
Watermarks [1] .
This is another sub-FLIP of DataStream API V2 [2].
After this FLIP one can declare generalized (custom) watermarks and define
their custom propagation and alignment process. This FLIP opens new
prospec
13 matches
Mail list logo
