Hi everyone,
Thanks a lot for all the feedback!
I will open a vote for it since there is no more concern.
Best,
Jing Zhang
Jing Zhang 于2022年7月5日周二 11:31写道：
> Hi ZhuZhu, Jiangjie,
>
> Thanks a lot for your feedback.
>
> I agree that it's better to support most existing events.
> I have updated
Hi ZhuZhu, Jiangjie,
Thanks a lot for your feedback.
I agree that it's better to support most existing events.
I have updated the FLIP to cover how to deal with the
RequestSplitEvent/SourceEventWrapper/ReaderRegistrationEvent.
The ReportedWatermarkEvent is only used in watermark alignment.
Water
Yes, that sounds reasonable to me. That said, supporting custom events
might still be preferable if that does not complicate the design too much.
It would be good to avoid having a tricky feature availability matrix when
we add new features to the project.
Thanks,
Jiangjie (Becket) Qin
On Mon,
Hi Jiangjie,
Yes you are that the goals of watermark alignment and speculative
execution do not conflict. For the example you gave, we can make it
work by only aligning watermarks for executions that are pipelined
connected (i.e. in the same execution attempt level pipelined region).
Even not cons
Hi Zhu,
I agree that if we are talking about a single execution region with
blocking shuffle, watermark alignment may not be that helpful as the
subtasks are running independently of each other.
That said, I don't think watermark alignment and speculative execution
necessarily conflict with each
I had another thought and now I think watermark alignment is actually
conceptually conflicted with speculative execution.
This is because the idea of watermark alignment is to limit the progress
of all sources to be around the progress of the slowest source in the
watermark group. However, speculat
Thanks for updating the FLIP!
I agree that at the moment users do not need watermark alignment(in
which case ReportedWatermarkEvent would happen) in batch cases.
However, I think the concept of watermark alignment is not conflicted
with speculative execution. It can work with speculative execution
Hi all,
After an offline discussion with Jiangjie (Becket) Qin, Guowei, Zhuzhu,
I've updated the FLIP-245[1] to including:
1. Complete the fault-tolerant processing flow.
2. Support for SourceEvent because it's useful for some user-defined
sources which have a custom event protocol between reader a
Hi Guowei,
Thanks a lot for your feedback.
Your advices are really helpful. I've updated the FLIP-245[1] to includes
these parts.
> First of all, please complete the fault-tolerant processing flow in the
FLIP.
After an execution is created and a source operator becomes ready to
receive events, s
Hi, Jing
Thanks a lot for writing this FLIP, which is very useful to Batch users.
Currently I have only two small questions:
1. First of all, please complete the fault-tolerant processing flow in the
FLIP. (Maybe you've already considered it, but it's better to explicitly
give the specific solut
Hi all,
One major problem of Flink batch jobs is slow tasks running on hot/bad
nodes, resulting in very long execution time.
In order to solve this problem, FLIP-168: Speculative Execution for Batch
Job[1] is introduced and approved recently.
Here, Zhu Zhu and I propose to support speculative exe
11 matches
Mail list logo
