Hi Thomas and Akshay,
Sorry for the late reply. Due to problems with other FLIPs, we had to scrap
our plans to publish FLIP-182 in Flink 1.14. However as far as I know,
Arvid is working on this right now, and we are tentatively aiming with this
feature for the 1.15.0 release. I hope Arvid will be
Thanks, Piotr and Arvid. Like Thomas even I'm interested in this feature
and was wondering if I can also contribute in some means in this effort.
Thanks
Akshay
On 2021/09/08 15:44:01, Thomas Weise wrote:
> Thank you Piotr and Arvid for the context.
>
> I'm interested in helping with this fe
Thank you Piotr and Arvid for the context.
I'm interested in helping with this feature. If I'm able to contribute
before October, then I will reach out.
Thanks,
Thomas
On Tue, Sep 7, 2021 at 11:33 PM Arvid Heise wrote:
> Just to clarify: I specifically asked Piotr to not persue the FLIP if th
Just to clarify: I specifically asked Piotr to not persue the FLIP if the
current state wouldn't make it in 1.14, such that someone else can take it
over and expand it towards per-split alignment. Having a minimalistic
version in 1.14 + amendment FLIP in 1.15 would have been fine but now I
rather w
Hi Thomas,
Unfortunately me/Arvid didn't have enough time to finish this off for
1.14.0 as we were firefighting other efforts and we have re-focused on
other more advanced FLIPs. We want to deliver it for 1.15 though. I'm not
sure, but I remember Arvid saying something that he would like to actual
Hi,
I wanted to check if there is active development on FLIP-182 and what the
target release for it might be? [1] still shows as under discussion.
Regarding the per-subtask vs. per-split limitation: I think it will be
important that this eventually works per split, since in some cases it
won't be
Hi,
> I would not fully advertise this before we have the second part
implemented as well.
I'm not sure, maybe we could advertise with a big warning about this
limitation. I mean it's not as if this change would break something. At
worst it just wouldn't fully solve the problem with multiple spl
@Eron Wright The per-split watermarks are the
default in the new source interface (FLIP-27) and come for free if you use
the SplitReader.
Based on that, it is also possible to unsubscribe individual splits to
solve the alignment in the case where operators have multiple splits
assigned.
Piotr an
The notion of per-split watermarks seems quite interesting. I think the
idleness feature could benefit from a per-split approach too, because
idleness is typically related to whether any splits are assigned to a given
operator instance.
On Mon, Jul 12, 2021 at 3:06 AM 刘建刚 wrote:
> +1 for the s
+1 for the source watermark alignment.
In the previous flink version, the source connectors are different in
implementation and it is hard to make this feature. When the consumed data
is not aligned or consuming history data, it is very easy to cause the
unalignment. Source alignment can resolve ma
+1
In my opinion, this limitation is perfectly fine for the MVP. Watermark
alignment is a long-standing issue and this already moves the ball so far
forward.
I don't expect this will cause many issues in practice, as I understand it
the FileSource always processes one split at a time, and in my e
Hey!
A couple of weeks ago me and Arvid Heise played around with an idea to
address a long standing issue of Flink: lack of watermark/event time
alignment between different parallel instances of sources, that can lead to
ever growing state size for downstream operators like WindowOperator.
We had
12 matches
Mail list logo
