. That function would however still need the info which
>> partitions were updated/added.
>> And in my case, I was not really interested in the watermark (I sent it
>> downstream only for metric purposes) but want to tell impala after each
>> commit which partitions changed, regardless
ng a notifier sounds great, if it can be executed with parallelism 1 for
all sink tasks..
Best regards
Theo
Von: "Guowei Ma"
An: "Theo Diefenthal"
CC: "user" , "yungao gy"
Gesendet: Mittwoch, 13. Mai 2020 09:15:37
Betreff: Re: 回复：Re: Writing _SUC
quot; , "Jingsong Li" <
> jingsongl...@gmail.com>
> *CC: *"Peter Groesbeck" , "user" <
> user@flink.apache.org>
> *Gesendet: *Dienstag, 12. Mai 2020 10:36:59
> *Betreff: *回复：Re: Writing _SUCCESS Files (Streaming and Batch)
>
> Hi Peter,
>
on this issue are warmly welcome.
>
> Best,
> Yun
>
>
> [1] OperatorCoordinator is introduced in FLIP-27:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface.
> This is a component resides in JobManager and could communicate with all
&
Gesendet: Dienstag, 12. Mai 2020 10:36:59
Betreff: 回复：Re: Writing _SUCCESS Files (Streaming and Batch)
Hi Peter,
Sorry for missing the question and response later, I'm currently sworking
together with Jingsong on the issue to support "global committing" (like
writing _SUCCES
ate
status from subtasks.
--原始邮件 --
发件人:Robert Metzger
发送时间:Tue May 12 15:36:26 2020
收件人:Jingsong Li
抄送:Peter Groesbeck , user
主题:Re: Writing _SUCCESS Files (Streaming and Batch)
Hi Peter,
I filed a ticket for this feature request:
https://issues.apache.org/jira/brow
Hi Peter,
I filed a ticket for this feature request:
https://issues.apache.org/jira/browse/FLINK-17627 (feel free to add your
thoughts / requirements to the ticket)
Best,
Robert
On Wed, May 6, 2020 at 3:41 AM Jingsong Li wrote:
> Hi Peter,
>
> The troublesome is how to know the "ending" for a
Hi Peter,
The troublesome is how to know the "ending" for a bucket in streaming job.
In 1.11, we are trying to implement a watermark-related bucket ending
mechanism[1] in Table/SQL.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A+Filesystem+connector+in+Table
Best,
Jingsong Lee
I am replacing an M/R job with a Streaming job using the StreamingFileSink
and there is a requirement to generate an empty _SUCCESS file like the old
Hadoop job. I have to implement a similar Batch job to read from backup
files in case of outages or downtime.
The Batch job question was answered he
