[jira] [Created] (FLINK-38180) Race condition between failing Task and cancelation hiding the real exception

2025-08-01 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-38180: -- Summary: Race condition between failing Task and cancelation hiding the real exception Key: FLINK-38180 URL: https://issues.apache.org/jira/browse/FLINK-38180

[jira] [Created] (FLINK-38158) FLIP-484: Add custom metric variables to operators

2025-07-30 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-38158: -- Summary: FLIP-484: Add custom metric variables to operators Key: FLINK-38158 URL: https://issues.apache.org/jira/browse/FLINK-38158 Project: Flink Issue

[jira] [Created] (FLINK-38056) Flink doesn't wait for s5cmd to finish when it is being cancelled

2025-07-07 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-38056: -- Summary: Flink doesn't wait for s5cmd to finish when it is being cancelled Key: FLINK-38056 URL: https://issues.apache.org/jira/browse/FLINK-38056 Project:

Re: [VOTE] FLIP-526: Model ML_PREDICT, ML_EVALUATE Table API

2025-05-29 Thread Piotr Nowojski
+1 (binding) Best, Piotrek czw., 29 maj 2025 o 05:58 Shengkai Fang napisał(a): > +1(binding) > > Best, > Shengkai > > Timo Walther 于2025年5月29日周四 00:00写道: > > > +1 (binding) > > > > We should still check whether changes to QueryOperationVisitor are > > necessary but this is internal API and sho

Re: [VOTE] FLIP-530: Dynamic job configuration

2025-05-20 Thread Piotr Nowojski
Hi, +1 (binding) Best, Piotrek wt., 20 maj 2025 o 20:29 Roman Khachatryan napisał(a): > Hi everyone, > > I'd like to start a vote on FLIP-530: Dynamic job configuration > [1] which has been discussed in this thread [2]. > > The vote will be open for at least 72 hours unless there is an objecti

[jira] [Created] (FLINK-37807) Report s5cmd cancellation in a nicer way

2025-05-15 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37807: -- Summary: Report s5cmd cancellation in a nicer way Key: FLINK-37807 URL: https://issues.apache.org/jira/browse/FLINK-37807 Project: Flink Issue Type

Re: [VOTE] FLIP-525: Model ML_PREDICT, ML_EVALUATE Implementation Design

2025-05-12 Thread Piotr Nowojski
+1 (binding) Best, Piotrek niedz., 11 maj 2025 o 18:14 Hao Li napisał(a): > Hi Dev, > > Thanks all for voting. I'm closing the vote and the result will be posted > in a separate email. > > Thanks, > Hao > > On Fri, May 9, 2025 at 12:43 AM Martijn Visser > wrote: > > > +1 (binding) > > > > On F

Re: [VOTE] FLIP-528: Support EXPLAIN EXECUTE STATEMENT SET Syntax in Flink SQL

2025-05-12 Thread Piotr Nowojski
+1 (binding) Best, Piotrek pon., 12 maj 2025 o 09:44 Gustavo de Morais napisał(a): > +1 (non-binding). > > Thanks Ramin, > Gustavo > > Am Mo., 12. Mai 2025 um 09:42 Uhr schrieb Dawid Wysakowicz < > dwysakow...@apache.org>: > > > +1(binding) > > Best, > > Dawid > > > > On Mon, 12 May 2025 at 09:

Re: [DISCUSS] FLIP-526: Model ML_PREDICT, ML_EVALUATE Table API

2025-05-05 Thread Piotr Nowojski
Hi, sounds like a valuable addition! Best, Piotrek wt., 29 kwi 2025 o 03:57 Shengkai Fang napisał(a): > Hi, Hao. > > +1 for the proposal. > > Best, > Shengkai > > Hao Li 于2025年4月29日周二 07:27写道: > > > Hi All, > > > > I would like to start a discussion about FLIP-526 [1]: Model ML_PREDICT, > > M

Re: [DISCUSS] FLIP-525: Model ML_PREDICT, ML_EVALUATE Implementation Design

2025-05-05 Thread Piotr Nowojski
Hi, sounds like an interesting feature! Best, Piotrek wt., 29 kwi 2025 o 03:52 Shengkai Fang napisał(a): > Hi, Hao. > > Thanks for your proposal about ML related functions. This FLIP will help > others to implement their own model provider. > > +1 for the proposal. > > Best, > Shengkai > > Hao

[jira] [Created] (FLINK-37670) Watermark alignment can deadlock job if there are no more splits to be assigned

2025-04-14 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37670: -- Summary: Watermark alignment can deadlock job if there are no more splits to be assigned Key: FLINK-37670 URL: https://issues.apache.org/jira/browse/FLINK-37670

Re: [VOTE] FLIP-511: Support transaction id pooling in Kafka connector

2025-03-27 Thread Piotr Nowojski
+1 (binding) czw., 27 mar 2025 o 15:23 Gyula Fóra napisał(a): > +1 (binding) > > On Thu, Mar 27, 2025 at 3:18 PM Martijn Visser > wrote: > > > +1 (binding) > > > > On Thu, Mar 27, 2025 at 1:47 PM Yuepeng Pan > wrote: > > > > > +1 (non-binding) > > > > > > Best regards, > > > Yuepeng Pan > > >

[jira] [Created] (FLINK-37570) NullPointerException thrown for split-level watermarks

2025-03-27 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37570: -- Summary: NullPointerException thrown for split-level watermarks Key: FLINK-37570 URL: https://issues.apache.org/jira/browse/FLINK-37570 Project: Flink

[jira] [Created] (FLINK-37561) Report system events

2025-03-26 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37561: -- Summary: Report system events Key: FLINK-37561 URL: https://issues.apache.org/jira/browse/FLINK-37561 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-37560) Document events reporting

2025-03-26 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37560: -- Summary: Document events reporting Key: FLINK-37560 URL: https://issues.apache.org/jira/browse/FLINK-37560 Project: Flink Issue Type: Sub-task

Re: [VOTE] FLIP-513: Split-level watermark metrics

2025-03-17 Thread Piotr Nowojski
+1 (binding) best, Piotrek pon., 17 mar 2025 o 11:06 Efrat Levitan napisał(a): > Hi everyone, > > I'd like to start a vote on FLIP-513: Split-level watermark metrics [1] > which has been discussed in this thread [2]. > The vote will be open for at least 72 hours unless there is an objection or

[jira] [Created] (FLINK-37426) FLIP-481: Introduce Event Reporting

2025-03-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37426: -- Summary: FLIP-481: Introduce Event Reporting Key: FLINK-37426 URL: https://issues.apache.org/jira/browse/FLINK-37426 Project: Flink Issue Type

[jira] [Created] (FLINK-37413) Introduce append-only RowTimeDeduplicateFunction with Async State API

2025-03-03 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37413: -- Summary: Introduce append-only RowTimeDeduplicateFunction with Async State API Key: FLINK-37413 URL: https://issues.apache.org/jira/browse/FLINK-37413 Project

Re: [DISCUSS] Split-level Watermark Metrics

2025-03-03 Thread Piotr Nowojski
Hi Efrat, Thanks for the proposal and +1 from my side for this FLIP. Flink currently has a huge observability gap when it comes to the state of the per-split watermarks, which makes it very difficult for users to understand why any given job generated any particular operator-level watermark. Prov

[jira] [Created] (FLINK-37399) Watermark alignment can prevent backlogged jobs from using all available resources

2025-02-27 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37399: -- Summary: Watermark alignment can prevent backlogged jobs from using all available resources Key: FLINK-37399 URL: https://issues.apache.org/jira/browse/FLINK-37399

[jira] [Created] (FLINK-37280) Inconsistent content of TableConfig in planner when mini batch is enabled

2025-02-07 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37280: -- Summary: Inconsistent content of TableConfig in planner when mini batch is enabled Key: FLINK-37280 URL: https://issues.apache.org/jira/browse/FLINK-37280

[jira] [Created] (FLINK-37256) Firing timers can block recovery process

2025-02-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37256: -- Summary: Firing timers can block recovery process Key: FLINK-37256 URL: https://issues.apache.org/jira/browse/FLINK-37256 Project: Flink Issue Type

[jira] [Created] (FLINK-37213) Improve performance of unbounded OVER aggregations

2025-01-23 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37213: -- Summary: Improve performance of unbounded OVER aggregations Key: FLINK-37213 URL: https://issues.apache.org/jira/browse/FLINK-37213 Project: Flink Issue

[jira] [Created] (FLINK-37005) Make StreamExecDeduplicate ouput insert only where possible

2025-01-03 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-37005: -- Summary: Make StreamExecDeduplicate ouput insert only where possible Key: FLINK-37005 URL: https://issues.apache.org/jira/browse/FLINK-37005 Project: Flink

[RESULT][VOTE] FLIP-484: Add custom metric variables to operators

2024-12-16 Thread Piotr Nowojski
Hi! FLIP-484 [1] has been accepted [2]. Votes in favour: - Roman Khachatryan (binding) - Rui Fan (binding) - Piotr Nowojski (binding) - David Radley (non-binding) There were no votes against. Best, Piotrek [1] https://cwiki.apache.org/confluence/x/44yMEw [2] https://lists.apache.org/thread

Re: [VOTE] FLIP-484: Add custom metric variables to operators

2024-12-16 Thread Piotr Nowojski
Thank you for voting. Voting is closed. wt., 10 gru 2024 o 14:34 Piotr Nowojski napisał(a): > > It sounds like we are using this mechanism to cover all three cases. If > there is an appetite to do the first 2 in a more > > metadata driven way then I guess they would appear in the

[RESULT][VOTE] FLIP-483: Add support for children Spans

2024-12-16 Thread Piotr Nowojski
Hi! FLIP-483 [1] has been accepted [2]. Votes in favour: - Roman Khachatryan (binding) - Rui Fan (binding) - Piotr Nowojski (binding) There were no votes against. Best, Piotrek [1] https://cwiki.apache.org/confluence/x/4IyMEw [2] https://lists.apache.org/thread

Re: [VOTE] FLIP-483: Add support for children Spans

2024-12-16 Thread Piotr Nowojski
Thank you for voting. Voting is closed. pt., 6 gru 2024 o 11:16 Rui Fan <1996fan...@gmail.com> napisał(a): > +1 (binding) > > Best, > Rui > > On Fri, Dec 6, 2024 at 5:33 PM Roman Khachatryan wrote: > > > +1 (binding) > > > > Regards, > > Roma

[RESULT][VOTE] FLIP-482: Add OpenTelemetryEventReporter

2024-12-16 Thread Piotr Nowojski
Hi! FLIP-482 [1] has been accepted [2]. Votes in favour: - Roman Khachatryan (binding) - Rui Fan (binding) - Piotr Nowojski (binding) There were no votes against. Best, Piotrek [1] https://cwiki.apache.org/confluence/x/3oyMEw [2] https://lists.apache.org/thread

Re: [VOTE] FLIP-482: Add OpenTelemetryEventReporter

2024-12-16 Thread Piotr Nowojski
Thank you for voting. Voting is closed. pt., 6 gru 2024 o 11:14 Rui Fan <1996fan...@gmail.com> napisał(a): > +1 (binding) > > Best, > Rui > > On Fri, Dec 6, 2024 at 5:32 PM Roman Khachatryan wrote: > > > +1 (binding) > > > > Regards, > > Roma

[RESULT][VOTE] FLIP-481: Introduce Event Reporting

2024-12-16 Thread Piotr Nowojski
Hi! FLIP-481 [1] has been accepted [2]. Votes in favour: - Roman Khachatryan (binding) - Rui Fan (binding) - Piotr Nowojski (binding) There were no votes against. Best, Piotrek [1] https://cwiki.apache.org/confluence/x/3IyMEw [2] https://lists.apache.org/thread

Re: [VOTE] FLIP-481: Introduce Event Reporting

2024-12-16 Thread Piotr Nowojski
Thank you for voting. Voting is closed. pt., 6 gru 2024 o 11:18 Rui Fan <1996fan...@gmail.com> napisał(a): > +1 (binding) > > Best, > Rui > > On Fri, Dec 6, 2024 at 5:33 PM Roman Khachatryan wrote: > > > +1 (binding) > > > > Regards, > > Roma

Re: [VOTE] FLIP-484: Add custom metric variables to operators

2024-12-10 Thread Piotr Nowojski
> > > Are these like the User variables defined for the connectors. What happen > if there is a name clash with the user variable? > > > > I am in favour of this Flip , but would like the above clarifications > please, > > > > Kind regards, David. > > &g

Re: [VOTE] FLIP-484: Add custom metric variables to operators

2024-12-06 Thread Piotr Nowojski
the table name as the metric scope – > could we not do this automatically in the code without any additional need > to add code? > > Kind regards, David. > > > From: Yuepeng Pan > Date: Friday, 6 December 2024 at 10:27 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re:

[VOTE] FLIP-484: Add custom metric variables to operators

2024-12-06 Thread Piotr Nowojski
Hi all! I would like to open the vote for FLIP-484 [1]. It has been discussed here [2]. The vote will remain open for at least 72 hours (excluding the weekend). Best, Piotrek [1] https://cwiki.apache.org/confluence/x/44yMEw [2] https://lists.apache.org/thread/1g1n2l8bc27zvw88ljw4l2gqzx6rg5zy

[VOTE] FLIP-483: Add support for children Spans

2024-12-06 Thread Piotr Nowojski
Hi all! I would like to open the vote for FLIP-483 [1]. It has been discussed here [2]. The vote will remain open for at least 72 hours (excluding the weekend). Best, Piotrek [1] https://cwiki.apache.org/confluence/x/4IyMEw [2] https://lists.apache.org/thread/hdrh4nx0zhjzb5q5gkp5w1cqkzb4v6po

[VOTE] FLIP-482: Add OpenTelemetryEventReporter

2024-12-06 Thread Piotr Nowojski
Hi all! I would like to open the vote for FLIP-482 [1]. It has been discussed here [2]. The vote will remain open for at least 72 hours (excluding the weekend). Best, Piotrek [1] https://cwiki.apache.org/confluence/x/3oyMEw [2] https://lists.apache.org/thread/db1v1hjcysxo88cjb6jq5vnrz4s3cdrm

[VOTE] FLIP-481: Introduce Event Reporting

2024-12-06 Thread Piotr Nowojski
Hi all! I would like to open the vote for FLIP-481 [1]. It has been discussed here [2]. The vote will remain open for at least 72 hours (excluding the weekend). Best, Piotrek [1] https://cwiki.apache.org/confluence/x/3IyMEw [2] https://lists.apache.org/thread/0y2yg1y27sv86rkxg0vpj18h5hgy2ppg

Re: [DISCUSS] FLIP-482: Add OpenTelemetryEventReporter

2024-12-06 Thread Piotr Nowojski
As there are no comments, I will open a voting thread later today. Best, Piotrek czw., 14 lis 2024 o 13:38 Roman Khachatryan napisał(a): > Hi Piotr, > > Adding OTel implementation makes sense, +1 for the proposal. > > Thanks > > Regards, > Roman > > > On

Re: FLIP-484: Add custom metric variables to operators

2024-12-06 Thread Piotr Nowojski
I hope my previous message has clarified the questions. If there are no further comments, I will open a voting thread later today. Best, Piotrek wt., 3 gru 2024 o 12:18 Piotr Nowojski napisał(a): > Hi Romand and Hongshun, > > > 1. The scope of the variables added - is it o

Re: [DISCUSS] FLIP-481: Introduce Event Reporting

2024-12-06 Thread Piotr Nowojski
existing traces/`SpanBuilder`. > > I'm not sure what you would propose in this context? Make those two > places > > inconsistent? > > Good point, I think consistency is more important. > > Regards, > Roman > > > On Tue, Dec 3, 2024 at 12:07 PM Piotr Nowojsk

Re: [DISCUSS] FLIP-483: Add support for children Spans

2024-12-06 Thread Piotr Nowojski
e. > > Regards, > Roman > > > On Tue, Dec 3, 2024 at 12:30 PM Piotr Nowojski > wrote: > > > Hi Roman! > > > > > 1. Why do we give up on the idea of reporting child spans independently > > > from the parent? I couldn't find much details in th

Re: [DISCUSS] FLIP-483: Add support for children Spans

2024-12-03 Thread Piotr Nowojski
? I couldn't find much details in the Rejected Alternatives > section > > 2. If at some point we come up with a way to address (1), then having a > reference from child to parent would be more flexible? And probably not in > the form of object reference, but just as a (Strin

Re: FLIP-484: Add custom metric variables to operators

2024-12-03 Thread Piotr Nowojski
Thu, Nov 14, 2024 at 8:36 PM Roman Khachatryan > wrote: > > > Hi Piotr, thanks for the proposal, > > > > Can you please clarify > > 1. The scope of the variables added - is it only the last transformation? > > Do I understand correctly, that chaining does

Re: [DISCUSS] FLIP-481: Introduce Event Reporting

2024-12-03 Thread Piotr Nowojski
mple scope="job-lifecycle" or > scope="task-manager". > It also adds a barrier to rename or move classes. > So I'd remove the mention of Class. > > 3. Interface Event - Body > Can you explain the purpose of this property? > In my opinion, most informatio

[jira] [Created] (FLINK-36764) Add checkpoint type to checkpoint trace

2024-11-21 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36764: -- Summary: Add checkpoint type to checkpoint trace Key: FLINK-36764 URL: https://issues.apache.org/jira/browse/FLINK-36764 Project: Flink Issue Type

[DISCUSS] FLIP-481: Introduce Event Reporting

2024-11-07 Thread Piotr Nowojski
Hi all! I would like to open up for the discussion a new FLIP [1] Motivation Currently, Flink observability has support for Metrics and Traces. We suggest enhancing the observability capabilities by adding support for Events. This can be used to track the most important events that happen in Fli

FLIP-484: Add custom metric variables to operators

2024-11-07 Thread Piotr Nowojski
Hi all! I would like to open up for discussion a new FLIP-484 [1]. Motivation When running a Flink job that reads data from multiple sources and writes to multiple sinks, it would be helpful to group metrics from each source/sink instance together, for example based on the underlying name of the

[DISCUSS] FLIP-484: Add custom metric variables to operators

2024-11-07 Thread Piotr Nowojski
czw., 7 lis 2024 o 14:44 Piotr Nowojski napisał(a): > Hi all! > > I would like to open up for discussion a new FLIP-484 [1]. > > Motivation > When running a Flink job that reads data from multiple sources and writes > to multiple sinks, it would be helpful to group metri

[DISCUSS] FLIP-482: Add OpenTelemetryEventReporter

2024-11-07 Thread Piotr Nowojski
Hi all! I would like to open up for discussion a new FLIP-482 [1]. Motivation FLIP-481 [2] is adding the EventReporter interface. However with FLIP-481 [2] alone, Log4jTraceReporter would be the only available implementation of the EventReporter interface, which is not very helpful. We propose t

[DISCUSS] FLIP-483: Add support for children Spans

2024-11-07 Thread Piotr Nowojski
Hi all! I would like to open up for discussion a new FLIP-483 [1]. Motivation FLIP-384 [2] added trace/span reporting capability to Flink, which has been used in a couple of places, like reporting checkpointing and recovery processes. With flat/childless structure of spans it is difficult to acc

[jira] [Created] (FLINK-36624) Log JobID in SourceCoordinator

2024-10-29 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36624: -- Summary: Log JobID in SourceCoordinator Key: FLINK-36624 URL: https://issues.apache.org/jira/browse/FLINK-36624 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-36621) Build failure: StatefulSink not found

2024-10-29 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36621: -- Summary: Build failure: StatefulSink not found Key: FLINK-36621 URL: https://issues.apache.org/jira/browse/FLINK-36621 Project: Flink Issue Type

[jira] [Created] (FLINK-36556) Allow to configure starting buffer size when using buffer debloating

2024-10-16 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36556: -- Summary: Allow to configure starting buffer size when using buffer debloating Key: FLINK-36556 URL: https://issues.apache.org/jira/browse/FLINK-36556 Project

[jira] [Created] (FLINK-36555) Guarantee debloated buffer size grows even with very small alpha

2024-10-16 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36555: -- Summary: Guarantee debloated buffer size grows even with very small alpha Key: FLINK-36555 URL: https://issues.apache.org/jira/browse/FLINK-36555 Project: Flink

[jira] [Created] (FLINK-36416) Enable splittable timers for temporal join, temporal sort and windowed aggregation in SQL/Table API

2024-10-01 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36416: -- Summary: Enable splittable timers for temporal join, temporal sort and windowed aggregation in SQL/Table API Key: FLINK-36416 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-36260) numBytesInLocal and numBuffersInLocal being reported as remote

2024-09-11 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36260: -- Summary: numBytesInLocal and numBuffersInLocal being reported as remote Key: FLINK-36260 URL: https://issues.apache.org/jira/browse/FLINK-36260 Project: Flink

[jira] [Created] (FLINK-36108) Wait for state download on cancellation to enforce cleanup

2024-08-20 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-36108: -- Summary: Wait for state download on cancellation to enforce cleanup Key: FLINK-36108 URL: https://issues.apache.org/jira/browse/FLINK-36108 Project: Flink

Re: [DISCUSS] FLIP-455: Declare async state processing and checkpoint the in-flight requests

2024-08-06 Thread Piotr Nowojski
ack); > } > } > ``` > > Apart from that, there will be another internal API support processing > something under a keyed context: > ``` > asyncProcessWithKey(key, () -> { // some logic}); > // the function will be processed in mailbox; > // equivalent to > //

Re: [DISCUSS] FLIP-455: Declare async state processing and checkpoint the in-flight requests

2024-08-06 Thread Piotr Nowojski
Hi Zakelly, Thanks for your responses! > IIUC, the AsyncWaitOperator and AsyncScalarFunction are all non-keyed, so > we don't need to worry too much Because the remote state backend/async state accesses are only supported on the keyed state? > Furthermore, as I mentioned, we currently > do not

[RESULT][VOTE] FLIP-471: Fixing watermark idleness timeout accounting

2024-08-05 Thread Piotr Nowojski
Hi all! The FLIP-471 [1] has been accepted in the voting thread [2]. There were 6 binding votes in favour: - Zakelly Lan - Stefan Richter - Timo Walther - Rui Fan - Roman Khachatryan - Piotr Nowojski zero against. Best, Piotrek [1] https://cwiki.apache.org/confluence/x/oQvOEg [2] https

Re: [VOTE] FLIP-471: Fixing watermark idleness timeout accounting

2024-08-05 Thread Piotr Nowojski
51, Stefan Richter wrote: > > > > > > > > +1 (binding) > > > > > > > > Best, > > > > Stefan > > > > > > > > > > > > > > > >> On 31. Jul 2024, at 04:56, Zakelly Lan > wrote: > > &g

Re: [DISCUSS] FLIP-455: Declare async state processing and checkpoint the in-flight requests

2024-08-05 Thread Piotr Nowojski
Hi Zakelly, Thanks for driving this! I also have a couple of questions. But overall the proposal LGTM. >> 2. Besides `processElement`, there are other methods that access state, >> such as >> `snapshotState`. Are these methods also not applicable under this FLIP? > > Well the async state APIs are

[VOTE] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-30 Thread Piotr Nowojski
Hi all! I would like to open the vote for FLIP-471 [1]. It has been discussed here [2]. The vote will remain open for at least 72 hours. Best, Piotrek [1] https://cwiki.apache.org/confluence/x/oQvOEg [2] https://lists.apache.org/thread/byj1l2236rfx3mcl3v4374rcbkq4rf85

Re: [DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-30 Thread Piotr Nowojski
ng `Clock`, as forcing the > relative time and absolute time to run at the same speed limits the use of > this clock. Anyway, I have no strong feelings. I understand your concern > and introducing a new interface is also fine with me. > > > Best, > Zakelly > > On Mon, Jul 29

Re: [DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-29 Thread Piotr Nowojski
to me. Also, at least > > currently, idle time at the watermark > > generator level is not calculated accurately. Exposing an approximate > value > > via API might make some > > false impression that it's accurate? > I understand your concern and we can k

Re: [DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-26 Thread Piotr Nowojski
to pause is an implementation-related thing (e.g. > org.apache.flink.util.clock.ManualClock). WDYT? > > > Best, > Zakelly > > > On Thu, Jul 25, 2024 at 9:02 PM Piotr Nowojski > wrote: > > > Hi Arvid, > > > > > 1. `RelativeClock` looks like a super-interface of > >

Re: [DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-25 Thread Piotr Nowojski
Wed, Jul 24, 2024 at 3:01 PM Martijn Visser > wrote: > > > Hi Piotr, > > > > We've talked offline about this proposal and I think it would be > beneficial > > for users to get this fixed. +1 overall, and thanks for writing it down. > > > > Best regar

[DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-23 Thread Piotr Nowojski
Hi All, A bit unusual FLIP [1], as this is a bug fix for a problem that I have recently discovered [2]. However I think FLIP is required, as properly fixing the issue requires changes to the public API. As this is a bug fix, I would propose to back-port this change to previous releases (1.19 and

[jira] [Created] (FLINK-35886) Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked

2024-07-23 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35886: -- Summary: Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked Key: FLINK-35886 URL: https://issues.apache.org/jira/browse/FLINK-35886

[jira] [Created] (FLINK-35821) ResumeCheckpointManuallyITCase failed with File X does not exist or the user running Flink C has insufficient permissions to access it

2024-07-11 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35821: -- Summary: ResumeCheckpointManuallyITCase failed with File X does not exist or the user running Flink C has insufficient permissions to access it Key: FLINK-35821 URL: https

[jira] [Created] (FLINK-35773) Document s5cmd

2024-07-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35773: -- Summary: Document s5cmd Key: FLINK-35773 URL: https://issues.apache.org/jira/browse/FLINK-35773 Project: Flink Issue Type: Sub-task Reporter

[jira] [Created] (FLINK-35772) Deprecate/remove DuplicatingFileSystem

2024-07-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35772: -- Summary: Deprecate/remove DuplicatingFileSystem Key: FLINK-35772 URL: https://issues.apache.org/jira/browse/FLINK-35772 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-35771) Limit s5cmd resource usage

2024-07-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35771: -- Summary: Limit s5cmd resource usage Key: FLINK-35771 URL: https://issues.apache.org/jira/browse/FLINK-35771 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35770) Interrupt s5cmd call on cancellation

2024-07-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35770: -- Summary: Interrupt s5cmd call on cancellation Key: FLINK-35770 URL: https://issues.apache.org/jira/browse/FLINK-35770 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-35768) Use native file copy in RocksDBStateDownloader

2024-07-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35768: -- Summary: Use native file copy in RocksDBStateDownloader Key: FLINK-35768 URL: https://issues.apache.org/jira/browse/FLINK-35768 Project: Flink Issue

[jira] [Created] (FLINK-35767) Provide native file copy support for S3 using s5cmd

2024-07-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35767: -- Summary: Provide native file copy support for S3 using s5cmd Key: FLINK-35767 URL: https://issues.apache.org/jira/browse/FLINK-35767 Project: Flink

[jira] [Created] (FLINK-35739) FLIP-444: Native file copy support

2024-07-02 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35739: -- Summary: FLIP-444: Native file copy support Key: FLINK-35739 URL: https://issues.apache.org/jira/browse/FLINK-35739 Project: Flink Issue Type: New

[RESULT][VOTE] FLIP-444: Native file copy support

2024-07-02 Thread Piotr Nowojski
I would like to announce that FLIP-444 [1] has been accepted. There were 13 votes in favour: - Roman (binding) - Zakelly (binding) - Hong (binding) - Stefan (binding) - Rui Fan (binding) - Marijn (binding) - Piotr Nowojski (binding) - Yuepeng Pan - Keith - Ahmed Hamdy - Yue Ma - Fiefan Wang

Re: [VOTE] FLIP-444: Native file copy support

2024-07-02 Thread Piotr Nowojski
t; > > > > > > +1 (non binding) > > > > > > > > > > —— > > > > Best regards, > > > > Feifan Wang > > > > > > > > > > At 2024-06-25 16:58:22, "Piotr Nowojski" pnowoj...@apache.org wrote: > > > >

Re: [VOTE] FLIP-444: Native file copy support

2024-06-26 Thread Piotr Nowojski
an be resumed. Best, Piotrek wt., 25 cze 2024 o 18:57 Piotr Nowojski napisał(a): > Ops, I must have forgotten to update the FLIP as we discussed. I will fix > it tomorrow and the vote period will be extended. > > Best, > Piotrek > > wt., 25 cze 2024 o 13:56 Zakelly Lan n

Re: [VOTE] FLIP-444: Native file copy support

2024-06-25 Thread Piotr Nowojski
ileSystem` in the FLIP, shall we do that as mentioned in the > discussion thread? > > > Best, > Zakelly > > On Tue, Jun 25, 2024 at 4:58 PM Piotr Nowojski > wrote: > > > Hi all, > > > > I would like to start a vote for the FLIP-444 [1]. The discussion

[VOTE] FLIP-444: Native file copy support

2024-06-25 Thread Piotr Nowojski
Hi all, I would like to start a vote for the FLIP-444 [1]. The discussion thread is here [2]. The vote will be open for at least 72. Best, Piotrek [1] https://cwiki.apache.org/confluence/x/rAn9EQ [2] https://lists.apache.org/thread/lkwmyjt2bnmvgx4qpp82rldwmtd4516c

Re: [DISCUSS] FLIP-444: Native file copy support

2024-06-17 Thread Piotr Nowojski
jira/browse/FLINK-35501 pt., 17 maj 2024 o 15:27 napisał(a): > Perfectly agree with all your considerations. > Wee said. > > Thank you! > On May 16, 2024 at 10:53 +0200, Piotr Nowojski , > wrote: > > Hi Lorenzo, > > > > > • concerns about memory and CPU used out o

[jira] [Created] (FLINK-35528) Skip execution of interruptible mails when yielding

2024-06-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35528: -- Summary: Skip execution of interruptible mails when yielding Key: FLINK-35528 URL: https://issues.apache.org/jira/browse/FLINK-35528 Project: Flink

[ANNOUNCE] New Apache Flink PMC Member - Fan Rui

2024-06-05 Thread Piotr Nowojski
Hi everyone, On behalf of the PMC, I'm very happy to announce another new Apache Flink PMC Member - Fan Rui. Rui has been active in the community since August 2019. During this time he has contributed a lot of new features. Among others: - Decoupling Autoscaler from Kubernetes Operator, and sup

[jira] [Created] (FLINK-35518) CI Bot doesn't run on PRs

2024-06-04 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35518: -- Summary: CI Bot doesn't run on PRs Key: FLINK-35518 URL: https://issues.apache.org/jira/browse/FLINK-35518 Project: Flink Issue Type

[RESULT][VOTE] FLIP-443: Interruptible watermark processing

2024-05-28 Thread Piotr Nowojski
Hi all! I'm happy to announce that FLIP-443 [1] has been accepted [2]. There were 7 votes in favour, all binding: - Zakelly - Yanfei Lei - Rui Fan - Wejie Guo - Martijn Visser - Stefan Richter - Piotr Nowojski [1] https://cwiki.apache.org/confluence/x/qgn9EQ [2] https://lists.apache.org/t

Re: [VOTE] FLIP-443: Interruptible watermark processing

2024-05-28 Thread Piotr Nowojski
t; > >>>> Thanks for driving this! > >>>> > >>>> +1 (binding) > >>>> > >>>> Best, > >>>> Yanfei > >>>> > >>>> Zakelly Lan 于2024年5月24日周五 10:13写道: > >>>> > >&g

[VOTE] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Piotr Nowojski
Hi all, After reaching what looks like a consensus in the discussion thread [1], I would like to put FLIP-443 [2] to the vote. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1] https://lists.apache.org/thread/flxm7rphvfgqdn2gq2z0bb7kl007olpz [2]

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Piotr Nowojski
, for the sake of > completeness, I could accept the doubling method plan. > > > Thanks & Best, > Zakelly > > On Wed, May 22, 2024 at 5:05 PM Piotr Nowojski > wrote: > > > Hi Zakelly, > > > > > I suggest not doubling the existing methods. Only provi

[jira] [Created] (FLINK-35420) WordCountMapredITCase fails to compile in IntelliJ

2024-05-22 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35420: -- Summary: WordCountMapredITCase fails to compile in IntelliJ Key: FLINK-35420 URL: https://issues.apache.org/jira/browse/FLINK-35420 Project: Flink Issue

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-22 Thread Piotr Nowojski
t, > > Object... descriptionArgs); > > > WDYT? > > > Best, > Zakelly > > > On Wed, May 22, 2024 at 12:53 AM Piotr Nowojski > wrote: > > > Hi Zakelly and others, > > > > > 1. I'd suggest also providing `isInterruptable()` in `Mail`,

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-21 Thread Piotr Nowojski
ady have fireProcessingTimers benchmark. > > If so, we can follow it after this FLIP is merged. > > +1 for this FLIP. > > Best, > Rui > > On Thu, May 16, 2024 at 5:13 PM Piotr Nowojski > wrote: > > > Hi Zakelly, > > > > > I'm sug

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-16 Thread Piotr Nowojski
state access. > > > > > > [1] > > > > > https://github.com/apache/flink/blob/1904b215e36e4fd48e48ece7ffdf2f1470653130/flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java#L305 > > > > Best, > > Zakelly >

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-16 Thread Piotr Nowojski
t; uploading fails, the job fails over, so actually using such interface > > > extension would be out of scope of this FLIP. In > > > that case, maybe if this could be extended in the future without > breaking > > > compatibility we could leave it as a > > >

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-10 Thread Piotr Nowojski
valent > > of > > > > > `DuplicatingFileSystem` - quickly duplicate/hard link remote path > > > > > - `canFastCopy(local, remote)` returns true - FS can natively > upload > > > > local > > > > > file to a remote location > >

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-06 Thread Piotr Nowojski
+1 (binding) Piotrek pon., 6 maj 2024 o 12:35 Roman Khachatryan napisał(a): > +1 (binding) > > Regards, > Roman > > > On Mon, May 6, 2024 at 11:56 AM gongzhongqiang > wrote: > > > +1 (non-binding) > > > > Best, > > Zhongqiang Gong > > > > yue ma 于2024年5月6日周一 10:54写道: > > > > > Hi everyone, >

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-06 Thread Piotr Nowojski
; > > > Some questions from my side: > > > > > > > > > In order to configure s5cmd Flink’s user would need > > > > > to specify path to the s5cmd binary. > > > > > > > > Could you please also add the configuration property > >

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-06 Thread Piotr Nowojski
道: > > > > > > > > > Thanks for the improvement proposal, I’m +1 for the change! > > > > > > Best, > > > Stefan > > > > > > > > > > > > > On 30. Apr 2024, at 15:23, Roman Khachatryan > wrote: > > >

  1   2   3   4   5   6   7   8   9   10   >