[jira] [Commented] (FLINK-33656) If json.ignose-parse-errors =true is configured and Array parsing errors occur, other columns will be empty

[Martijn Visser (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789955#comment-17789955
 ] 

Martijn Visser commented on FLINK-33656:


[~duke13] Can you please provide a reproducer? I don't see how we could 
reproduce this based off the images

> If json.ignose-parse-errors =true is configured and Array parsing errors 
> occur, other columns will be empty
> ---
>
> Key: FLINK-33656
> URL: https://issues.apache.org/jira/browse/FLINK-33656
> Project: Flink
>  Issue Type: Bug
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.16.0, 1.17.0, 1.16.1, 1.16.2, 1.18.0, 1.17.1
>Reporter: duke
>Priority: Critical
> Attachments: image-2023-11-27-13-58-22-513.png, 
> image-2023-11-27-13-59-42-066.png, image-2023-11-27-14-00-04-672.png, 
> image-2023-11-27-14-00-41-176.png, image-2023-11-27-14-01-12-187.png, 
> image-2023-11-27-14-02-01-252.png, image-2023-11-27-14-02-30-666.png, 
> image-2023-11-27-14-02-52-065.png, image-2023-11-27-14-03-10-885.png
>
>
> If json.ignore-parse-errors is set to true and Array parsing errors occur, 
> the fields following array are resolved as empty in the complete json message
> 1. !image-2023-11-27-13-59-42-066.png!
> !image-2023-11-27-14-00-04-672.png!
>  
> 2. !image-2023-11-27-14-00-41-176.png!
> !image-2023-11-27-14-01-12-187.png!
> 3. !image-2023-11-27-14-02-52-065.png!
> !image-2023-11-27-14-03-10-885.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-33443) Make the test "testWriteComplexType" stable

[Martijn Visser (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789957#comment-17789957
 ] 

Martijn Visser commented on FLINK-33443:


[~hannahchi] Looking at the number of open tickets with actually reported test 
instabilities, I don't see why adding more tickets that don't have a direct 
reproducer is valuable for the project. It just creates noise for the 
maintainers who are already trying their best to keep Flink in a good shape

> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
>  * The following command can be run to execute the test with the 
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive 
> edu.illinois:nondex-maven-plugin:2.1.1:nondex 
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType 
> {code}
>  * The following error will be encountered:
> {code:java}
> [ERROR] Failures: 
> [ERROR]   HiveRunnerITCase.testWriteComplexType:166 
> expected: "[1,2,3]{1:"a",2:"b"}   {"f1":3,"f2":"c"}"
>  but was: "[1,2,3]{2:"b",1:"a"}   {"f1":3,"f2":"c"}"
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of 
> elements received in the _result_ variable will be consistent. There are 
> currently two versions of query output that can be stored in _result._
>  # The actual order that is expected where the output of the map attribute is 
> \{1: "a", 2: "b"}.
>  # The other order is the one shown in the error extract above where the 
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
>  * The fix that I can suggest/have ready to raise a PR for is introducing 
> another assertion on the second variant of the query output.
>  * By asserting on whether the contents in _result_ are in one of the two 
> orders, we can ascertain that the expected attributes with their contents are 
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other 
> potential fix is possible for this test.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33657) Insert message in top n without row number didn't consider it's number and may not correct

[zlzhang0122 (Jira)]

zlzhang0122 created FLINK-33657:
---

 Summary: Insert message in top n without row number didn't 
consider it's number and may not correct
 Key: FLINK-33657
 URL: https://issues.apache.org/jira/browse/FLINK-33657
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Affects Versions: 1.17.1, 1.16.2
Reporter: zlzhang0122


The new insert message in top n without row number didn't consider it's order 
and just 
collectInsert to the next operator, this may not correct when the next operator 
collect all the top n records and aggregate it.
 
For example：
create table user_info(

user_id int,

item_id int,
app string,

dt timestamp
) whith(
'connector'='kafka',

...
);
create table redis_sink (
redis_key string,
hash_key string,
hash_value string
)
with (
'connector' = 'redis',
'command' = 'hmset'
'nodes' = 'xxx',
'additional-ttl' = 'xx'
);
create view base_lastn

as select * from(
select user_id,item_id,app,dt,row_number() over(partition by item_id, app order 
by dt desc) as rn from user_action
)t where rn<=5;

insert into redis_sink

select
concat("id", item_id) as redis_key, app as hash_key,user_id as hash_value from 
base_lastn where rn=1;
insert into redis_sink

select
concat("list_", item_id) as redis_key, app as hash_key,listagg(user_id,",") as 
hash_value from base_lastn where group by item_id, app;
 

There will be a scene that the value in the top 1 will not appear in the first 
or last value of the top 5.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33657) Insert message in top n without row number didn't consider it's number and may not correct

[zlzhang0122 (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zlzhang0122 updated FLINK-33657:

Description: 
The new insert message in top n without row number didn't consider it's order 
and just 
collectInsert() to the next operator, this may incorrect when the next operator 
collect all the top n records and aggregate it.
 
For example：
create table user_info(

user_id int,

item_id int,
app string,

dt timestamp
) whith(
'connector'='kafka',

...
);
create table redis_sink (
redis_key string,
hash_key string,
hash_value string
)
with (
'connector' = 'redis',
'command' = 'hmset'
'nodes' = 'xxx',
'additional-ttl' = 'xx'
);
create view base_lastn

as select * from(
select user_id,item_id,app,dt,row_number() over(partition by item_id, app order 
by dt desc) as rn from user_action
)t where rn<=5;

insert into redis_sink

select
concat("id", item_id) as redis_key, app as hash_key,user_id as hash_value from 
base_lastn where rn=1;
insert into redis_sink

select
concat("list_", item_id) as redis_key, app as hash_key,listagg(user_id,",") as 
hash_value from base_lastn where group by item_id, app;
 

There will be a scene that the value in the top 1 will not appear in the first 
or last value of the top 5.

  was:
The new insert message in top n without row number didn't consider it's order 
and just 
collectInsert() to the next operator, this may not correct when the next 
operator collect all the top n records and aggregate it.
 
For example：
create table user_info(

user_id int,

item_id int,
app string,

dt timestamp
) whith(
'connector'='kafka',

...
);
create table redis_sink (
redis_key string,
hash_key string,
hash_value string
)
with (
'connector' = 'redis',
'command' = 'hmset'
'nodes' = 'xxx',
'additional-ttl' = 'xx'
);
create view base_lastn

as select * from(
select user_id,item_id,app,dt,row_number() over(partition by item_id, app order 
by dt desc) as rn from user_action
)t where rn<=5;

insert into redis_sink

select
concat("id", item_id) as redis_key, app as hash_key,user_id as hash_value from 
base_lastn where rn=1;
insert into redis_sink

select
concat("list_", item_id) as redis_key, app as hash_key,listagg(user_id,",") as 
hash_value from base_lastn where group by item_id, app;
 

There will be a scene that the value in the top 1 will not appear in the first 
or last value of the top 5.


> Insert message in top n without row number didn't consider it's number and 
> may not correct
> --
>
> Key: FLINK-33657
> URL: https://issues.apache.org/jira/browse/FLINK-33657
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / Runtime
>Affects Versions: 1.16.2, 1.17.1
>Reporter: zlzhang0122
>Priority: Major
>
> The new insert message in top n without row number didn't consider it's order 
> and just 
> collectInsert() to the next operator, this may incorrect when the next 
> operator collect all the top n records and aggregate it.
>  
> For example：
> create table user_info(
> user_id int,
> item_id int,
> app string,
> dt timestamp
> ) whith(
> 'connector'='kafka',
> ...
> );
> create table redis_sink (
> redis_key string,
> hash_key string,
> hash_value string
> )
> with (
> 'connector' = 'redis',
> 'command' = 'hmset'
> 'nodes' = 'xxx',
> 'additional-ttl' = 'xx'
> );
> create view base_lastn
> as select * from(
> select user_id,item_id,app,dt,row_number() over(partition by item_id, app 
> order by dt desc) as rn from user_action
> )t where rn<=5;
> insert into redis_sink
> select
> concat("id", item_id) as redis_key, app as hash_key,user_id as hash_value 
> from base_lastn where rn=1;
> insert into redis_sink
> select
> concat("list_", item_id) as redis_key, app as hash_key,listagg(user_id,",") 
> as hash_value from base_lastn where group by item_id, app;
>  
> There will be a scene that the value in the top 1 will not appear in the 
> first or last value of the top 5.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33657) Insert message in top n without row number didn't consider it's number and may not correct

[zlzhang0122 (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zlzhang0122 updated FLINK-33657:

Description: 
The new insert message in top n without row number didn't consider it's order 
and just 
collectInsert() to the next operator, this may not correct when the next 
operator collect all the top n records and aggregate it.
 
For example：
create table user_info(

user_id int,

item_id int,
app string,

dt timestamp
) whith(
'connector'='kafka',

...
);
create table redis_sink (
redis_key string,
hash_key string,
hash_value string
)
with (
'connector' = 'redis',
'command' = 'hmset'
'nodes' = 'xxx',
'additional-ttl' = 'xx'
);
create view base_lastn

as select * from(
select user_id,item_id,app,dt,row_number() over(partition by item_id, app order 
by dt desc) as rn from user_action
)t where rn<=5;

insert into redis_sink

select
concat("id", item_id) as redis_key, app as hash_key,user_id as hash_value from 
base_lastn where rn=1;
insert into redis_sink

select
concat("list_", item_id) as redis_key, app as hash_key,listagg(user_id,",") as 
hash_value from base_lastn where group by item_id, app;
 

There will be a scene that the value in the top 1 will not appear in the first 
or last value of the top 5.

  was:
The new insert message in top n without row number didn't consider it's order 
and just 
collectInsert to the next operator, this may not correct when the next operator 
collect all the top n records and aggregate it.
 
For example：
create table user_info(

user_id int,

item_id int,
app string,

dt timestamp
) whith(
'connector'='kafka',

...
);
create table redis_sink (
redis_key string,
hash_key string,
hash_value string
)
with (
'connector' = 'redis',
'command' = 'hmset'
'nodes' = 'xxx',
'additional-ttl' = 'xx'
);
create view base_lastn

as select * from(
select user_id,item_id,app,dt,row_number() over(partition by item_id, app order 
by dt desc) as rn from user_action
)t where rn<=5;

insert into redis_sink

select
concat("id", item_id) as redis_key, app as hash_key,user_id as hash_value from 
base_lastn where rn=1;
insert into redis_sink

select
concat("list_", item_id) as redis_key, app as hash_key,listagg(user_id,",") as 
hash_value from base_lastn where group by item_id, app;
 

There will be a scene that the value in the top 1 will not appear in the first 
or last value of the top 5.


> Insert message in top n without row number didn't consider it's number and 
> may not correct
> --
>
> Key: FLINK-33657
> URL: https://issues.apache.org/jira/browse/FLINK-33657
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / Runtime
>Affects Versions: 1.16.2, 1.17.1
>Reporter: zlzhang0122
>Priority: Major
>
> The new insert message in top n without row number didn't consider it's order 
> and just 
> collectInsert() to the next operator, this may not correct when the next 
> operator collect all the top n records and aggregate it.
>  
> For example：
> create table user_info(
> user_id int,
> item_id int,
> app string,
> dt timestamp
> ) whith(
> 'connector'='kafka',
> ...
> );
> create table redis_sink (
> redis_key string,
> hash_key string,
> hash_value string
> )
> with (
> 'connector' = 'redis',
> 'command' = 'hmset'
> 'nodes' = 'xxx',
> 'additional-ttl' = 'xx'
> );
> create view base_lastn
> as select * from(
> select user_id,item_id,app,dt,row_number() over(partition by item_id, app 
> order by dt desc) as rn from user_action
> )t where rn<=5;
> insert into redis_sink
> select
> concat("id", item_id) as redis_key, app as hash_key,user_id as hash_value 
> from base_lastn where rn=1;
> insert into redis_sink
> select
> concat("list_", item_id) as redis_key, app as hash_key,listagg(user_id,",") 
> as hash_value from base_lastn where group by item_id, app;
>  
> There will be a scene that the value in the top 1 will not appear in the 
> first or last value of the top 5.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33657) Insert message in top n without row number didn't consider it's number and may incorrect

[zlzhang0122 (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zlzhang0122 updated FLINK-33657:

Summary: Insert message in top n without row number didn't consider it's 
number and may incorrect  (was: Insert message in top n without row number 
didn't consider it's number and may not correct)

> Insert message in top n without row number didn't consider it's number and 
> may incorrect
> 
>
> Key: FLINK-33657
> URL: https://issues.apache.org/jira/browse/FLINK-33657
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / Runtime
>Affects Versions: 1.16.2, 1.17.1
>Reporter: zlzhang0122
>Priority: Major
>
> The new insert message in top n without row number didn't consider it's order 
> and just 
> collectInsert() to the next operator, this may incorrect when the next 
> operator collect all the top n records and aggregate it.
>  
> For example：
> create table user_info(
> user_id int,
> item_id int,
> app string,
> dt timestamp
> ) whith(
> 'connector'='kafka',
> ...
> );
> create table redis_sink (
> redis_key string,
> hash_key string,
> hash_value string
> )
> with (
> 'connector' = 'redis',
> 'command' = 'hmset'
> 'nodes' = 'xxx',
> 'additional-ttl' = 'xx'
> );
> create view base_lastn
> as select * from(
> select user_id,item_id,app,dt,row_number() over(partition by item_id, app 
> order by dt desc) as rn from user_action
> )t where rn<=5;
> insert into redis_sink
> select
> concat("id", item_id) as redis_key, app as hash_key,user_id as hash_value 
> from base_lastn where rn=1;
> insert into redis_sink
> select
> concat("list_", item_id) as redis_key, app as hash_key,listagg(user_id,",") 
> as hash_value from base_lastn where group by item_id, app;
>  
> There will be a scene that the value in the top 1 will not appear in the 
> first or last value of the top 5.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-25857) Add committer metrics to track the status of committables

[Peter Vary (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789963#comment-17789963
 ] 

Peter Vary commented on FLINK-25857:


If you check the discussion [1] the diamond inheritance of the 
`Sink.createWriter` method prevents any backward compatible change of the 
method. One could argue that this is a flawed design.

*About the process and the compatibility*
[~Weijie Guo]: Here is my understanding of the FLIP process, please correct me, 
if I am wrong somewhere:
- If there is a change which modifies or creates a new API we should create a 
FLIP to discuss the change [2]
- We start the discussion on the mailing list, so everyone who is interested 
in, could participate [3]
- If there is a consensus on the design, we start a voting thread [4]
- If the voting is successful, we announce the result and close the FLIP [5]
- If during the implementation we found issues we discuss it there - we do not 
modify the finalised FLIP [6]

Maybe it would be good to have an additional step, that when there is a change 
related to the original design of the FLIP. We should send a letter to the 
mailing list as well, to notify interested parties who are not following the 
actual implementation.

About the deprecation process, I have been working based on the API 
compatibility guarantees [7] stated in the docs. Based on the table there a 
PublicEvolving API should be source and binary compatible for patch releases, 
but there is no guarantees for minor releases. Maybe the same redesign process 
happened during the implementation of FLIP-321 [8]? I was not involved there, 
so I do not have a first hand information.

[1] - https://github.com/apache/flink/pull/23555#discussion_r1371740397
[2] - 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
[3] - https://lists.apache.org/thread/v3mrspdlrqrzvbwm0lcgr0j4v03dx97c
[4] - https://lists.apache.org/thread/4f7w4n3nywk8ygnwlxk39oncl3cntp3n
[5] - https://lists.apache.org/thread/jw39s55tzzpdkzmlh0vshmjnfrjg02nr
[6] - https://github.com/apache/flink/pull/23555#discussion_r1369177945
[7] - 
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
[8] - 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process



> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33658) Hosted runner lost communication

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33658:
-

 Summary: Hosted runner lost communication
 Key: FLINK-33658
 URL: https://issues.apache.org/jira/browse/FLINK-33658
 Project: Flink
  Issue Type: Sub-task
  Components: Test Infrastructure
Reporter: Matthias Pohl


Some jobs failed due to lost communication: 
https://github.com/XComp/flink/actions/runs/6997726518

{quote}
The hosted runner: GitHub Actions 15 lost communication with the server. 
Anything in your workflow that terminates the runner process, starves it for 
CPU/Memory, or blocks its network access can cause this error.
{quote}

This is not really something we can fix. The issue is created for documentation 
purposes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Closed] (FLINK-33242) misc module: YARN tests are flaky

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl closed FLINK-33242.
-
Resolution: Cannot Reproduce

The issue didn't reappear for a while. I'm closing it as not-reproducible.

> misc module: YARN tests are flaky
> -
>
> Key: FLINK-33242
> URL: https://issues.apache.org/jira/browse/FLINK-33242
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> https://github.com/XComp/flink/actions/runs/6473584177/job/17581942919
> {code}
> 2023-10-10T23:16:09.3548634Z Oct 10 23:16:09 23:16:09.354 [ERROR] Tests run: 
> 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.664 s <<< FAILURE! - 
> in org.apache.flink.yarn.YarnPrioritySchedulingITCase
> 2023-10-10T23:16:09.3564980Z Oct 10 23:16:09 23:16:09.354 [ERROR] 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority
>   Time elapsed: 1.226 s  <<< ERROR!
> 2023-10-10T23:16:09.3565608Z Oct 10 23:16:09 java.lang.RuntimeException: 
> Runner failed with exception.
> 2023-10-10T23:16:09.3566290Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnTestBase.startWithArgs(YarnTestBase.java:949)
> 2023-10-10T23:16:09.3566954Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.lambda$yarnApplication_submissionWithPriority_shouldRespectPriority$0(YarnPrioritySchedulingITCase.java:45)
> 2023-10-10T23:16:09.3567646Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:303)
> 2023-10-10T23:16:09.3568447Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority(YarnPrioritySchedulingITCase.java:41)
> 2023-10-10T23:16:09.3569187Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2023-10-10T23:16:09.3569805Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2023-10-10T23:16:09.3570485Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2023-10-10T23:16:09.3571052Z Oct 10 23:16:09  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2023-10-10T23:16:09.3571527Z Oct 10 23:16:09  at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> 2023-10-10T23:16:09.3572075Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2023-10-10T23:16:09.3572716Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2023-10-10T23:16:09.3573350Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> 2023-10-10T23:16:09.3573954Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> 2023-10-10T23:16:09.3574665Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> 2023-10-10T23:16:09.3575378Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> 2023-10-10T23:16:09.3576139Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> 2023-10-10T23:16:09.3576852Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2023-10-10T23:16:09.3577539Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2023-10-10T23:16:09.3578225Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2023-10-10T23:16:09.3578898Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2023-10-10T23:16:09.3579568Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> 2023-10-10T23:16:09.3580243Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> 2023-10-10T23:16:09.3580917Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMeth

[jira] [Closed] (FLINK-33281) e2e 2 stage: PyFlink end-to-end test

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl closed FLINK-33281.
-
Resolution: Cannot Reproduce

The issue didn't reappear for a while. I'm closing it as not-reproducible.

> e2e 2 stage: PyFlink end-to-end test
> 
>
> Key: FLINK-33281
> URL: https://issues.apache.org/jira/browse/FLINK-33281
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> [https://github.com/XComp/flink/actions/runs/6529754573/job/17728244982#step:15:7938]
> {code:java}
>  Oct 16 08:02:13 pyflink.util.exceptions.TableException: 
> org.apache.flink.table.api.TableException: Failed to execute sql
> 7868Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1048)
> 7869Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:864)
> 7870Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1097)
> 7871Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TablePipelineImpl.execute(TablePipelineImpl.java:59)
> 7872Oct 16 08:02:13   at 
> org.apache.flink.table.api.Table.executeInsert(Table.java:1074)
> 7873Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 7874Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 7875Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 7876Oct 16 08:02:13   at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 7877Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> 7878Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
> 7879Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
> 7880Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> 7881Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
> 7882Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> 7883Oct 16 08:02:13   at java.base/java.lang.Thread.run(Thread.java:829)
> 7884Oct 16 08:02:13 Caused by: org.apache.flink.util.FlinkException: Failed 
> to execute job 'insert-into_default_catalog.default_database.Results'.
> 7885Oct 16 08:02:13   at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2253)
> 7886Oct 16 08:02:13   at 
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189)
> 7887Oct 16 08:02:13   at 
> org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110)
> 7888Oct 16 08:02:13   at 
> org.apache.flink.table.executor.python.ChainingOptimizingExecutor.executeAsync(ChainingOptimizingExecutor.java:88)
> 7889Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1020)
> 7890Oct 16 08:02:13   ... 15 more
> 7891Oct 16 08:02:13 Caused by: 
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit 
> JobGraph.
> 7892Oct 16 08:02:13   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$12(RestClusterClient.java:479)
> 7893Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> 7894Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> 7895Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> 7896Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> 7897Oct 16 08:02:13   at 
> org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:272)
> 7898Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> 7899Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> 7900Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> 7901Oct 16 08:02:13   at 
> java.base/java.util.concurre

[jira] [Closed] (FLINK-33283) core stage: WebFrontendBootstrapTest.testHandlersMustBeLoaded

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl closed FLINK-33283.
-
Resolution: Cannot Reproduce

The issue didn't reappear for a while. I'm closing it as not-reproducible.

> core stage: WebFrontendBootstrapTest.testHandlersMustBeLoaded
> -
>
> Key: FLINK-33283
> URL: https://issues.apache.org/jira/browse/FLINK-33283
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> [https://github.com/XComp/flink/actions/runs/6525957588/job/17719339666#step:10:12279]
> {code:java}
>  Error: 20:06:13 20:06:13.132 [ERROR] 
> org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrapTest.testHandlersMustBeLoaded
>   Time elapsed: 2.298 s  <<< FAILURE!
> 12279Oct 15 20:06:13 org.opentest4j.AssertionFailedError: 
> 12280Oct 15 20:06:13 
> 12281Oct 15 20:06:13 expected: 404
> 12282Oct 15 20:06:13  but was: 200
> 12283Oct 15 20:06:13  at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method)
> 12284Oct 15 20:06:13  at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 12285Oct 15 20:06:13  at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 12286Oct 15 20:06:13  at 
> org.apache.flink.runtime.webmonitor.utils.WebFrontendBootstrapTest.testHandlersMustBeLoaded(WebFrontendBootstrapTest.java:89)
> 12287Oct 15 20:06:13  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Closed] (FLINK-33284) core stage: HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl closed FLINK-33284.
-
Resolution: Fixed

The issue didn't reappear for a while. I'm closing it as not-reproducible.

> core stage: HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)
> --
>
> Key: FLINK-33284
> URL: https://issues.apache.org/jira/browse/FLINK-33284
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> https://github.com/XComp/flink/actions/runs/6525957588/job/17719339666#step:10:12209
> {code}
> Error: 20:06:13 20:06:13.081 [ERROR] 
> org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)
>   Time elapsed: 1.981 s  <<< FAILURE!
> Oct 15 20:06:13 org.opentest4j.AssertionFailedError: 
> Oct 15 20:06:13 
> Oct 15 20:06:13 expected: 200
> Oct 15 20:06:13  but was: 404
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Oct 15 20:06:13   at 
> org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandlerTest.testRespondWithFile(HistoryServerStaticFileServerHandlerTest.java:70)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-33284) core stage: HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789972#comment-17789972
 ] 

Matthias Pohl edited comment on FLINK-33284 at 11/27/23 8:42 AM:
-

The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into it for now. We can re-open this Jira 
again if the problem re-occurs.


was (Author: mapohl):
The issue didn't reappear for a while. I'm closing it as not-reproducible.

> core stage: HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)
> --
>
> Key: FLINK-33284
> URL: https://issues.apache.org/jira/browse/FLINK-33284
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> https://github.com/XComp/flink/actions/runs/6525957588/job/17719339666#step:10:12209
> {code}
> Error: 20:06:13 20:06:13.081 [ERROR] 
> org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)
>   Time elapsed: 1.981 s  <<< FAILURE!
> Oct 15 20:06:13 org.opentest4j.AssertionFailedError: 
> Oct 15 20:06:13 
> Oct 15 20:06:13 expected: 200
> Oct 15 20:06:13  but was: 404
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Oct 15 20:06:13   at 
> org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandlerTest.testRespondWithFile(HistoryServerStaticFileServerHandlerTest.java:70)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-33242) misc module: YARN tests are flaky

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789969#comment-17789969
 ] 

Matthias Pohl edited comment on FLINK-33242 at 11/27/23 8:42 AM:
-

The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into investigating it further for now. We can 
re-open this Jira again if the problem re-occurs.


was (Author: mapohl):
The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into it for now. We can re-open this Jira 
again if the problem re-occurs.

> misc module: YARN tests are flaky
> -
>
> Key: FLINK-33242
> URL: https://issues.apache.org/jira/browse/FLINK-33242
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> https://github.com/XComp/flink/actions/runs/6473584177/job/17581942919
> {code}
> 2023-10-10T23:16:09.3548634Z Oct 10 23:16:09 23:16:09.354 [ERROR] Tests run: 
> 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.664 s <<< FAILURE! - 
> in org.apache.flink.yarn.YarnPrioritySchedulingITCase
> 2023-10-10T23:16:09.3564980Z Oct 10 23:16:09 23:16:09.354 [ERROR] 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority
>   Time elapsed: 1.226 s  <<< ERROR!
> 2023-10-10T23:16:09.3565608Z Oct 10 23:16:09 java.lang.RuntimeException: 
> Runner failed with exception.
> 2023-10-10T23:16:09.3566290Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnTestBase.startWithArgs(YarnTestBase.java:949)
> 2023-10-10T23:16:09.3566954Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.lambda$yarnApplication_submissionWithPriority_shouldRespectPriority$0(YarnPrioritySchedulingITCase.java:45)
> 2023-10-10T23:16:09.3567646Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:303)
> 2023-10-10T23:16:09.3568447Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority(YarnPrioritySchedulingITCase.java:41)
> 2023-10-10T23:16:09.3569187Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2023-10-10T23:16:09.3569805Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2023-10-10T23:16:09.3570485Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2023-10-10T23:16:09.3571052Z Oct 10 23:16:09  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2023-10-10T23:16:09.3571527Z Oct 10 23:16:09  at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> 2023-10-10T23:16:09.3572075Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2023-10-10T23:16:09.3572716Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2023-10-10T23:16:09.3573350Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> 2023-10-10T23:16:09.3573954Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> 2023-10-10T23:16:09.3574665Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> 2023-10-10T23:16:09.3575378Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> 2023-10-10T23:16:09.3576139Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> 2023-10-10T23:16:09.3576852Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2023-10-10T23:16:09.3577539Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2023-10-10T23:16:09.3578225Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2023-10-10T23:16:09.3578898Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2023-10-10T23:16:09.3579568Z Oct 10 23:16:09  at 
> org.j

[jira] [Comment Edited] (FLINK-33242) misc module: YARN tests are flaky

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789969#comment-17789969
 ] 

Matthias Pohl edited comment on FLINK-33242 at 11/27/23 8:42 AM:
-

The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into it for now. We can re-open this Jira 
again if the problem re-occurs.


was (Author: mapohl):
The issue didn't reappear for a while. I'm closing it as not-reproducible.

> misc module: YARN tests are flaky
> -
>
> Key: FLINK-33242
> URL: https://issues.apache.org/jira/browse/FLINK-33242
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> https://github.com/XComp/flink/actions/runs/6473584177/job/17581942919
> {code}
> 2023-10-10T23:16:09.3548634Z Oct 10 23:16:09 23:16:09.354 [ERROR] Tests run: 
> 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.664 s <<< FAILURE! - 
> in org.apache.flink.yarn.YarnPrioritySchedulingITCase
> 2023-10-10T23:16:09.3564980Z Oct 10 23:16:09 23:16:09.354 [ERROR] 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority
>   Time elapsed: 1.226 s  <<< ERROR!
> 2023-10-10T23:16:09.3565608Z Oct 10 23:16:09 java.lang.RuntimeException: 
> Runner failed with exception.
> 2023-10-10T23:16:09.3566290Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnTestBase.startWithArgs(YarnTestBase.java:949)
> 2023-10-10T23:16:09.3566954Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.lambda$yarnApplication_submissionWithPriority_shouldRespectPriority$0(YarnPrioritySchedulingITCase.java:45)
> 2023-10-10T23:16:09.3567646Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:303)
> 2023-10-10T23:16:09.3568447Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority(YarnPrioritySchedulingITCase.java:41)
> 2023-10-10T23:16:09.3569187Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2023-10-10T23:16:09.3569805Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2023-10-10T23:16:09.3570485Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2023-10-10T23:16:09.3571052Z Oct 10 23:16:09  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2023-10-10T23:16:09.3571527Z Oct 10 23:16:09  at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> 2023-10-10T23:16:09.3572075Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2023-10-10T23:16:09.3572716Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2023-10-10T23:16:09.3573350Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> 2023-10-10T23:16:09.3573954Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> 2023-10-10T23:16:09.3574665Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> 2023-10-10T23:16:09.3575378Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> 2023-10-10T23:16:09.3576139Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> 2023-10-10T23:16:09.3576852Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2023-10-10T23:16:09.3577539Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2023-10-10T23:16:09.3578225Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2023-10-10T23:16:09.3578898Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2023-10-10T23:16:09.3579568Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> 2023-10-10T23:16:09.35802

[jira] [Comment Edited] (FLINK-33281) e2e 2 stage: PyFlink end-to-end test

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789970#comment-17789970
 ] 

Matthias Pohl edited comment on FLINK-33281 at 11/27/23 8:42 AM:
-

The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into investigating it further for now. We can 
re-open this Jira again if the problem re-occurs.


was (Author: mapohl):
The issue didn't reappear for a while. I'm closing it as not-reproducible.

> e2e 2 stage: PyFlink end-to-end test
> 
>
> Key: FLINK-33281
> URL: https://issues.apache.org/jira/browse/FLINK-33281
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> [https://github.com/XComp/flink/actions/runs/6529754573/job/17728244982#step:15:7938]
> {code:java}
>  Oct 16 08:02:13 pyflink.util.exceptions.TableException: 
> org.apache.flink.table.api.TableException: Failed to execute sql
> 7868Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1048)
> 7869Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:864)
> 7870Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1097)
> 7871Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TablePipelineImpl.execute(TablePipelineImpl.java:59)
> 7872Oct 16 08:02:13   at 
> org.apache.flink.table.api.Table.executeInsert(Table.java:1074)
> 7873Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 7874Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 7875Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 7876Oct 16 08:02:13   at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 7877Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> 7878Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
> 7879Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
> 7880Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> 7881Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
> 7882Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> 7883Oct 16 08:02:13   at java.base/java.lang.Thread.run(Thread.java:829)
> 7884Oct 16 08:02:13 Caused by: org.apache.flink.util.FlinkException: Failed 
> to execute job 'insert-into_default_catalog.default_database.Results'.
> 7885Oct 16 08:02:13   at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2253)
> 7886Oct 16 08:02:13   at 
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189)
> 7887Oct 16 08:02:13   at 
> org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110)
> 7888Oct 16 08:02:13   at 
> org.apache.flink.table.executor.python.ChainingOptimizingExecutor.executeAsync(ChainingOptimizingExecutor.java:88)
> 7889Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1020)
> 7890Oct 16 08:02:13   ... 15 more
> 7891Oct 16 08:02:13 Caused by: 
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit 
> JobGraph.
> 7892Oct 16 08:02:13   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$12(RestClusterClient.java:479)
> 7893Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> 7894Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> 7895Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> 7896Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> 7897Oct 16 08:02:13   at 
> org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:272)
> 7898Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)

[jira] [Comment Edited] (FLINK-33284) core stage: HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789972#comment-17789972
 ] 

Matthias Pohl edited comment on FLINK-33284 at 11/27/23 8:43 AM:
-

The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into investigating it further for now. We can 
re-open this Jira again if the problem re-occurs.


was (Author: mapohl):
The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into it for now. We can re-open this Jira 
again if the problem re-occurs.

> core stage: HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)
> --
>
> Key: FLINK-33284
> URL: https://issues.apache.org/jira/browse/FLINK-33284
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> https://github.com/XComp/flink/actions/runs/6525957588/job/17719339666#step:10:12209
> {code}
> Error: 20:06:13 20:06:13.081 [ERROR] 
> org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path)
>   Time elapsed: 1.981 s  <<< FAILURE!
> Oct 15 20:06:13 org.opentest4j.AssertionFailedError: 
> Oct 15 20:06:13 
> Oct 15 20:06:13 expected: 200
> Oct 15 20:06:13  but was: 404
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Oct 15 20:06:13   at 
> org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandlerTest.testRespondWithFile(HistoryServerStaticFileServerHandlerTest.java:70)
> Oct 15 20:06:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-33281) e2e 2 stage: PyFlink end-to-end test

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789970#comment-17789970
 ] 

Matthias Pohl edited comment on FLINK-33281 at 11/27/23 8:43 AM:
-

The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into investigating it further for now. We can 
re-open this Jira again if the problem re-occurs.


was (Author: mapohl):
The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into investigating it further for now. We can 
re-open this Jira again if the problem re-occurs.

> e2e 2 stage: PyFlink end-to-end test
> 
>
> Key: FLINK-33281
> URL: https://issues.apache.org/jira/browse/FLINK-33281
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> [https://github.com/XComp/flink/actions/runs/6529754573/job/17728244982#step:15:7938]
> {code:java}
>  Oct 16 08:02:13 pyflink.util.exceptions.TableException: 
> org.apache.flink.table.api.TableException: Failed to execute sql
> 7868Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1048)
> 7869Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:864)
> 7870Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1097)
> 7871Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TablePipelineImpl.execute(TablePipelineImpl.java:59)
> 7872Oct 16 08:02:13   at 
> org.apache.flink.table.api.Table.executeInsert(Table.java:1074)
> 7873Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 7874Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 7875Oct 16 08:02:13   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 7876Oct 16 08:02:13   at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 7877Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> 7878Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
> 7879Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
> 7880Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> 7881Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
> 7882Oct 16 08:02:13   at 
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> 7883Oct 16 08:02:13   at java.base/java.lang.Thread.run(Thread.java:829)
> 7884Oct 16 08:02:13 Caused by: org.apache.flink.util.FlinkException: Failed 
> to execute job 'insert-into_default_catalog.default_database.Results'.
> 7885Oct 16 08:02:13   at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2253)
> 7886Oct 16 08:02:13   at 
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189)
> 7887Oct 16 08:02:13   at 
> org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110)
> 7888Oct 16 08:02:13   at 
> org.apache.flink.table.executor.python.ChainingOptimizingExecutor.executeAsync(ChainingOptimizingExecutor.java:88)
> 7889Oct 16 08:02:13   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1020)
> 7890Oct 16 08:02:13   ... 15 more
> 7891Oct 16 08:02:13 Caused by: 
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit 
> JobGraph.
> 7892Oct 16 08:02:13   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$12(RestClusterClient.java:479)
> 7893Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> 7894Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> 7895Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> 7896Oct 16 08:02:13   at 
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> 7897Oct 16 08:02:13   at 
> org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.

[jira] [Comment Edited] (FLINK-33242) misc module: YARN tests are flaky

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789969#comment-17789969
 ] 

Matthias Pohl edited comment on FLINK-33242 at 11/27/23 8:43 AM:
-

The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into investigating it further for now. We can 
re-open this Jira again if the problem re-occurs.


was (Author: mapohl):
The issue didn't reappear for a while. I'm closing it as not-reproducible to 
track that there's no effort put into investigating it further for now. We can 
re-open this Jira again if the problem re-occurs.

> misc module: YARN tests are flaky
> -
>
> Key: FLINK-33242
> URL: https://issues.apache.org/jira/browse/FLINK-33242
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> https://github.com/XComp/flink/actions/runs/6473584177/job/17581942919
> {code}
> 2023-10-10T23:16:09.3548634Z Oct 10 23:16:09 23:16:09.354 [ERROR] Tests run: 
> 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.664 s <<< FAILURE! - 
> in org.apache.flink.yarn.YarnPrioritySchedulingITCase
> 2023-10-10T23:16:09.3564980Z Oct 10 23:16:09 23:16:09.354 [ERROR] 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority
>   Time elapsed: 1.226 s  <<< ERROR!
> 2023-10-10T23:16:09.3565608Z Oct 10 23:16:09 java.lang.RuntimeException: 
> Runner failed with exception.
> 2023-10-10T23:16:09.3566290Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnTestBase.startWithArgs(YarnTestBase.java:949)
> 2023-10-10T23:16:09.3566954Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.lambda$yarnApplication_submissionWithPriority_shouldRespectPriority$0(YarnPrioritySchedulingITCase.java:45)
> 2023-10-10T23:16:09.3567646Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:303)
> 2023-10-10T23:16:09.3568447Z Oct 10 23:16:09  at 
> org.apache.flink.yarn.YarnPrioritySchedulingITCase.yarnApplication_submissionWithPriority_shouldRespectPriority(YarnPrioritySchedulingITCase.java:41)
> 2023-10-10T23:16:09.3569187Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2023-10-10T23:16:09.3569805Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2023-10-10T23:16:09.3570485Z Oct 10 23:16:09  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2023-10-10T23:16:09.3571052Z Oct 10 23:16:09  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2023-10-10T23:16:09.3571527Z Oct 10 23:16:09  at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> 2023-10-10T23:16:09.3572075Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2023-10-10T23:16:09.3572716Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2023-10-10T23:16:09.3573350Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> 2023-10-10T23:16:09.3573954Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> 2023-10-10T23:16:09.3574665Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> 2023-10-10T23:16:09.3575378Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> 2023-10-10T23:16:09.3576139Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> 2023-10-10T23:16:09.3576852Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2023-10-10T23:16:09.3577539Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2023-10-10T23:16:09.3578225Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2023-10-10T23:16:09.3578898Z Oct 10 23:16:09  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2023-10-10T23:16:09.3579568Z Oct 10

[jira] [Created] (FLINK-33659) Avoid unnecessary retries when restore from savepoint failed.

[zhouli (Jira)]

zhouli created FLINK-33659:
--

 Summary: Avoid unnecessary retries when restore from savepoint 
failed.
 Key: FLINK-33659
 URL: https://issues.apache.org/jira/browse/FLINK-33659
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: zhouli


when restore a job from savepoint failed, if restart strategy is enabled, flink 
will try to restart the job, and the restore would fail again.

We may wrap the exception as 
[SuppressRestartsException|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/execution/SuppressRestartsException.java]
 to avoid unnecessary retries.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33655][tests] Upgrade ArchUnit to 1.2.0 to support java 21 [flink]

[via GitHub]

snuyanzin commented on PR #23808:
URL: https://github.com/apache/flink/pull/23808#issuecomment-1827380143

   Thanks for taking a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-33656) If json.ignose-parse-errors =true is configured and Array parsing errors occur, other columns will be empty

[duke (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789973#comment-17789973
 ] 

duke commented on FLINK-33656:
--

If the type of the attentionList field is defined as Array, but the value does 
not conform to the array parsing structure, and the attentionList field is at 
the head or not at the end of the json message, all subsequent fields of 
attentionList are returned as null.

For example: \{"attentionList":{},"id":"1234","sheetNo":"1700803757032"} In the 
json string, attentionList is located at the head of the entire json message, 
which is read in the flink sql json format. The source table structure is as 
follows:
create table default_catalog.default_database.test (
id                   STRING
,sheetNo              STRING
,attentionList        ARRAY
)
WITH ('connector' = 'kafka',
'topic' = 'test',
'properties.bootstrap.servers' = 'xxx',
'scan.startup.mode' = 'latest-offset',
'format' = 'json',
'json.ignore-parse-errors' = 'true',
'json.map-null-key.mode' = 'LITERAL'
);
Obviously the attentionList field is not parsed correctly, which will result in 
the id and sheetNo fields being empty;
Suppose you modify the json message 
:\{"id":"1234","attentionList":{},"sheetNo":"1700803757032"}, depending on the 
location of attentionList, sheetNo will be empty.

> If json.ignose-parse-errors =true is configured and Array parsing errors 
> occur, other columns will be empty
> ---
>
> Key: FLINK-33656
> URL: https://issues.apache.org/jira/browse/FLINK-33656
> Project: Flink
>  Issue Type: Bug
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.16.0, 1.17.0, 1.16.1, 1.16.2, 1.18.0, 1.17.1
>Reporter: duke
>Priority: Critical
> Attachments: image-2023-11-27-13-58-22-513.png, 
> image-2023-11-27-13-59-42-066.png, image-2023-11-27-14-00-04-672.png, 
> image-2023-11-27-14-00-41-176.png, image-2023-11-27-14-01-12-187.png, 
> image-2023-11-27-14-02-01-252.png, image-2023-11-27-14-02-30-666.png, 
> image-2023-11-27-14-02-52-065.png, image-2023-11-27-14-03-10-885.png
>
>
> If json.ignore-parse-errors is set to true and Array parsing errors occur, 
> the fields following array are resolved as empty in the complete json message
> 1. !image-2023-11-27-13-59-42-066.png!
> !image-2023-11-27-14-00-04-672.png!
>  
> 2. !image-2023-11-27-14-00-41-176.png!
> !image-2023-11-27-14-01-12-187.png!
> 3. !image-2023-11-27-14-02-52-065.png!
> !image-2023-11-27-14-03-10-885.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33655][tests] Upgrade ArchUnit to 1.2.0 to support java 21 [flink]

[via GitHub]

snuyanzin merged PR #23808:
URL: https://github.com/apache/flink/pull/23808


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-33655) Upgrade Archunit to 1.1.0+

[Sergey Nuyanzin (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789976#comment-17789976
 ] 

Sergey Nuyanzin commented on FLINK-33655:
-

Merged to master as 
[83a6d1d57d7d544243a924a70267437e5e0eb906|https://github.com/apache/flink/commit/83a6d1d57d7d544243a924a70267437e5e0eb906]

> Upgrade Archunit to 1.1.0+
> --
>
> Key: FLINK-33655
> URL: https://issues.apache.org/jira/browse/FLINK-33655
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Major
>  Labels: pull-request-available
>
> ASM 9.5 (which has support for java 21[1]) came only with ArchUnit 1.1.0 [2]
> With current ArchUnit(1.0.0) in case of jdk21 it fails
> {noformat}
> mvn clean install -DskipTests -Dfast -Pjava21-target
> mvn verify -pl flink-architecture-tests/flink-architecture-tests-production/ 
> -Darchunit.freeze.store.default.allowStoreUpdate=false
> {noformat}
> like
> {noformat}
> Nov 26 16:07:42 16:07:42.024 [ERROR] Tests run: 2, Failures: 0, Errors: 1, 
> Skipped: 0, Time elapsed: 5.742 s <<< FAILURE! - in 
> org.apache.flink.architecture.rules.ITCaseRules
> Nov 26 16:07:42 16:07:42.025 [ERROR] ITCaseRules.ITCASE_USE_MINICLUSTER  Time 
> elapsed: 0.005 s  <<< ERROR!
> Nov 26 16:07:42 
> com.tngtech.archunit.library.freeze.StoreUpdateFailedException: Updating 
> frozen violations is disabled (enable by configuration 
> freeze.store.default.allowStoreUpdate=true)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.ViolationStoreFactory$TextFileBasedViolationStore.save(ViolationStoreFactory.java:125)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule$ViolationStoreLineBreakAdapter.save(FreezingArchRule.java:277)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.removeObsoleteViolationsFromStore(FreezingArchRule.java:154)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.removeObsoleteViolationsFromStoreAndReturnNewViolations(FreezingArchRule.java:146)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.evaluate(FreezingArchRule.java:127)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:84)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:97)
> {noformat}
> [1] https://asm.ow2.io/versions.html#9.5
> [2] https://github.com/TNG/ArchUnit/pull/1098



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Closed] (FLINK-33655) Upgrade Archunit to 1.1.0+

[Sergey Nuyanzin (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Nuyanzin closed FLINK-33655.
---

> Upgrade Archunit to 1.1.0+
> --
>
> Key: FLINK-33655
> URL: https://issues.apache.org/jira/browse/FLINK-33655
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
>
> ASM 9.5 (which has support for java 21[1]) came only with ArchUnit 1.1.0 [2]
> With current ArchUnit(1.0.0) in case of jdk21 it fails
> {noformat}
> mvn clean install -DskipTests -Dfast -Pjava21-target
> mvn verify -pl flink-architecture-tests/flink-architecture-tests-production/ 
> -Darchunit.freeze.store.default.allowStoreUpdate=false
> {noformat}
> like
> {noformat}
> Nov 26 16:07:42 16:07:42.024 [ERROR] Tests run: 2, Failures: 0, Errors: 1, 
> Skipped: 0, Time elapsed: 5.742 s <<< FAILURE! - in 
> org.apache.flink.architecture.rules.ITCaseRules
> Nov 26 16:07:42 16:07:42.025 [ERROR] ITCaseRules.ITCASE_USE_MINICLUSTER  Time 
> elapsed: 0.005 s  <<< ERROR!
> Nov 26 16:07:42 
> com.tngtech.archunit.library.freeze.StoreUpdateFailedException: Updating 
> frozen violations is disabled (enable by configuration 
> freeze.store.default.allowStoreUpdate=true)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.ViolationStoreFactory$TextFileBasedViolationStore.save(ViolationStoreFactory.java:125)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule$ViolationStoreLineBreakAdapter.save(FreezingArchRule.java:277)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.removeObsoleteViolationsFromStore(FreezingArchRule.java:154)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.removeObsoleteViolationsFromStoreAndReturnNewViolations(FreezingArchRule.java:146)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.evaluate(FreezingArchRule.java:127)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:84)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:97)
> {noformat}
> [1] https://asm.ow2.io/versions.html#9.5
> [2] https://github.com/TNG/ArchUnit/pull/1098



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-33552) Job (and its data) is lost

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789975#comment-17789975
 ] 

Matthias Pohl commented on FLINK-33552:
---

https://github.com/XComp/flink/actions/runs/6997726518

> Job (and its data) is lost
> --
>
> Key: FLINK-33552
> URL: https://issues.apache.org/jira/browse/FLINK-33552
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> No investigation possible.
> https://github.com/XComp/flink/actions/runs/6866313050
> {code}
> Flink CI (Java 8/Scala 2.12/Hadoop 2.10.2) / Test (module: misc)
> The hosted runner: GitHub Actions 18 lost communication with the server. 
> Anything in your workflow that terminates the runner process, starves it for 
> CPU/Memory, or blocks its network access can cause this error.
> {code}
> This is kept for documentation purposes for now.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Closed] (FLINK-33658) Hosted runner lost communication

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl closed FLINK-33658.
-
Resolution: Duplicate

> Hosted runner lost communication
> 
>
> Key: FLINK-33658
> URL: https://issues.apache.org/jira/browse/FLINK-33658
> Project: Flink
>  Issue Type: Sub-task
>  Components: Test Infrastructure
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: github-actions, test-stability
>
> Some jobs failed due to lost communication: 
> https://github.com/XComp/flink/actions/runs/6997726518
> {quote}
> The hosted runner: GitHub Actions 15 lost communication with the server. 
> Anything in your workflow that terminates the runner process, starves it for 
> CPU/Memory, or blocks its network access can cause this error.
> {quote}
> This is not really something we can fix. The issue is created for 
> documentation purposes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-33659) Avoid unnecessary retries when restore from savepoint failed.

[zhouli (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789974#comment-17789974
 ] 

zhouli commented on FLINK-33659:


Hi, [~zhuzh] , wdyt ?

> Avoid unnecessary retries when restore from savepoint failed.
> -
>
> Key: FLINK-33659
> URL: https://issues.apache.org/jira/browse/FLINK-33659
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing
>Reporter: zhouli
>Priority: Major
>
> when restore a job from savepoint failed, if restart strategy is enabled, 
> flink will try to restart the job, and the restore would fail again.
> We may wrap the exception as 
> [SuppressRestartsException|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/execution/SuppressRestartsException.java]
>  to avoid unnecessary retries.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (FLINK-33655) Upgrade Archunit to 1.1.0+

[Sergey Nuyanzin (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Nuyanzin resolved FLINK-33655.
-
Fix Version/s: 1.19.0
   Resolution: Fixed

> Upgrade Archunit to 1.1.0+
> --
>
> Key: FLINK-33655
> URL: https://issues.apache.org/jira/browse/FLINK-33655
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
>
> ASM 9.5 (which has support for java 21[1]) came only with ArchUnit 1.1.0 [2]
> With current ArchUnit(1.0.0) in case of jdk21 it fails
> {noformat}
> mvn clean install -DskipTests -Dfast -Pjava21-target
> mvn verify -pl flink-architecture-tests/flink-architecture-tests-production/ 
> -Darchunit.freeze.store.default.allowStoreUpdate=false
> {noformat}
> like
> {noformat}
> Nov 26 16:07:42 16:07:42.024 [ERROR] Tests run: 2, Failures: 0, Errors: 1, 
> Skipped: 0, Time elapsed: 5.742 s <<< FAILURE! - in 
> org.apache.flink.architecture.rules.ITCaseRules
> Nov 26 16:07:42 16:07:42.025 [ERROR] ITCaseRules.ITCASE_USE_MINICLUSTER  Time 
> elapsed: 0.005 s  <<< ERROR!
> Nov 26 16:07:42 
> com.tngtech.archunit.library.freeze.StoreUpdateFailedException: Updating 
> frozen violations is disabled (enable by configuration 
> freeze.store.default.allowStoreUpdate=true)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.ViolationStoreFactory$TextFileBasedViolationStore.save(ViolationStoreFactory.java:125)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule$ViolationStoreLineBreakAdapter.save(FreezingArchRule.java:277)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.removeObsoleteViolationsFromStore(FreezingArchRule.java:154)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.removeObsoleteViolationsFromStoreAndReturnNewViolations(FreezingArchRule.java:146)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.evaluate(FreezingArchRule.java:127)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:84)
> Nov 26 16:07:42   at 
> com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:97)
> {noformat}
> [1] https://asm.ow2.io/versions.html#9.5
> [2] https://github.com/TNG/ArchUnit/pull/1098



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33660) AggregateITCase.testConstantGroupKeyWithUpsertSink due to unexpected behavior

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl updated FLINK-33660:
--
Labels: test-stability  (was: )

> AggregateITCase.testConstantGroupKeyWithUpsertSink due to unexpected behavior
> -
>
> Key: FLINK-33660
> URL: https://issues.apache.org/jira/browse/FLINK-33660
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Coordination, Table SQL / Runtime
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
>  Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54895&view=logs&j=a9db68b9-a7e0-54b6-0f98-010e0aff39e2&t=cdd32e0b-6047-565b-c58f-14054472f1be&l=11063
> {code}
> Nov 25 05:02:47 05:02:47.850 [ERROR] 
> org.apache.flink.table.planner.runtime.stream.sql.AggregateITCase.testConstantGroupKeyWithUpsertSink
>   Time elapsed: 107.302 s  <<< ERROR!
> Nov 25 05:02:47 java.util.concurrent.ExecutionException: 
> org.apache.flink.table.api.TableException: Failed to wait job finish
> Nov 25 05:02:47   at 
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> Nov 25 05:02:47   at 
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Nov 25 05:02:47   at 
> org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:118)
> Nov 25 05:02:47   at 
> org.apache.flink.table.api.internal.TableResultImpl.await(TableResultImpl.java:81)
> Nov 25 05:02:47   at 
> org.apache.flink.table.planner.runtime.stream.sql.AggregateITCase.testConstantGroupKeyWithUpsertSink(AggregateITCase.scala:1620)
> Nov 25 05:02:47   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
> Nov 25 05:02:47   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 25 05:02:47   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 25 05:02:47   at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 25 05:02:47   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
> Nov 25 05:02:47   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
> Nov 25 05:02:47   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
> Nov 25 05:02:47   at 
> org.junit.platform.engine.support.hierarchical.Node

[jira] [Created] (FLINK-33660) AggregateITCase.testConstantGroupKeyWithUpsertSink due to unexpected behavior

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33660:
-

 Summary: AggregateITCase.testConstantGroupKeyWithUpsertSink due to 
unexpected behavior
 Key: FLINK-33660
 URL: https://issues.apache.org/jira/browse/FLINK-33660
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination, Table SQL / Runtime
Affects Versions: 1.19.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54895&view=logs&j=a9db68b9-a7e0-54b6-0f98-010e0aff39e2&t=cdd32e0b-6047-565b-c58f-14054472f1be&l=11063

{code}

Nov 25 05:02:47 05:02:47.850 [ERROR] 
org.apache.flink.table.planner.runtime.stream.sql.AggregateITCase.testConstantGroupKeyWithUpsertSink
  Time elapsed: 107.302 s  <<< ERROR!
Nov 25 05:02:47 java.util.concurrent.ExecutionException: 
org.apache.flink.table.api.TableException: Failed to wait job finish
Nov 25 05:02:47 at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
Nov 25 05:02:47 at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
Nov 25 05:02:47 at 
org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:118)
Nov 25 05:02:47 at 
org.apache.flink.table.api.internal.TableResultImpl.await(TableResultImpl.java:81)
Nov 25 05:02:47 at 
org.apache.flink.table.planner.runtime.stream.sql.AggregateITCase.testConstantGroupKeyWithUpsertSink(AggregateITCase.scala:1620)
Nov 25 05:02:47 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Nov 25 05:02:47 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Nov 25 05:02:47 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Nov 25 05:02:47 at java.lang.reflect.Method.invoke(Method.java:498)
Nov 25 05:02:47 at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
Nov 25 05:02:47 at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
Nov 25 05:02:47 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
Nov 25 05:02:47 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
Nov 25 05:02:47 at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Nov 25 05:02:47 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
Nov 25 05:02:47 at 
org.junit

[jira] [Reopened] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl reopened FLINK-31472:
---

I'm reopening this issue because it now also appeared in CI:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54895&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd&l=10265

> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
> 
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1
>Reporter: Ran Tao
>Priority: Major
>
> when run mvn clean test, this case failed occasionally.
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955 
> s <<< FAILURE! - in 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR] 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
>   Time elapsed: 0.492 s  <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be 
> called from inside the mailbox thread!
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
>         at 
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
>         at 
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>         at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>         at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>         at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>         at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>         at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>         at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
>         at 
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
>         at 
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
>         at 
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
>         at 
> org.juni

[jira] [Comment Edited] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789982#comment-17789982
 ] 

Matthias Pohl edited comment on FLINK-31472 at 11/27/23 9:00 AM:
-

I'm reopening this issue because it now also appeared in CI (master, i.e. 1.19):
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54895&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd&l=10265


was (Author: mapohl):
I'm reopening this issue because it now also appeared in CI:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54895&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd&l=10265

> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
> 
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Priority: Major
>  Labels: test-stability
>
> when run mvn clean test, this case failed occasionally.
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955 
> s <<< FAILURE! - in 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR] 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
>   Time elapsed: 0.492 s  <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be 
> called from inside the mailbox thread!
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
>         at 
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
>         at 
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>         at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>         at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>         at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>         at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>         at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>         at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
>         at 
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
>         at 
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
>         at 
> org.junit.vintage.engine.VintageTestEngi

[jira] [Updated] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl updated FLINK-31472:
--
Labels: test-stability  (was: )

> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
> 
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Priority: Major
>  Labels: test-stability
>
> when run mvn clean test, this case failed occasionally.
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955 
> s <<< FAILURE! - in 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR] 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
>   Time elapsed: 0.492 s  <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be 
> called from inside the mailbox thread!
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
>         at 
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
>         at 
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>         at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>         at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>         at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>         at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>         at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>         at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
>         at 
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
>         at 
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
>         at 
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
>         at 
> org.junit.platform.launcher.core

[jira] [Updated] (FLINK-31472) AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl updated FLINK-31472:
--
Affects Version/s: 1.19.0

> AsyncSinkWriterThrottlingTest failed with Illegal mailbox thread
> 
>
> Key: FLINK-31472
> URL: https://issues.apache.org/jira/browse/FLINK-31472
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common
>Affects Versions: 1.17.0, 1.16.1, 1.19.0
>Reporter: Ran Tao
>Priority: Major
>
> when run mvn clean test, this case failed occasionally.
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.955 
> s <<< FAILURE! - in 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest
> [ERROR] 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize
>   Time elapsed: 0.492 s  <<< ERROR!
> java.lang.IllegalStateException: Illegal thread detected. This method must be 
> called from inside the mailbox thread!
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.checkIsMailboxThread(TaskMailboxImpl.java:262)
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:137)
>         at 
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:84)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:367)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.lambda$registerCallback$3(AsyncSinkWriter.java:315)
>         at 
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$CallbackTask.onProcessingTime(TestProcessingTimeService.java:199)
>         at 
> org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.setCurrentTime(TestProcessingTimeService.java:76)
>         at 
> org.apache.flink.connector.base.sink.writer.AsyncSinkWriterThrottlingTest.testSinkThroughputShouldThrottleToHalfBatchSize(AsyncSinkWriterThrottlingTest.java:64)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>         at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>         at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>         at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>         at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>         at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>         at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
>         at 
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
>         at 
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
>         at 
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
>         at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.withIntercepted

[jira] [Updated] (FLINK-30132) Test LocalRecoveryITCase#testRecoverLocallyFromProcessCrashWithWorkingDirectory failed on azure due to File not exists

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-30132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl updated FLINK-30132:
--
Labels: test-stability  (was: )

> Test 
> LocalRecoveryITCase#testRecoverLocallyFromProcessCrashWithWorkingDirectory 
> failed on azure due to File not exists
> --
>
> Key: FLINK-30132
> URL: https://issues.apache.org/jira/browse/FLINK-30132
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.16.0
>Reporter: Leonard Xu
>Priority: Major
>  Labels: test-stability
>
> {noformat}
>   at 
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
>   at java.nio.file.Files.readAttributes(Files.java:1737)
>   at 
> java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
>   at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
>   at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
>   at java.nio.file.Files.walkFileTree(Files.java:2706)
>   at java.nio.file.Files.walkFileTree(Files.java:2742)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
>   ... 51 more
>   Suppressed: java.nio.file.NoSuchFileException: 
> /tmp/junit2010448393472419340/tm_taskManager_2/localState/aid_21c128b018cc61989c323cda6e36b0b1/jid_e5dbf7bc4ebb72baf20387e555083439/vtx_bc764cd8ddf7a0cff126f51c16239658_sti_1
>   at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>   at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>   at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>   at 
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.resetPermissionsAndTryToDeleteAgain(TempDirectory.java:250)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFileFailed(TempDirectory.java:212)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFileFailed(TempDirectory.java:199)
>   at java.nio.file.Files.walkFileTree(Files.java:2672)
>   ... 54 more
> Nov 21 19:52:57 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, 
> Time elapsed: 15.971 s <<< FAILURE! - in 
> org.apache.flink.test.recovery.LocalRecoveryITCase
> Nov 21 19:52:57 [ERROR] 
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
>   Time elapsed: 15.942 s  <<< ERROR!
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43366&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-30132) Test LocalRecoveryITCase#testRecoverLocallyFromProcessCrashWithWorkingDirectory failed on azure due to File not exists

[Matthias Pohl (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-30132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789983#comment-17789983
 ] 

Matthias Pohl commented on FLINK-30132:
---

Same issue in {{JoinITCase.testLeftOuterJoin}}: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54925&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4&l=11391

> Test 
> LocalRecoveryITCase#testRecoverLocallyFromProcessCrashWithWorkingDirectory 
> failed on azure due to File not exists
> --
>
> Key: FLINK-30132
> URL: https://issues.apache.org/jira/browse/FLINK-30132
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.16.0
>Reporter: Leonard Xu
>Priority: Major
>  Labels: test-stability
>
> {noformat}
>   at 
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
>   at java.nio.file.Files.readAttributes(Files.java:1737)
>   at 
> java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
>   at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
>   at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
>   at java.nio.file.Files.walkFileTree(Files.java:2706)
>   at java.nio.file.Files.walkFileTree(Files.java:2742)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
>   ... 51 more
>   Suppressed: java.nio.file.NoSuchFileException: 
> /tmp/junit2010448393472419340/tm_taskManager_2/localState/aid_21c128b018cc61989c323cda6e36b0b1/jid_e5dbf7bc4ebb72baf20387e555083439/vtx_bc764cd8ddf7a0cff126f51c16239658_sti_1
>   at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>   at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>   at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>   at 
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.resetPermissionsAndTryToDeleteAgain(TempDirectory.java:250)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFileFailed(TempDirectory.java:212)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFileFailed(TempDirectory.java:199)
>   at java.nio.file.Files.walkFileTree(Files.java:2672)
>   ... 54 more
> Nov 21 19:52:57 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, 
> Time elapsed: 15.971 s <<< FAILURE! - in 
> org.apache.flink.test.recovery.LocalRecoveryITCase
> Nov 21 19:52:57 [ERROR] 
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
>   Time elapsed: 15.942 s  <<< ERROR!
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43366&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-30132) Test LocalRecoveryITCase#testRecoverLocallyFromProcessCrashWithWorkingDirectory failed on azure due to File not exists

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-30132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl updated FLINK-30132:
--
Affects Version/s: 1.19.0

> Test 
> LocalRecoveryITCase#testRecoverLocallyFromProcessCrashWithWorkingDirectory 
> failed on azure due to File not exists
> --
>
> Key: FLINK-30132
> URL: https://issues.apache.org/jira/browse/FLINK-30132
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.16.0, 1.19.0
>Reporter: Leonard Xu
>Priority: Major
>  Labels: test-stability
>
> {noformat}
>   at 
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
>   at java.nio.file.Files.readAttributes(Files.java:1737)
>   at 
> java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
>   at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
>   at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
>   at java.nio.file.Files.walkFileTree(Files.java:2706)
>   at java.nio.file.Files.walkFileTree(Files.java:2742)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
>   ... 51 more
>   Suppressed: java.nio.file.NoSuchFileException: 
> /tmp/junit2010448393472419340/tm_taskManager_2/localState/aid_21c128b018cc61989c323cda6e36b0b1/jid_e5dbf7bc4ebb72baf20387e555083439/vtx_bc764cd8ddf7a0cff126f51c16239658_sti_1
>   at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>   at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>   at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>   at 
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.resetPermissionsAndTryToDeleteAgain(TempDirectory.java:250)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFileFailed(TempDirectory.java:212)
>   at 
> org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFileFailed(TempDirectory.java:199)
>   at java.nio.file.Files.walkFileTree(Files.java:2672)
>   ... 54 more
> Nov 21 19:52:57 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, 
> Time elapsed: 15.971 s <<< FAILURE! - in 
> org.apache.flink.test.recovery.LocalRecoveryITCase
> Nov 21 19:52:57 [ERROR] 
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
>   Time elapsed: 15.942 s  <<< ERROR!
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43366&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-25857) Add committer metrics to track the status of committables

[Leonard Xu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Leonard Xu updated FLINK-25857:
---
Description: 
With Sink V2 we can now track the progress of a committable during committing 
and show metrics about the committing status. (i.e. failed, retried, succeeded).


The voted FLIP 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink

  was:With Sink V2 we can now track the progress of a committable during 
committing and show metrics about the committing status. (i.e. failed, retried, 
succeeded).


> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-25857) Add committer metrics to track the status of committables

[Qingsheng Ren (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789989#comment-17789989
 ] 

Qingsheng Ren commented on FLINK-25857:
---

[~pvary] I'm afraid we have to do something to recover the backward 
compatibility, according to FLIP-321.

I take a look at the initial content of FLIP-371, and it did not mention any 
changes to SinkWriter#InitContext, but the implementation touched it. A 
discussion and voting thread is definitely a must-have if there is any 
modification to the original FLIP, because committers only gave +1 to the 
original design. 

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-25857) Add committer metrics to track the status of committables

[Weijie Guo (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789988#comment-17789988
 ] 

Weijie Guo commented on FLINK-25857:


[~pvary], Well, sorry, I didn't realize we already had a FLIP discussing this 
(FLIP-371).

In fact, my main concern is that the current approach seems to be inconsistent 
with the deprecation process for PublicEvolving API in FLIP-321[1]. In other 
words, this ticket is a zero migration period. I understand the diamond 
inheritance problem you pointed out, but I would have preferred to see a 
solution that was properly deprecated (which may require more subtle design).

I don't mean to question existing efforts, I just feel that we should do our 
best to comply with the work like FLIP-321 (which is probably also being pushed 
concurrently when you start this work, so not realizing it exists at first 
isn't your problem :) )

Maybe we need to take some time to think about whether we can find a win-win 
solution. After all, there is still plenty of time before the release of 1.19. 
If it is indeed impossible to implement, it's fine for me, but don't forget to 
remind the connector owner to release a compatible new version.

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[PR] [FLINK-33638] Support variable-length data generation for variable-length data types [flink]

[via GitHub]

liyubin117 opened a new pull request, #23810:
URL: https://github.com/apache/flink/pull/23810

   ## What is the purpose of the change
   
   Extending datagen to generate variable length values (using a new option to 
enable it, e.g.,'fields.f0.var-len'='true').
   
   ## Brief change log
   
   - introduce random length for variable-length types in RandomGeneratorVisitor
   
   ## Verifying this change
   
   
org.apache.flink.table.factories.DataGenTableSourceFactoryTest#testVariableLengthDataGeneration
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? yes
 - If yes, how is the feature documented? yes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-33638) Support variable-length data generation for variable-length data types

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-33638:
---
Labels: pull-request-available  (was: )

> Support variable-length data generation for variable-length data types
> --
>
> Key: FLINK-33638
> URL: https://issues.apache.org/jira/browse/FLINK-33638
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Common
>Affects Versions: 1.19.0
>Reporter: Yubin Li
>Assignee: Yubin Li
>Priority: Major
>  Labels: pull-request-available
>
> Currently, for variable-length data types (varchar, varbinary, string, 
> bytes), datagen connector always generates max-length data, we can extending 
> datagen to generate variable length values(using a new option to enable it, 
> e.g.,'fields.f0.var-len'='true').
> the topic has been discussed in the mail thread 
> [https://lists.apache.org/thread/kp6popo4cnhl6vx31sdn6mlscpzj9tgc]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33661) 'RocksDB Memory Management end-to-end test' failed due to unexpected error in logs

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33661:
-

 Summary: 'RocksDB Memory Management end-to-end test' failed due to 
unexpected error in logs
 Key: FLINK-33661
 URL: https://issues.apache.org/jira/browse/FLINK-33661
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Affects Versions: 1.19.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54942&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=43ba8ce7-ebbf-57cd-9163-444305d74117&l=5132

This seems to be the same issue as FLINK-30785.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33638] Support variable-length data generation for variable-length data types [flink]

[via GitHub]

flinkbot commented on PR #23810:
URL: https://github.com/apache/flink/pull/23810#issuecomment-1827450201

   
   ## CI report:
   
   * a693b54cd283381ccd0b5e8e79470e7592f8cc30 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33638] Support variable-length data generation for variable-length data types [flink]

[via GitHub]

liyubin117 commented on PR #23810:
URL: https://github.com/apache/flink/pull/23810#issuecomment-1827455391

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33418][test,ci] Uses getHost() to access HiveContainer (instead of hard-coded IP) [flink]

[via GitHub]

XComp merged PR #23649:
URL: https://github.com/apache/flink/pull/23649


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [BP-1.18][FLINK-33418][test] Uses getHost() [flink]

[via GitHub]

XComp merged PR #23650:
URL: https://github.com/apache/flink/pull/23650


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (FLINK-33418) SqlGatewayE2ECase failed due to ConnectException

[Matthias Pohl (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Pohl resolved FLINK-33418.
---
Fix Version/s: 1.19.0
   1.18.1
   Resolution: Fixed

master: 
[f6f785fd1a927ade00f91bf0de0f96a2268aa4e5|https://github.com/apache/flink/commit/f6f785fd1a927ade00f91bf0de0f96a2268aa4e5]
1.18: 
[e9548a6d420ed97e0b3d50523092a6663d624ab4|https://github.com/apache/flink/commit/e9548a6d420ed97e0b3d50523092a6663d624ab4]

> SqlGatewayE2ECase failed due to ConnectException
> 
>
> Key: FLINK-33418
> URL: https://issues.apache.org/jira/browse/FLINK-33418
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Client, Tests
>Affects Versions: 1.18.0, 1.19.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>  Labels: github-actions, pull-request-available, test-stability
> Fix For: 1.19.0, 1.18.1
>
>
> The container couldn't be started in [this 
> build|https://github.com/XComp/flink/actions/runs/6696839844/job/18195926497#step:15:11765]:
> {code}
> Error: 20:18:40 20:18:40.111 [ERROR] Tests run: 1, Failures: 0, Errors: 1, 
> Skipped: 0, Time elapsed: 110.789 s <<< FAILURE! - in 
> org.apache.flink.table.gateway.SqlGatewayE2ECase
> Error: 20:18:40 20:18:40.111 [ERROR] 
> org.apache.flink.table.gateway.SqlGatewayE2ECase  Time elapsed: 110.789 s  
> <<< ERROR!
> Oct 30 20:18:40 org.testcontainers.containers.ContainerLaunchException: 
> Container startup failed for image prestodb/hdp2.6-hive:10
> Oct 30 20:18:40   at 
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:349)
> Oct 30 20:18:40   at 
> org.apache.flink.table.gateway.containers.HiveContainer.doStart(HiveContainer.java:69)
> Oct 30 20:18:40   at 
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
> Oct 30 20:18:40   at 
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1131)
> Oct 30 20:18:40   at 
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
> Oct 30 20:18:40   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Oct 30 20:18:40   at 
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Oct 30 20:18:40   at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Oct 30 20:18:40   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Oct 30 20:18:40   at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Oct 30 20:18:40   at 
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> Oct 30 20:18:40   at 
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> Oct 30 20:18:40   at 
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
> Oct 30 20:18:40   at 
> org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
> Oct 30 20:18:40   at 
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
> Oct 30 20:18:40   at 
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
> Oct 30 20:18:40   at 
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
> Oct 30 20:18:40   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
> Oct 30 20:18:40   at 
> org.apache.maven.su

[jira] [Commented] (FLINK-27681) Improve the availability of Flink when the RocksDB file is corrupted.

[Yue Ma (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789994#comment-17789994
 ] 

Yue Ma commented on FLINK-27681:


[~masteryhx] Thanks for the explaining , I agree with it

> Improve the availability of Flink when the RocksDB file is corrupted.
> -
>
> Key: FLINK-27681
> URL: https://issues.apache.org/jira/browse/FLINK-27681
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / State Backends
>Reporter: Ming Li
>Assignee: Yue Ma
>Priority: Critical
>  Labels: pull-request-available
> Attachments: image-2023-08-23-15-06-16-717.png
>
>
> We have encountered several times when the RocksDB checksum does not match or 
> the block verification fails when the job is restored. The reason for this 
> situation is generally that there are some problems with the machine where 
> the task is located, which causes the files uploaded to HDFS to be incorrect, 
> but it has been a long time (a dozen minutes to half an hour) when we found 
> this problem. I'm not sure if anyone else has had a similar problem.
> Since this file is referenced by incremental checkpoints for a long time, 
> when the maximum number of checkpoints reserved is exceeded, we can only use 
> this file until it is no longer referenced. When the job failed, it cannot be 
> recovered.
> Therefore we consider:
> 1. Can RocksDB periodically check whether all files are correct and find the 
> problem in time?
> 2. Can Flink automatically roll back to the previous checkpoint when there is 
> a problem with the checkpoint data, because even with manual intervention, it 
> just tries to recover from the existing checkpoint or discard the entire 
> state.
> 3. Can we increase the maximum number of references to a file based on the 
> maximum number of checkpoints reserved? When the number of references exceeds 
> the maximum number of checkpoints -1, the Task side is required to upload a 
> new file for this reference. Not sure if this way will ensure that the new 
> file we upload will be correct.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33638] Support variable-length data generation for variable-length data types [flink]

[via GitHub]

liyubin117 commented on PR #23810:
URL: https://github.com/apache/flink/pull/23810#issuecomment-1827468541

   @lincoln-lil Looking forward your review, thanks :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-25857) Add committer metrics to track the status of committables

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789995#comment-17789995
 ] 

Leonard Xu commented on FLINK-25857:


I am very surprised that the public interface was removed without being 
discussed in the community dev mailing list. The previously voted FLIP-371[1] 
does not involve this modification. I fully understand that the implementation 
of some voted FLIPs may also involve modifications to the public interface, but 
at this time we should return to the dev mailing list to continue the 
discussion, rather than closing it directly after discussing it in the github 
PR. Moreover, even if it was discussed in github, [~jiabao.sun] had already 
pointed out this problem, but the PR was still merged without resolving 
jiabao's reasonable concerns. To be honest, I think this is incorrect both in 
terms of the community FLIP process and the PR review process.

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2] https://github.com/apache/flink/pull/23555/files#r1399035189

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33638] Support variable-length data generation for variable-length data types [flink]

[via GitHub]

liyubin117 commented on PR #23810:
URL: https://github.com/apache/flink/pull/23810#issuecomment-1827470955

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-25857) Add committer metrics to track the status of committables

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789995#comment-17789995
 ] 

Leonard Xu edited comment on FLINK-25857 at 11/27/23 9:36 AM:
--

[~pvary] I am very surprised that the public interface was removed without 
being discussed in the community dev mailing list. The previously voted 
FLIP-371[1] does not involve this modification. I fully understand that the 
implementation of some voted FLIPs may also involve modifications to the public 
interface, but at this time we should return to the dev mailing list to 
continue the discussion, rather than closing it directly after discussing it in 
the github PR. Moreover, even if it was discussed in github, [~jiabao.sun] had 
already pointed out this problem, but the PR was still merged without resolving 
jiabao's reasonable concerns. To be honest, I think this is incorrect both in 
terms of the community FLIP process and the PR review process.

CC: [~martijnvisser][~mbalassi][~danny.cranmer]

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2] https://github.com/apache/flink/pull/23555/files#r1399035189


was (Author: leonard xu):
I am very surprised that the public interface was removed without being 
discussed in the community dev mailing list. The previously voted FLIP-371[1] 
does not involve this modification. I fully understand that the implementation 
of some voted FLIPs may also involve modifications to the public interface, but 
at this time we should return to the dev mailing list to continue the 
discussion, rather than closing it directly after discussing it in the github 
PR. Moreover, even if it was discussed in github, [~jiabao.sun] had already 
pointed out this problem, but the PR was still merged without resolving 
jiabao's reasonable concerns. To be honest, I think this is incorrect both in 
terms of the community FLIP process and the PR review process.

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2] https://github.com/apache/flink/pull/23555/files#r1399035189

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-25857) Add committer metrics to track the status of committables

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789995#comment-17789995
 ] 

Leonard Xu edited comment on FLINK-25857 at 11/27/23 9:36 AM:
--

[~pvary] I am very surprised that the public interface was removed without 
being discussed in the community dev mailing list. The previously voted 
FLIP-371[1] does not involve this modification. I fully understand that the 
implementation of some voted FLIPs may also involve modifications to the public 
interface, but at this time we should return to the dev mailing list to 
continue the discussion, rather than closing it directly after discussing it in 
the github PR. Moreover, even if it was discussed in github, [~jiabao.sun] had 
already pointed out this problem, but the PR was still merged without resolving 
jiabao's reasonable concerns. To be honest, I think this is incorrect both in 
terms of the community FLIP process and the PR review process.

CC: [~martijnvisser] [~mbalassi] [~danny.cranmer]

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2] https://github.com/apache/flink/pull/23555/files#r1399035189


was (Author: leonard xu):
[~pvary] I am very surprised that the public interface was removed without 
being discussed in the community dev mailing list. The previously voted 
FLIP-371[1] does not involve this modification. I fully understand that the 
implementation of some voted FLIPs may also involve modifications to the public 
interface, but at this time we should return to the dev mailing list to 
continue the discussion, rather than closing it directly after discussing it in 
the github PR. Moreover, even if it was discussed in github, [~jiabao.sun] had 
already pointed out this problem, but the PR was still merged without resolving 
jiabao's reasonable concerns. To be honest, I think this is incorrect both in 
terms of the community FLIP process and the PR review process.

CC: [~martijnvisser][~mbalassi][~danny.cranmer]

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2] https://github.com/apache/flink/pull/23555/files#r1399035189

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-25857) Add committer metrics to track the status of committables

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789995#comment-17789995
 ] 

Leonard Xu edited comment on FLINK-25857 at 11/27/23 9:37 AM:
--

[~pvary] I am very surprised that the public interface was removed without 
being discussed in the community dev mailing list. The previously voted 
FLIP-371[1] does not involve this modification. I fully understand that the 
implementation of some voted FLIPs may also involve modifications to the public 
interface, but at this time we should return to the dev mailing list to 
continue the discussion, rather than closing it directly after discussing it in 
the github PR. Moreover, even if it was discussed in github, [~jiabao.sun] had 
already pointed out this problem[2], but the PR was still merged without 
resolving jiabao's reasonable concerns. To be honest, I think this is incorrect 
both in terms of the community FLIP process and the PR review process.

CC: [~martijnvisser] [~mbalassi] [~danny.cranmer]

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2] https://github.com/apache/flink/pull/23555/files#r1399035189


was (Author: leonard xu):
[~pvary] I am very surprised that the public interface was removed without 
being discussed in the community dev mailing list. The previously voted 
FLIP-371[1] does not involve this modification. I fully understand that the 
implementation of some voted FLIPs may also involve modifications to the public 
interface, but at this time we should return to the dev mailing list to 
continue the discussion, rather than closing it directly after discussing it in 
the github PR. Moreover, even if it was discussed in github, [~jiabao.sun] had 
already pointed out this problem, but the PR was still merged without resolving 
jiabao's reasonable concerns. To be honest, I think this is incorrect both in 
terms of the community FLIP process and the PR review process.

CC: [~martijnvisser] [~mbalassi] [~danny.cranmer]

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2] https://github.com/apache/flink/pull/23555/files#r1399035189

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (FLINK-25857) Add committer metrics to track the status of committables

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789995#comment-17789995
 ] 

Leonard Xu edited comment on FLINK-25857 at 11/27/23 9:39 AM:
--

[~pvary] I am very surprised that the public interface was removed without 
being discussed in the community dev mailing list. The previously voted 
FLIP-371[1] does not involve this modification. I fully understand that the 
implementation of some voted FLIPs may also involve modifications to the public 
interface, but at this time we should return to the dev mailing list to 
continue the discussion, rather than closing it directly after discussing it in 
the github PR. Moreover, even if it was discussed in github, [~jiabao.sun] had 
already pointed out this problem[2], but the PR was still merged without 
resolving jiabao's reasonable concerns. To be honest, I think this is incorrect 
both in terms of the community FLIP process and the PR review process.

CC: [~martijnvisser] [~mbalassi] [~danny.cranmer]

[1]  
https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
[2] https://github.com/apache/flink/pull/23555/files#r1399035189


was (Author: leonard xu):
[~pvary] I am very surprised that the public interface was removed without 
being discussed in the community dev mailing list. The previously voted 
FLIP-371[1] does not involve this modification. I fully understand that the 
implementation of some voted FLIPs may also involve modifications to the public 
interface, but at this time we should return to the dev mailing list to 
continue the discussion, rather than closing it directly after discussing it in 
the github PR. Moreover, even if it was discussed in github, [~jiabao.sun] had 
already pointed out this problem[2], but the PR was still merged without 
resolving jiabao's reasonable concerns. To be honest, I think this is incorrect 
both in terms of the community FLIP process and the PR review process.

CC: [~martijnvisser] [~mbalassi] [~danny.cranmer]

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2] https://github.com/apache/flink/pull/23555/files#r1399035189

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] Bump h2 from 2.1.210 to 2.2.220 in /flink-connector-jdbc [flink-connector-jdbc]

[via GitHub]

eskabetxe commented on PR #69:
URL: 
https://github.com/apache/flink-connector-jdbc/pull/69#issuecomment-1827482345

   @dependabot rebase


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-33635) Some connectors can not compile in 1.19-SNAPSHOT

[Leonard Xu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Leonard Xu updated FLINK-33635:
---
Fix Version/s: 1.19.0

> Some connectors can not compile in 1.19-SNAPSHOT
> 
>
> Key: FLINK-33635
> URL: https://issues.apache.org/jira/browse/FLINK-33635
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Blocker
> Fix For: 1.19.0
>
>
> The sink API compatibility was broken in FLINK-25857. 
> org.apache.flink.api.connector.sink2.Sink#createWriter(InitContext) was 
> changed to 
> org.apache.flink.api.connector.sink2.Sink#createWriter(WriterInitContext).
> All external connectors sink can not compile as this change.
> For example:
> es: 
> https://github.com/apache/flink-connector-elasticsearch/actions/runs/6976181890/job/18984287421
> aws: 
> https://github.com/apache/flink-connector-aws/actions/runs/6975253086/job/18982104160



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-33635) Some connectors can not compile in 1.19-SNAPSHOT

[Leonard Xu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Leonard Xu updated FLINK-33635:
---
Affects Version/s: 1.19.0

> Some connectors can not compile in 1.19-SNAPSHOT
> 
>
> Key: FLINK-33635
> URL: https://issues.apache.org/jira/browse/FLINK-33635
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common
>Affects Versions: 1.19.0
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Blocker
> Fix For: 1.19.0
>
>
> The sink API compatibility was broken in FLINK-25857. 
> org.apache.flink.api.connector.sink2.Sink#createWriter(InitContext) was 
> changed to 
> org.apache.flink.api.connector.sink2.Sink#createWriter(WriterInitContext).
> All external connectors sink can not compile as this change.
> For example:
> es: 
> https://github.com/apache/flink-connector-elasticsearch/actions/runs/6976181890/job/18984287421
> aws: 
> https://github.com/apache/flink-connector-aws/actions/runs/6975253086/job/18982104160



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] Bump h2 from 2.1.210 to 2.2.220 in /flink-connector-jdbc [flink-connector-jdbc]

[via GitHub]

dependabot[bot] commented on PR #69:
URL: 
https://github.com/apache/flink-connector-jdbc/pull/69#issuecomment-1827482407

   Sorry, only users with push access can use that command.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33628][runtime] Upgrades default ZooKeeper version from 3.7.1 to 3.8.3 [flink]

[via GitHub]

XComp commented on PR #23784:
URL: https://github.com/apache/flink/pull/23784#issuecomment-1827488207

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-33635) Some connectors can not compile in 1.19-SNAPSHOT

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789998#comment-17789998
 ] 

Leonard Xu commented on FLINK-33635:


A blocker must be fixed in 1.19, my attitude here: 
https://issues.apache.org/jira/browse/FLINK-25857?focusedCommentId=17789995&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17789995

> Some connectors can not compile in 1.19-SNAPSHOT
> 
>
> Key: FLINK-33635
> URL: https://issues.apache.org/jira/browse/FLINK-33635
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common
>Affects Versions: 1.19.0
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Blocker
> Fix For: 1.19.0
>
>
> The sink API compatibility was broken in FLINK-25857. 
> org.apache.flink.api.connector.sink2.Sink#createWriter(InitContext) was 
> changed to 
> org.apache.flink.api.connector.sink2.Sink#createWriter(WriterInitContext).
> All external connectors sink can not compile as this change.
> For example:
> es: 
> https://github.com/apache/flink-connector-elasticsearch/actions/runs/6976181890/job/18984287421
> aws: 
> https://github.com/apache/flink-connector-aws/actions/runs/6975253086/job/18982104160



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-30593] Add tests for ScalingTracking#removeOldRecords() [flink-kubernetes-operator]

[via GitHub]

mxm commented on PR #715:
URL: 
https://github.com/apache/flink-kubernetes-operator/pull/715#issuecomment-1827493243

   > @mxm is this the right ticket? 
[[FLINK-33572](https://issues.apache.org/jira/browse/FLINK-33572)] Minimize 
ConfigMap API operations for autoscaler state
   
   Next time, please add the corresponding ticket number when you open a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-30593] Add tests for ScalingTracking#removeOldRecords() [flink-kubernetes-operator]

[via GitHub]

mxm commented on code in PR #715:
URL: 
https://github.com/apache/flink-kubernetes-operator/pull/715#discussion_r1405904882


##
flink-autoscaler/src/test/java/org/apache/flink/autoscaler/ScalingTrackingTest.java:
##
@@ -52,6 +52,7 @@ void setUp() {
 conf.set(AutoScalerOptions.PREFER_TRACKED_RESTART_TIME, true);
 }
 
+// - getMaxRestartTimeOrDefault -

Review Comment:
   I would advise against these kind of comments. Similarly to JavaDocs, they 
tend to add more confusion than actually helping new people. Plus, they become 
outdated quickly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-31339][tests] Fix unstable tests of flink-end-to-end-tests-sql module [flink]

[via GitHub]

ruanhang1993 commented on code in PR #23507:
URL: https://github.com/apache/flink/pull/23507#discussion_r1405905020


##
flink-end-to-end-tests/flink-end-to-end-tests-sql/src/test/java/org/apache/flink/table/sql/codegen/SqlITCaseBase.java:
##
@@ -190,4 +231,76 @@ private static List readResultFiles(Path path) 
throws IOException {
 }
 return result;
 }
+
+protected List formatRawResult(List rawResults) {

Review Comment:
   Add some docs for this method on how to use or override it.



##
flink-end-to-end-tests/flink-end-to-end-tests-sql/src/test/java/org/apache/flink/table/sql/codegen/SqlITCaseBase.java:
##
@@ -190,4 +231,76 @@ private static List readResultFiles(Path path) 
throws IOException {
 }
 return result;
 }
+
+protected List formatRawResult(List rawResults) {
+return rawResults;
+}
+
+protected static List convertToMaterializedResult(
+List rawResults,
+ResolvedSchema schema,
+DeserializationSchema deserializationSchema) {
+DataCollector collector = new DataCollector();
+try {
+deserializationSchema.open(new TestingDeserializationContext());
+for (String rawResult : rawResults) {
+deserializationSchema.deserialize(rawResult.getBytes(), 
collector);
+}
+} catch (Exception e) {
+fail("deserialize error: ", e);
+}
+
+RowRowConverter converter = 
RowRowConverter.create(schema.toPhysicalRowDataType());
+Map upsertResult = new HashMap<>();
+
+for (RowData rowData : collector.dataList) {
+RowKind kind = rowData.getRowKind();
+
+Row row = converter.toExternal(rowData);
+assertThat(row).isNotNull();
+
+Row key = Row.project(row, schema.getPrimaryKeyIndexes());
+key.setKind(RowKind.INSERT);
+
+Row upsertRow = Row.copy(row);
+upsertRow.setKind(RowKind.INSERT);
+
+if (kind == RowKind.INSERT || kind == RowKind.UPDATE_AFTER) {
+upsertResult.put(key, upsertRow);
+} else {
+Row oldValue = upsertResult.remove(key);
+if (oldValue == null) {
+throw new RuntimeException(
+"Tried to delete a value that wasn't inserted 
first. "
++ "This is probably an incorrectly 
implemented test.");
+}
+}
+}
+
+return 
upsertResult.values().stream().map(Row::toString).collect(Collectors.toList());
+}
+
+protected static DebeziumJsonDeserializationSchema 
createDebeziumDeserializationSchema(
+ResolvedSchema schema) {
+return new DebeziumJsonDeserializationSchema(
+schema.toPhysicalRowDataType(),
+Collections.emptyList(),
+
InternalTypeInfo.of(schema.toPhysicalRowDataType().getLogicalType()),
+false,
+true,
+TimestampFormat.ISO_8601);
+}
+

Review Comment:
   add some java docs



##
flink-end-to-end-tests/flink-end-to-end-tests-sql/src/test/java/org/apache/flink/table/sql/codegen/SqlITCaseBase.java:
##
@@ -190,4 +231,76 @@ private static List readResultFiles(Path path) 
throws IOException {
 }
 return result;
 }
+
+protected List formatRawResult(List rawResults) {
+return rawResults;
+}
+
+protected static List convertToMaterializedResult(
+List rawResults,
+ResolvedSchema schema,
+DeserializationSchema deserializationSchema) {
+DataCollector collector = new DataCollector();
+try {
+deserializationSchema.open(new TestingDeserializationContext());
+for (String rawResult : rawResults) {
+deserializationSchema.deserialize(rawResult.getBytes(), 
collector);
+}
+} catch (Exception e) {
+fail("deserialize error: ", e);
+}
+
+RowRowConverter converter = 
RowRowConverter.create(schema.toPhysicalRowDataType());
+Map upsertResult = new HashMap<>();
+
+for (RowData rowData : collector.dataList) {
+RowKind kind = rowData.getRowKind();
+
+Row row = converter.toExternal(rowData);
+assertThat(row).isNotNull();
+
+Row key = Row.project(row, schema.getPrimaryKeyIndexes());
+key.setKind(RowKind.INSERT);
+
+Row upsertRow = Row.copy(row);
+upsertRow.setKind(RowKind.INSERT);

Review Comment:
   `upsertRow` is only useful when the kind is insert or update_after. So we 
could create it when `kind == RowKind.INSERT || kind == RowKind.UPDATE_AFTER`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific commen

[jira] [Created] (FLINK-33662) Bump com.h2database:h2

[Martijn Visser (Jira)]

Martijn Visser created FLINK-33662:
--

 Summary: Bump com.h2database:h2
 Key: FLINK-33662
 URL: https://issues.apache.org/jira/browse/FLINK-33662
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / JDBC
Reporter: Martijn Visser
Assignee: Martijn Visser






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33638] Support variable-length data generation for variable-length data types [flink]

[via GitHub]

liyubin117 commented on PR #23810:
URL: https://github.com/apache/flink/pull/23810#issuecomment-1827514306

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-32881][checkpoint] Support triggering savepoint in detach mode for CLI and dumping all pending savepoint ids by rest api [flink]

[via GitHub]

xiangforever2014 commented on code in PR #23253:
URL: https://github.com/apache/flink/pull/23253#discussion_r1405754517


##
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java:
##
@@ -157,6 +157,14 @@ public class CliFrontendParser {
 + " for changing state backends, native = a 
specific format for the"
 + " chosen state backend, might be faster to take 
and restore from.");
 
+public static final Option SAVEPOINT_DETACH_OPTION =

Review Comment:
   Thanks for your comment, I have updated the related docs manually.



##
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java:
##
@@ -157,6 +157,14 @@ public class CliFrontendParser {
 + " for changing state backends, native = a 
specific format for the"
 + " chosen state backend, might be faster to take 
and restore from.");
 
+public static final Option SAVEPOINT_DETACH_OPTION =
+new Option(
+"dcp",

Review Comment:
   These is a "-d" option for savepoints already(SAVEPOINT_DISPOSE_OPTION), so 
I think we can not just simply reuse the existing detached 
option(DETACHED_OPTION) since it's short option is also "-d", if we force to 
reuse the option, we need to modify either SAVEPOINT_DISPOSE_OPTION or 
DETACHED_OPTION,  which may cause some influence to existing users, so I think 
it's better to add a new option to support detached savepoint, WDYT?



##
flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java:
##
@@ -178,6 +178,23 @@ CompletableFuture stopWithSavepoint(
 CompletableFuture triggerSavepoint(
 JobID jobId, @Nullable String savepointDirectory, 
SavepointFormatType formatType);
 
+/**
+ * Triggers a detach savepoint for the job identified by the job id. The 
savepoint will be
+ * written to the given savepoint directory, or {@link
+ * 
org.apache.flink.configuration.CheckpointingOptions#SAVEPOINT_DIRECTORY} if it 
is null.
+ * Notice that: detach savepoint will return with a savepoint trigger id 
instead of the path
+ * future, that means the client will return very quickly.
+ *
+ * @param jobId job id
+ * @param savepointDirectory directory the savepoint should be written to
+ * @param formatType a binary format of the savepoint
+ * @return The savepoint trigger id
+ */
+default CompletableFuture triggerDetachSavepoint(
+JobID jobId, @Nullable String savepointDirectory, 
SavepointFormatType formatType) {
+return triggerSavepoint(jobId, savepointDirectory, formatType);

Review Comment:
   Thanks for your insightful comments, the reason we add the default method 
here is that we only want to implement the detached mode for rest api in this 
commit. I have resolved this comment, PTAL.



##
flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java:
##
@@ -178,6 +178,23 @@ CompletableFuture stopWithSavepoint(
 CompletableFuture triggerSavepoint(
 JobID jobId, @Nullable String savepointDirectory, 
SavepointFormatType formatType);
 
+/**
+ * Triggers a detach savepoint for the job identified by the job id. The 
savepoint will be

Review Comment:
   Thanks for your observant comments, resolved~



##
flink-clients/src/main/java/org/apache/flink/client/program/rest/RestClusterClient.java:
##
@@ -608,6 +616,27 @@ private CompletableFuture triggerSavepoint(
 });
 }
 
+private CompletableFuture triggerDetachSavepoint(
+final JobID jobId,
+final @Nullable String savepointDirectory,
+final boolean cancelJob,
+final SavepointFormatType formatType) {
+final SavepointTriggerHeaders savepointTriggerHeaders =

Review Comment:
   Thanks for your comment, I have unified the detached/non-detached savepoint 
to one method~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33459][Connector/JDBC] Support the new source that keeps the same functionality as the original JDBC input format [flink-connector-jdbc]

[via GitHub]

eskabetxe commented on code in PR #78:
URL: 
https://github.com/apache/flink-connector-jdbc/pull/78#discussion_r1405917676


##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/source/Utils.java:
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.jdbc.source;
+
+import org.apache.flink.connector.jdbc.source.split.CheckpointedOffset;
+import org.apache.flink.connector.jdbc.source.split.JdbcSourceSplit;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+
+/** Utils class to hold common static methods. */
+public class Utils {
+private Utils() {}
+
+public static void serializeJdbcSourceSplit(DataOutputStream out, 
JdbcSourceSplit sourceSplit)

Review Comment:
   I would say that this should be on JdbcSourceSplitSerializer.java, and we 
should instantiate that on JdbcSourceEnumStateSerializer.
   This because any change to this code will affect the state and should be 
versioned, and is more easy to understand if is there.
   
   Other option is to have this on JdbcSourceSplit, but I think that the same 
problem of versioning will happen.
   
   But we should avoid this kind of utility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-25857) Add committer metrics to track the status of committables

[Martijn Visser (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790006#comment-17790006
 ] 

Martijn Visser commented on FLINK-25857:


I think everyone is already on the same page (that it should have been brought 
back to the Dev mailing list) so let's just do that to complete the discussion 
there

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-25857) Add committer metrics to track the status of committables

[Jira]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790007#comment-17790007
 ] 

Márton Balassi commented on FLINK-25857:


Hi team,

Thanks for looking into this. I am working with [~pvary] on this effort and 
acknowledge it that we deviated from the original design of FLIP-371, strictly 
based on the comments we received during the PR.

[~leonard] : [~jiabao.sun] raised his concern *after* I merged the PR, I would 
not have merged the PR with such an important concern not being answered.

Please allow a couple days for [~pvary] and myself to gather all input and 
weigh our options. This runs deeper than this specific change, which we can 
return to the original design - this change ended up being a test for both 
FLIP-321 (which does not seem to be fully implemented even though the vote has 
passed) and the connector externalization. More important than the specifics we 
need to find a consensus for these (the decisions can be decoupled).

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-30593] Add tests for ScalingTracking#removeOldRecords() [flink-kubernetes-operator]

[via GitHub]

afedulov commented on code in PR #715:
URL: 
https://github.com/apache/flink-kubernetes-operator/pull/715#discussion_r1405924492


##
flink-autoscaler/src/test/java/org/apache/flink/autoscaler/ScalingTrackingTest.java:
##
@@ -52,6 +52,7 @@ void setUp() {
 conf.set(AutoScalerOptions.PREFER_TRACKED_RESTART_TIME, true);
 }
 
+// - getMaxRestartTimeOrDefault -

Review Comment:
   removed...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-32881][checkpoint] Support triggering savepoint in detach mode for CLI and dumping all pending savepoint ids by rest api [flink]

[via GitHub]

xiangforever2014 commented on PR #23253:
URL: https://github.com/apache/flink/pull/23253#issuecomment-1827533221

   @masteryhx Many thanks for your insightful comments, I have updated the code 
according to the comments, PTAL, thanks again bro~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33459][Connector/JDBC] Support the new source that keeps the same functionality as the original JDBC input format [flink-connector-jdbc]

[via GitHub]

eskabetxe commented on code in PR #78:
URL: 
https://github.com/apache/flink-connector-jdbc/pull/78#discussion_r1405924661


##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/source/JdbcSource.java:
##
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.jdbc.source;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeinfo.TypeHint;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import 
org.apache.flink.connector.jdbc.internal.connection.JdbcConnectionProvider;
+import 
org.apache.flink.connector.jdbc.source.enumerator.JdbcSourceEnumStateSerializer;
+import org.apache.flink.connector.jdbc.source.enumerator.JdbcSourceEnumerator;
+import 
org.apache.flink.connector.jdbc.source.enumerator.JdbcSourceEnumeratorState;
+import 
org.apache.flink.connector.jdbc.source.enumerator.JdbcSqlSplitEnumeratorBase;
+import org.apache.flink.connector.jdbc.source.reader.JdbcSourceReader;
+import org.apache.flink.connector.jdbc.source.reader.JdbcSourceSplitReader;
+import org.apache.flink.connector.jdbc.source.reader.extractor.ResultExtractor;
+import org.apache.flink.connector.jdbc.source.split.JdbcSourceSplit;
+import org.apache.flink.connector.jdbc.source.split.JdbcSourceSplitSerializer;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Objects;
+
+/** JDBC source. */
+@Internal

Review Comment:
   I think this should be PublicEvolving



##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/source/JdbcSourceBuilder.java:
##
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.jdbc.source;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.connector.jdbc.JdbcConnectionOptions;
+import 
org.apache.flink.connector.jdbc.internal.connection.JdbcConnectionProvider;
+import 
org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider;
+import 
org.apache.flink.connector.jdbc.source.enumerator.SqlTemplateSplitEnumerator;
+import org.apache.flink.connector.jdbc.source.reader.extractor.ResultExtractor;
+import org.apache.flink.connector.jdbc.split.JdbcParameterValuesProvider;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.StringUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.sql.ResultSet;
+
+import static 
org.apach

Re: [PR] [FLINK-33459][Connector/JDBC] Support the new source that keeps the same functionality as the original JDBC input format [flink-connector-jdbc]

[via GitHub]

eskabetxe commented on code in PR #78:
URL: 
https://github.com/apache/flink-connector-jdbc/pull/78#discussion_r1405924661


##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/source/JdbcSource.java:
##
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.jdbc.source;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeinfo.TypeHint;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import 
org.apache.flink.connector.jdbc.internal.connection.JdbcConnectionProvider;
+import 
org.apache.flink.connector.jdbc.source.enumerator.JdbcSourceEnumStateSerializer;
+import org.apache.flink.connector.jdbc.source.enumerator.JdbcSourceEnumerator;
+import 
org.apache.flink.connector.jdbc.source.enumerator.JdbcSourceEnumeratorState;
+import 
org.apache.flink.connector.jdbc.source.enumerator.JdbcSqlSplitEnumeratorBase;
+import org.apache.flink.connector.jdbc.source.reader.JdbcSourceReader;
+import org.apache.flink.connector.jdbc.source.reader.JdbcSourceSplitReader;
+import org.apache.flink.connector.jdbc.source.reader.extractor.ResultExtractor;
+import org.apache.flink.connector.jdbc.source.split.JdbcSourceSplit;
+import org.apache.flink.connector.jdbc.source.split.JdbcSourceSplitSerializer;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Objects;
+
+/** JDBC source. */
+@Internal

Review Comment:
   should not be PublicEvolving?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-27681) Improve the availability of Flink when the RocksDB file is corrupted.

[Rui Fan (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790010#comment-17790010
 ] 

Rui Fan commented on FLINK-27681:
-

{quote} * That's why we'd like to focus on not uploading the corruped files 
(Also for just fail the job simply to make job restore from the last complete 
checkpoint).{quote}
Fail job directly is fine for me, but I guess the PR doesn't fail the job, it 
just fails the current checkpoint, right?

 
{quote}File corruption will not affect the read path because the checksum will 
be checked when reading rocksdb block. The job will failover when read the 
corrupted one.
{quote}
If the checksum is called for each reading, can we think the check is very 
quick? If so, could we enable it directly without any option? Hey [~mayuehappy] 
 , could you provide some simple benchmark here?

> Improve the availability of Flink when the RocksDB file is corrupted.
> -
>
> Key: FLINK-27681
> URL: https://issues.apache.org/jira/browse/FLINK-27681
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / State Backends
>Reporter: Ming Li
>Assignee: Yue Ma
>Priority: Critical
>  Labels: pull-request-available
> Attachments: image-2023-08-23-15-06-16-717.png
>
>
> We have encountered several times when the RocksDB checksum does not match or 
> the block verification fails when the job is restored. The reason for this 
> situation is generally that there are some problems with the machine where 
> the task is located, which causes the files uploaded to HDFS to be incorrect, 
> but it has been a long time (a dozen minutes to half an hour) when we found 
> this problem. I'm not sure if anyone else has had a similar problem.
> Since this file is referenced by incremental checkpoints for a long time, 
> when the maximum number of checkpoints reserved is exceeded, we can only use 
> this file until it is no longer referenced. When the job failed, it cannot be 
> recovered.
> Therefore we consider:
> 1. Can RocksDB periodically check whether all files are correct and find the 
> problem in time?
> 2. Can Flink automatically roll back to the previous checkpoint when there is 
> a problem with the checkpoint data, because even with manual intervention, it 
> just tries to recover from the existing checkpoint or discard the entire 
> state.
> 3. Can we increase the maximum number of references to a file based on the 
> maximum number of checkpoints reserved? When the number of references exceeds 
> the maximum number of checkpoints -1, the Task side is required to upload a 
> new file for this reference. Not sure if this way will ensure that the new 
> file we upload will be correct.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-25857) Add committer metrics to track the status of committables

[Leonard Xu (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790011#comment-17790011
 ] 

Leonard Xu commented on FLINK-25857:


> Leonard Xu : Jiabao Sun raised his concern after I merged the PR, I would not 
> have merged the PR with such an important concern not being answered.

Thanks [~mbalassi] for the quick response, the `after` word let me better than 
before, let's bring the discussion to dev mailing list and find a way to 
resolve the blocker ASAP.

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] FLINK-28050][connectors] Introduce FLIP-27 alternative to StreamExecutionEnvironment#fromElements() [flink]

[via GitHub]

zentol merged PR #23553:
URL: https://github.com/apache/flink/pull/23553


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] FLINK-28050][connectors] Introduce FLIP-27 alternative to StreamExecutionEnvironment#fromElements() [flink]

[via GitHub]

zentol commented on code in PR #23553:
URL: https://github.com/apache/flink/pull/23553#discussion_r1405936668


##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:
##
@@ -1185,8 +1295,10 @@ public DataStreamSource fromSequence(long from, 
long to) {
  * @param data The array of elements to create the data stream from.
  * @param  The type of the returned data stream
  * @return The data stream representing the given array of elements
+ * @deprecated Use {@link #fromData(OUT...)} instead

Review Comment:
   eh fair enough, Ill merge it as is. 🤷 



##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:
##
@@ -1185,8 +1295,10 @@ public DataStreamSource fromSequence(long from, 
long to) {
  * @param data The array of elements to create the data stream from.
  * @param  The type of the returned data stream
  * @return The data stream representing the given array of elements
+ * @deprecated Use {@link #fromData(OUT...)} instead

Review Comment:
   eh fair enough, Ill merge it as is. 🤷 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-25857] Add committer metrics to track the status of committables [flink]

[via GitHub]

MartijnVisser commented on code in PR #23555:
URL: https://github.com/apache/flink/pull/23555#discussion_r1399126249


##
flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java:
##
@@ -55,17 +53,11 @@ public interface Sink extends Serializable {
  * @return A sink writer.
  * @throws IOException for any failure during creation.
  */
-SinkWriter createWriter(InitContext context) throws IOException;
+SinkWriter createWriter(WriterInitContext context) throws 
IOException;
 
 /** The interface exposes some runtime info for creating a {@link 
SinkWriter}. */
 @PublicEvolving
-interface InitContext {
-/**
- * The first checkpoint id when an application is started and not 
recovered from a
- * previously taken checkpoint or savepoint.
- */
-long INITIAL_CHECKPOINT_ID = 1;
-
+interface WriterInitContext extends InitContext {

Review Comment:
   That's why we had a FLIP and a vote @Jiabao-Sun 
   
   Edit: in hindsight, we didn't have a vote on this specific change, so we 
should have brought it back to the ML



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-28050) Introduce Source API alternative to SourceExecutionContext#fromElements(*) methods

[Chesnay Schepler (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-28050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-28050:
-
Fix Version/s: 1.19.0

> Introduce Source API alternative to SourceExecutionContext#fromElements(*) 
> methods
> --
>
> Key: FLINK-28050
> URL: https://issues.apache.org/jira/browse/FLINK-28050
> Project: Flink
>  Issue Type: Sub-task
>  Components: API / DataStream
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Closed] (FLINK-28050) Introduce Source API alternative to SourceExecutionContext#fromElements(*) methods

[Chesnay Schepler (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-28050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-28050.

Resolution: Fixed

master:
68437b937a60c647abe1a4104289849c006b8fe7..d351c5bd9c1f28a3e5ffe98fb549c1b94618485b

> Introduce Source API alternative to SourceExecutionContext#fromElements(*) 
> methods
> --
>
> Key: FLINK-28050
> URL: https://issues.apache.org/jira/browse/FLINK-28050
> Project: Flink
>  Issue Type: Sub-task
>  Components: API / DataStream
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>  Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-25857) Add committer metrics to track the status of committables

[Martijn Visser (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790019#comment-17790019
 ] 

Martijn Visser commented on FLINK-25857:


Hey all, I've opened 
https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57 to move the 
discussion back to the ML. Let's continue this good discussion there :)

> Add committer metrics to track the status of committables
> -
>
> Key: FLINK-25857
> URL: https://issues.apache.org/jira/browse/FLINK-25857
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common
>Reporter: Fabian Paul
>Assignee: Peter Vary
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-10-20-17-23-09-595.png, screenshot-1.png
>
>
> With Sink V2 we can now track the progress of a committable during committing 
> and show metrics about the committing status. (i.e. failed, retried, 
> succeeded).
> The voted FLIP 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-25857] Add committer metrics to track the status of committables [flink]

[via GitHub]

MartijnVisser commented on code in PR #23555:
URL: https://github.com/apache/flink/pull/23555#discussion_r1399126249


##
flink-core/src/main/java/org/apache/flink/api/connector/sink2/Sink.java:
##
@@ -55,17 +53,11 @@ public interface Sink extends Serializable {
  * @return A sink writer.
  * @throws IOException for any failure during creation.
  */
-SinkWriter createWriter(InitContext context) throws IOException;
+SinkWriter createWriter(WriterInitContext context) throws 
IOException;
 
 /** The interface exposes some runtime info for creating a {@link 
SinkWriter}. */
 @PublicEvolving
-interface InitContext {
-/**
- * The first checkpoint id when an application is started and not 
recovered from a
- * previously taken checkpoint or savepoint.
- */
-long INITIAL_CHECKPOINT_ID = 1;
-
+interface WriterInitContext extends InitContext {

Review Comment:
   That's why we had a FLIP and a vote @Jiabao-Sun 
   
   Edit: in hindsight, we didn't have a vote on breaking this specific public 
API change, so we should have brought it back to the ML



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-33659) Avoid unnecessary retries when restore from savepoint failed.

[zhouli (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhouli closed FLINK-33659.
--
Resolution: Invalid

> Avoid unnecessary retries when restore from savepoint failed.
> -
>
> Key: FLINK-33659
> URL: https://issues.apache.org/jira/browse/FLINK-33659
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing
>Reporter: zhouli
>Priority: Major
>
> when restore a job from savepoint failed, if restart strategy is enabled, 
> flink will try to restart the job, and the restore would fail again.
> We may wrap the exception as 
> [SuppressRestartsException|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/execution/SuppressRestartsException.java]
>  to avoid unnecessary retries.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] (FLINK-33659) Avoid unnecessary retries when restore from savepoint failed.

[zhouli (Jira)]

[ https://issues.apache.org/jira/browse/FLINK-33659 ]


zhouli deleted comment on FLINK-33659:


was (Author: leo zhou):
Hi, [~zhuzh] , wdyt ?

> Avoid unnecessary retries when restore from savepoint failed.
> -
>
> Key: FLINK-33659
> URL: https://issues.apache.org/jira/browse/FLINK-33659
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing
>Reporter: zhouli
>Priority: Major
>
> when restore a job from savepoint failed, if restart strategy is enabled, 
> flink will try to restart the job, and the restore would fail again.
> We may wrap the exception as 
> [SuppressRestartsException|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/execution/SuppressRestartsException.java]
>  to avoid unnecessary retries.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33663) Serialize CallExpressions into SQL

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-33663:


 Summary: Serialize CallExpressions into SQL
 Key: FLINK-33663
 URL: https://issues.apache.org/jira/browse/FLINK-33663
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.19.0


The task is about introducing {{CallSyntax}} and implementing versions for 
non-standard SQL functions



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] Bump h2 from 2.1.210 to 2.2.220 in /flink-connector-jdbc [flink-connector-jdbc]

[via GitHub]

MartijnVisser merged PR #69:
URL: https://github.com/apache/flink-connector-jdbc/pull/69


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Bump org.apache.derby:derby from 10.14.2.0 to 10.17.1.0 in /flink-connector-jdbc [flink-connector-jdbc]

[via GitHub]

dependabot[bot] opened a new pull request, #80:
URL: https://github.com/apache/flink-connector-jdbc/pull/80

   Bumps org.apache.derby:derby from 10.14.2.0 to 10.17.1.0.
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.derby:derby&package-manager=maven&previous-version=10.14.2.0&new-version=10.17.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot show  ignore conditions` will show all of 
the ignore conditions of the specified dependency
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 
upgrade to it yourself)
   You can disable automated security fix PRs for this repo from the [Security 
Alerts page](https://github.com/apache/flink-connector-jdbc/network/alerts).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Bump com.google.guava:guava from 31.1-jre to 32.0.0-jre [flink-connector-jdbc]

[via GitHub]

dependabot[bot] opened a new pull request, #81:
URL: https://github.com/apache/flink-connector-jdbc/pull/81

   Bumps [com.google.guava:guava](https://github.com/google/guava) from 
31.1-jre to 32.0.0-jre.
   
   Release notes
   Sourced from https://github.com/google/guava/releases";>com.google.guava:guava's 
releases.
   
   32.0.0
   Maven
   
 com.google.guava
 guava
 32.0.0-jre
 
 32.0.0-android
   
   
   Jar files
   
   https://repo1.maven.org/maven2/com/google/guava/guava/32.0.0-jre/guava-32.0.0-jre.jar";>32.0.0-jre.jar
   https://repo1.maven.org/maven2/com/google/guava/guava/32.0.0-android/guava-32.0.0-android.jar";>32.0.0-android.jar
   
   Guava requires https://github.com/google/guava/wiki/UseGuavaInYourBuild#what-about-guavas-own-dependencies";>one
 runtime dependency, which you can download here:
   
   https://repo1.maven.org/maven2/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar";>failureaccess-1.0.1.jar
   
   Javadoc
   
   http://guava.dev/releases/32.0.0-jre/api/docs/";>32.0.0-jre
   http://guava.dev/releases/32.0.0-android/api/docs/";>32.0.0-android
   
   JDiff
   
   http://guava.dev/releases/32.0.0-jre/api/diffs/";>32.0.0-jre vs. 
31.1-jre
   http://guava.dev/releases/32.0.0-android/api/diffs/";>32.0.0-android vs. 
31.1-android
   http://guava.dev/releases/32.0.0-android/api/androiddiffs/";>32.0.0-android
 vs. 32.0.0-jre
   
   Changelog
   Security fixes
   
   Reimplemented Files.createTempDir and 
FileBackedOutputStream to further address CVE-2020-8908 (https://redirect.github.com/google/guava/issues/4011";>#4011) and 
CVE-2023-2976 (https://redirect.github.com/google/guava/issues/2575";>#2575). 
(feb83a1c8f)
   
   While CVE-2020-8908 was officially closed when we deprecated 
Files.createTempDir in https://github.com/google/guava/releases/tag/v30.0";>Guava 30.0, we've 
heard from users that even recent versions of Guava have been listed as 
vulnerable in other databases of security vulnerabilities. In 
response, we've reimplemented the method (and the very rarely used 
FileBackedOutputStream class, which had a similar issue) to 
eliminate the insecure behavior entirely. This change could technically affect 
users in a number of different ways (discussed under "Incompatible 
changes" below), but in practice, the only problem users are likely to 
encounter is with Windows. If you are using those APIs under Windows, you 
should skip 32.0.0 and go straight to https://github.com/google/guava/releases/tag/v32.0.1";>32.0.1 which 
fixes the problem. (Unfortunately, we didn't think of the Windows problem until 
after the release. And while w
 e https://github.com/google/guava#important-warnings";>warn that 
common.io in particular may not work under Windows, we didn't 
intend to regress support.) Sorry for the trouble.
   Incompatible changes
   Although this release bumps Guava's major version number, it makes 
no binary-incompatible changes to the guava 
artifact.
   One change could cause issues for Widows users, and a few other changes 
could cause issues for users in more usual situations:
   
   The new implementations of Files.createTempDir and 
FileBackedOutputStream https://redirect.github.com/google/guava/issues/6535";>throw an exception 
under Windows. This is fixed in https://github.com/google/guava/releases/tag/v32.0.1";>32.0.1. Sorry 
for the trouble.
   guava-gwt now https://redirect.github.com/google/guava/issues/6627";>requires GWT https://github.com/gwtproject/gwt/releases/tag/2.10.0";>2.10.0.
   This release makes a binary-incompatible change to a @Beta 
API in the separate artifact guava-testlib. 
Specifically, we changed the return type of 
TestingExecutors.sameThreadScheduledExecutor to 
ListeningScheduledExecutorService. The old return type was a 
package-private class, which caused the Kotlin compiler to produce warnings. 
(dafaa3e435)
   
   
   
   ... (truncated)
   
   
   Commits
   
   See full diff in https://github.com/google/guava/commits";>compare view
   
   
   
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.google.guava:guava&package-manager=maven&previous-version=31.1-jre&new-version=32.0.0-jre)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overw

Re: [PR] [FLINK-30593] Add tests for ScalingTracking#removeOldRecords() [flink-kubernetes-operator]

[via GitHub]

mxm merged PR #715:
URL: https://github.com/apache/flink-kubernetes-operator/pull/715


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-33662) Bump com.h2database:h2

[Martijn Visser (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-33662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martijn Visser closed FLINK-33662.
--
Fix Version/s: jdbc-3.2.0
   Resolution: Fixed

Fixed in apache/flink-connector-jdbc:main 
b477d452ba3aac38d53d1f5d4c4820bdad3ad9cd

> Bump com.h2database:h2
> --
>
> Key: FLINK-33662
> URL: https://issues.apache.org/jira/browse/FLINK-33662
> Project: Flink
>  Issue Type: Technical Debt
>  Components: Connectors / JDBC
>Reporter: Martijn Visser
>Assignee: Martijn Visser
>Priority: Major
> Fix For: jdbc-3.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] Support Flink 1.17.1 [flink-statefun]

[via GitHub]

MartijnVisser commented on code in PR #332:
URL: https://github.com/apache/flink-statefun/pull/332#discussion_r1405970736


##
pom.xml:
##
@@ -76,15 +76,28 @@ under the License.
 1.8
 1.20.0
 1.0-rc6
-3.7.1
-2.3.2
-
3.11.1
-1.15.2
+3.23.2
+2.6.2
+
3.11.4
+1.17.1
 2.12
 2.12.7
 1.8.0
-
2.12.4-15.0
+
2.14.2-17.0

Review Comment:
   This is the wrong version for Flink 1.17.1



##
tools/docker/Dockerfile:
##
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM apache/flink:1.15.2-scala_2.12-java8
+FROM flink:1.17.1-scala_2.12-java11

Review Comment:
   Statefun still requires Java 8, so I don't think we should bump it as part 
of this PR



##
statefun-flink/statefun-flink-io-bundle/pom.xml:
##
@@ -90,7 +90,12 @@ under the License.
 
 org.apache.flink
 flink-connector-kinesis
-${flink.version}
+${flink-connector-kinesis.version}
+
+
+org.apache.flink
+flink-connector-aws-kinesis-streams
+${flink-connector-aws-kinesis-streams.version}

Review Comment:
   Is this necessary for the Flink upgrade itself?



##
pom.xml:
##
@@ -234,10 +311,12 @@ under the License.
 
 descriptor
 
${basedir}/target/test-classes
+main

Review Comment:
   Is this needed for the 1.171. upgrade?



##
pom.xml:
##
@@ -129,7 +147,66 @@ under the License.
 
 com.fasterxml.jackson.core
 jackson-databind
-2.13.2.2
+${jackson-databind.version}
+
+
+
+org.apache.flink
+flink-shaded-netty
+${flink-shaded-netty.version}
+
+
+
+org.apache.flink
+flink-core
+${flink.version}
+
+
+
+org.apache.flink
+flink-table-common
+${flink.version}
+
+
+
+org.apache.flink
+flink-connector-base
+${flink.version}
+
+
+
+org.apache.flink
+flink-shaded-force-shading
+${flink-shaded-force-shading.version}
+
+
+
+org.apache.flink
+flink-shaded-jackson
+${flink-shaded-jackson.version}
+
+
+
+commons-codec
+commons-codec
+${commons-codec.version}
+
+
+
+commons-logging
+commons-logging
+${commons-logging.version}
+
+
+
+org.slf4j
+slf4j-api
+${slf4j-api.version}

Review Comment:
   Why are we adding these? That shouldn't be necessary in an upgrade to 1.17.1



##
pom.xml:
##
@@ -76,15 +76,28 @@ under the License.
 1.8
 1.20.0
 1.0-rc6
-3.7.1
-2.3.2
-
3.11.1
-1.15.2
+3.23.2
+2.6.2
+
3.11.4
+1.17.1
 2.12
 2.12.7
 1.8.0
-
2.12.4-15.0
+
2.14.2-17.0
 1.7.32
+
4.1.0-1.17
+
4.1.0-1.17
+3.14.6
+
4.1.82.Final-16.1
+4.12
+1.3
+2.24.0
+2.13.2.2
+
4.1.82.Final-16.1
+
16.1
+1.15
+1.2
+1.7.36

Review Comment:
   It doesn't appear as we re-use these variables anywhere else, should we just 
leave them as they were? 



##
statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/StatefulFunctionsConfig.java:
##
@@ -265,7 +265,7 @@ public void setEmbedded(boolean embedded) {
*/
   public StatefulFunctionsUniverseProvider getProvider(ClassLoader cl) {
 try {
-  return 
InstantiationUtil.deserializeObject(universeInitializerClassBytes, cl, false);
+  return 
InstantiationUtil.deserializeObject(universeInitializerClassBytes, cl);

Review Comment:
   Is this needed for the Flink upgrade? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Support Flink 1.17.1 [flink-statefun]

[via GitHub]

MartijnVisser commented on code in PR #332:
URL: https://github.com/apache/flink-statefun/pull/332#discussion_r1405987733


##
statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/StatefulFunctionsConfig.java:
##
@@ -265,7 +265,7 @@ public void setEmbedded(boolean embedded) {
*/
   public StatefulFunctionsUniverseProvider getProvider(ClassLoader cl) {
 try {
-  return 
InstantiationUtil.deserializeObject(universeInitializerClassBytes, cl, false);
+  return 
InstantiationUtil.deserializeObject(universeInitializerClassBytes, cl);

Review Comment:
   Edit: yes it is :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [FLINK-33638] Support variable-length data generation for variable-length data types [flink]

[via GitHub]

liyubin117 commented on PR #23810:
URL: https://github.com/apache/flink/pull/23810#issuecomment-1827611717

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-28051) Promote ExternallyInducedSourceReader to non-experimental @Public

[Alexander Fedulov (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-28051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790032#comment-17790032
 ] 

Alexander Fedulov commented on FLINK-28051:
---

Hi [~Brian Zhou], thanks a lot for the clarification. Looking forward to 
contributions from the Pravega community.

> Promote ExternallyInducedSourceReader to non-experimental @Public
> -
>
> Key: FLINK-28051
> URL: https://issues.apache.org/jira/browse/FLINK-28051
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common, Tests
>Reporter: Alexander Fedulov
>Priority: Major
>
> It needs to be evaluated if ExternallyInducedSourceReader can be promoted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-28051) Promote ExternallyInducedSourceReader to non-experimental @Public

[Alexander Fedulov (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-28051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790034#comment-17790034
 ] 

Alexander Fedulov commented on FLINK-28051:
---

[~Brian Zhou] can I assign this ticket to you or one of your colleagues?

> Promote ExternallyInducedSourceReader to non-experimental @Public
> -
>
> Key: FLINK-28051
> URL: https://issues.apache.org/jira/browse/FLINK-28051
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Common, Tests
>Reporter: Alexander Fedulov
>Priority: Major
>
> It needs to be evaluated if ExternallyInducedSourceReader can be promoted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] [FLINK-33638] Support variable-length data generation for variable-length data types [flink]

[via GitHub]

liyubin117 commented on PR #23810:
URL: https://github.com/apache/flink/pull/23810#issuecomment-1827646121

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] [FLINK-33663] Serialize CallExpressions into SQL [flink]

[via GitHub]

dawidwys opened a new pull request, #23811:
URL: https://github.com/apache/flink/pull/23811

   ## What is the purpose of the change
   
   This PR adds a `CallSyntax` as described in the FLIP and implementations for 
specific non standard builtin functions.
   
   
   ## Brief change log
   
   * adds CallSyntax interface and changes in BuiltinFunctionDefinitions
   * implements call syntax for functions in BuiltinFunctionDefinitions
   
   ## Verifying this change
   
   Added tests in org.apache.flink.table.expressions.ExpressionSerializationTest
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (**yes** / no)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't 
know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (**yes** / no)
 - If yes, how is the feature documented? (not applicable / docs / 
**JavaDocs** / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org