[jira] [Commented] (FLINK-37346) [CI] Unit Tests (mysql) / compile_and_test (8, generic) (pull_request)Failing after 60m
[ https://issues.apache.org/jira/browse/FLINK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928246#comment-17928246 ] Yanquan Lv commented on FLINK-37346: I'm trying to find out the cause. > [CI] Unit Tests (mysql) / compile_and_test (8, generic) (pull_request)Failing > after 60m > --- > > Key: FLINK-37346 > URL: https://issues.apache.org/jira/browse/FLINK-37346 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.3.0 >Reporter: Yanquan Lv >Priority: Major > Fix For: cdc-3.4.0 > > > There is a test timeout in master branch now. > Refer to the following cases: > [1] > [https://github.com/apache/flink-cdc/actions/runs/13381521564/job/37378173507] > [2] > [https://github.com/apache/flink-cdc/actions/runs/13390330565/job/37400625838?pr=3868] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-37346) [CI] Unit Tests (mysql) / compile_and_test (8, generic) (pull_request)Failing after 60m
Yanquan Lv created FLINK-37346: -- Summary: [CI] Unit Tests (mysql) / compile_and_test (8, generic) (pull_request)Failing after 60m Key: FLINK-37346 URL: https://issues.apache.org/jira/browse/FLINK-37346 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.3.0 Reporter: Yanquan Lv Fix For: cdc-3.4.0 There is a test timeout in master branch now. Refer to the following cases: [1] [https://github.com/apache/flink-cdc/actions/runs/13381521564/job/37378173507] [2] [https://github.com/apache/flink-cdc/actions/runs/13390330565/job/37400625838?pr=3868] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-37334) Fix mysql's add column after event becomes add column last
[
https://issues.apache.org/jira/browse/FLINK-37334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MOBIN closed FLINK-37334.
-
Resolution: Not A Bug
> Fix mysql's add column after event becomes add column last
> --
>
> Key: FLINK-37334
> URL: https://issues.apache.org/jira/browse/FLINK-37334
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: cdc-3.3.0
>Reporter: MOBIN
>Assignee: MOBIN
>Priority: Major
>
> Repeat step：
> {color:#80}{color}
> {code:java}
> ALTER TABLE cdc_sink19
> add COLUMN a1_after1 bigint after a1 {code}
> log: [add column after event becomes add column last]
> 2025-02-17 14:49:22,184 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 1 - Start deducing evolved schema change for
> AddColumnEvent{tableId=dw_app.cdc_sink19,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}AFTER{color}, existedColumnName=a1}]}
> 2025-02-17 14:49:22,184 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 2 - Affected downstream tables are: [rt_ods.cdc_sink19_add_column]
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 3.1 - For to-be-evolved table rt_ods.cdc_sink19_add_column with schema
> columns=\{`id` DECIMAL(20, 0) NOT NULL '主键ID',`a1` VARCHAR(50),`a1_after`
> BIGINT,`ab2` BIGINT,`table_name` STRING NOT NULL}, primaryKeys=id,
> options=()...
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 3.2 - upstream dependency tables are: [dw_app.cdc_sink19]
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 3.3 - It's an one-by-one routing and could be forwarded as
> AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}AFTER{color}, existedColumnName=a1}]}.
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 4 - After being normalized with LENIENT behavior, final schema change
> events are: [AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}LAST{color}, existedColumnName=null}]}]
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> All sink subtask have flushed for table dw_app.cdc_sink19. Start to apply
> schema change request:
> SchemaChangeRequest{tableId=dw_app.cdc_sink19,
> schemaChangeEvent=AddColumnEvent{tableId=dw_app.cdc_sink19,
> addedColumns=[ColumnWithPosition
> {column=`a1_after1` BIGINT, position=AFTER, existedColumnName=a1}
> ]}, subTaskId=0}
> that extracts to:
> AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}LAST{color}, existedColumnName=null}]}
> 2025-02-17 14:49:22,307 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Successfully applied schema change event
> AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}LAST{color}, existedColumnName=null}]} to external
> system.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-37334) Fix mysql's add column after event becomes add column last
[
https://issues.apache.org/jira/browse/FLINK-37334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928248#comment-17928248
]
MOBIN commented on FLINK-37334:
---
sorry，because I used old version, closed
> Fix mysql's add column after event becomes add column last
> --
>
> Key: FLINK-37334
> URL: https://issues.apache.org/jira/browse/FLINK-37334
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: cdc-3.3.0
>Reporter: MOBIN
>Assignee: MOBIN
>Priority: Major
>
> Repeat step：
> {color:#80}{color}
> {code:java}
> ALTER TABLE cdc_sink19
> add COLUMN a1_after1 bigint after a1 {code}
> log: [add column after event becomes add column last]
> 2025-02-17 14:49:22,184 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 1 - Start deducing evolved schema change for
> AddColumnEvent{tableId=dw_app.cdc_sink19,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}AFTER{color}, existedColumnName=a1}]}
> 2025-02-17 14:49:22,184 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 2 - Affected downstream tables are: [rt_ods.cdc_sink19_add_column]
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 3.1 - For to-be-evolved table rt_ods.cdc_sink19_add_column with schema
> columns=\{`id` DECIMAL(20, 0) NOT NULL '主键ID',`a1` VARCHAR(50),`a1_after`
> BIGINT,`ab2` BIGINT,`table_name` STRING NOT NULL}, primaryKeys=id,
> options=()...
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 3.2 - upstream dependency tables are: [dw_app.cdc_sink19]
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 3.3 - It's an one-by-one routing and could be forwarded as
> AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}AFTER{color}, existedColumnName=a1}]}.
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Step 4 - After being normalized with LENIENT behavior, final schema change
> events are: [AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}LAST{color}, existedColumnName=null}]}]
> 2025-02-17 14:49:22,185 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> All sink subtask have flushed for table dw_app.cdc_sink19. Start to apply
> schema change request:
> SchemaChangeRequest{tableId=dw_app.cdc_sink19,
> schemaChangeEvent=AddColumnEvent{tableId=dw_app.cdc_sink19,
> addedColumns=[ColumnWithPosition
> {column=`a1_after1` BIGINT, position=AFTER, existedColumnName=a1}
> ]}, subTaskId=0}
> that extracts to:
> AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}LAST{color}, existedColumnName=null}]}
> 2025-02-17 14:49:22,307 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> Successfully applied schema change event
> AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column,
> addedColumns=[ColumnWithPosition{column=`a1_after1` BIGINT,
> position={color:#ff}LAST{color}, existedColumnName=null}]} to external
> system.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
[ https://issues.apache.org/jira/browse/FLINK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928251#comment-17928251 ] RocMarshal commented on FLINK-33463: Hi, [~kunni] Thanks for your attention on it. The ticket was planned to completed in the next two versions at least. Please let me try to explain the reason. The Stream API-level feature is not yet fully user-facing, and we are still gathering user feedback on its stability and usability. Therefore, we plan to consider merging the SQL-level API changes in a future version, such as JDBC 3.3.1, after the release of JDBC 3.3.0 and if no negative feedback is received. By the way, this MR was previously merged, but we have reverted this feature for the reasons mentioned above. Thank you very much! If any assistance or validation is needed regarding FLIP-239 Source, I’d be happy to help. > Support the implementation of dynamic source tables based on the new source > --- > > Key: FLINK-33463 > URL: https://issues.apache.org/jira/browse/FLINK-33463 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / JDBC >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > Fix For: jdbc-3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35810) AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry fails
[ https://issues.apache.org/jira/browse/FLINK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35810: --- Affects Version/s: 2.0.0 2.1.0 > AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry > fails > > > Key: FLINK-35810 > URL: https://issues.apache.org/jira/browse/FLINK-35810 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 2.0.0, 2.0-preview, 2.1.0 >Reporter: Rui Fan >Priority: Blocker > Labels: test-stability > > AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry > fails > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-36290) OutOfMemoryError in connect test run
[
https://issues.apache.org/jira/browse/FLINK-36290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928252#comment-17928252
]
Weijie Guo commented on FLINK-36290:
Downgrade to critical due to lack of recurrence recent. But the heap oom came
up frequently in another test(FLINK-35810), upgrade it to a blocker.
> OutOfMemoryError in connect test run
>
>
> Key: FLINK-36290
> URL: https://issues.apache.org/jira/browse/FLINK-36290
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Tests
>Affects Versions: 2.0-preview
>Reporter: Matthias Pohl
>Assignee: Weijie Guo
>Priority: Critical
> Fix For: 2.0.0
>
>
> We saw a OOM in the connect stage that's caused a fatal error:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=62173&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd&l=12182
> {code}
> 03:19:59,975 [ flink-scheduler-1] ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'flink-scheduler-1' produced an uncaught exception. Stopping the
> process...
> java.lang.OutOfMemoryError: Java heap space
> [...]
> 03:19:59,981 [jobmanager_62-main-scheduler-thread-1] ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'jobmanager_62-main-scheduler-thread-1' produced an uncaught
> exception. Stopping the process...
> java.lang.OutOfMemoryError: Java heap space
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35810) AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry fails
[
https://issues.apache.org/jira/browse/FLINK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-35810:
---
Description:
AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry fails
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159]
I always saw an OOM before this test failure.
{code:java}
Jul 25 04:58:33 java.lang.OutOfMemoryError: Java heap space
Jul 25 04:58:33 Dumping heap to java_pid23540.hprof ...
Jul 25 04:58:34 Heap dump file created [36427163 bytes in 0.226 secs]{code}
was:
AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry fails
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> fails
>
>
> Key: FLINK-35810
> URL: https://issues.apache.org/jira/browse/FLINK-35810
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.0.0, 2.0-preview, 2.1.0
>Reporter: Rui Fan
>Priority: Blocker
> Labels: test-stability
>
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> fails
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159]
>
> I always saw an OOM before this test failure.
> {code:java}
> Jul 25 04:58:33 java.lang.OutOfMemoryError: Java heap space
> Jul 25 04:58:33 Dumping heap to java_pid23540.hprof ...
> Jul 25 04:58:34 Heap dump file created [36427163 bytes in 0.226 secs]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35810) OOM in AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
[
https://issues.apache.org/jira/browse/FLINK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo reassigned FLINK-35810:
--
Assignee: Weijie Guo
> OOM in
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> -
>
> Key: FLINK-35810
> URL: https://issues.apache.org/jira/browse/FLINK-35810
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.0.0, 2.0-preview, 2.1.0
>Reporter: Rui Fan
>Assignee: Weijie Guo
>Priority: Blocker
> Labels: test-stability
>
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> fails
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159]
>
> I always saw an OOM before this test failure.
> {code:java}
> Jul 25 04:58:33 java.lang.OutOfMemoryError: Java heap space
> Jul 25 04:58:33 Dumping heap to java_pid23540.hprof ...
> Jul 25 04:58:34 Heap dump file created [36427163 bytes in 0.226 secs]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35810) OOM in AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
[
https://issues.apache.org/jira/browse/FLINK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-35810:
---
Summary: OOM in
AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
(was:
AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
fails)
> OOM in
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> -
>
> Key: FLINK-35810
> URL: https://issues.apache.org/jira/browse/FLINK-35810
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.0.0, 2.0-preview, 2.1.0
>Reporter: Rui Fan
>Priority: Blocker
> Labels: test-stability
>
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> fails
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159]
>
> I always saw an OOM before this test failure.
> {code:java}
> Jul 25 04:58:33 java.lang.OutOfMemoryError: Java heap space
> Jul 25 04:58:33 Dumping heap to java_pid23540.hprof ...
> Jul 25 04:58:34 Heap dump file created [36427163 bytes in 0.226 secs]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-35810][test] increase unittest Xmx to debug FLINK-35810 [flink]
reswqa opened a new pull request, #26174: URL: https://github.com/apache/flink/pull/26174 I download the heap dump file in failed build. But don't find any suspicious leak objects. Slightly increase the memory limit to help troubleshoot this problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35810) OOM in AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
[
https://issues.apache.org/jira/browse/FLINK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35810:
---
Labels: pull-request-available test-stability (was: test-stability)
> OOM in
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> -
>
> Key: FLINK-35810
> URL: https://issues.apache.org/jira/browse/FLINK-35810
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.0.0, 2.0-preview, 2.1.0
>Reporter: Rui Fan
>Assignee: Weijie Guo
>Priority: Blocker
> Labels: pull-request-available, test-stability
>
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> fails
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159]
>
> I always saw an OOM before this test failure.
> {code:java}
> Jul 25 04:58:33 java.lang.OutOfMemoryError: Java heap space
> Jul 25 04:58:33 Dumping heap to java_pid23540.hprof ...
> Jul 25 04:58:34 Heap dump file created [36427163 bytes in 0.226 secs]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-36290) OutOfMemoryError in connect test run
[
https://issues.apache.org/jira/browse/FLINK-36290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-36290:
---
Priority: Critical (was: Blocker)
> OutOfMemoryError in connect test run
>
>
> Key: FLINK-36290
> URL: https://issues.apache.org/jira/browse/FLINK-36290
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Tests
>Affects Versions: 2.0-preview
>Reporter: Matthias Pohl
>Assignee: Weijie Guo
>Priority: Critical
> Fix For: 2.0.0
>
>
> We saw a OOM in the connect stage that's caused a fatal error:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=62173&view=logs&j=1c002d28-a73d-5309-26ee-10036d8476b4&t=d1c117a6-8f13-5466-55f0-d48dbb767fcd&l=12182
> {code}
> 03:19:59,975 [ flink-scheduler-1] ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'flink-scheduler-1' produced an uncaught exception. Stopping the
> process...
> java.lang.OutOfMemoryError: Java heap space
> [...]
> 03:19:59,981 [jobmanager_62-main-scheduler-thread-1] ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'jobmanager_62-main-scheduler-thread-1' produced an uncaught
> exception. Stopping the process...
> java.lang.OutOfMemoryError: Java heap space
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35810) AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry fails
[ https://issues.apache.org/jira/browse/FLINK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35810: --- Priority: Blocker (was: Critical) > AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry > fails > > > Key: FLINK-35810 > URL: https://issues.apache.org/jira/browse/FLINK-35810 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 2.0-preview >Reporter: Rui Fan >Priority: Blocker > Labels: test-stability > > AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry > fails > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-36409] Publish some autoscaler metrics during stabilisation period [flink-kubernetes-operator]
morhidi opened a new pull request, #945: URL: https://github.com/apache/flink-kubernetes-operator/pull/945 ## What is the purpose of the change Currently autoscaler metrics although collected are not published during stabilization period. We report metrics after the stabilization period only. In practice this could result in larger gaps in metric charts during scale operations that makes it hard for end users to interpret. The metrics could appear to be broken, especially when multiple scale operations executed in a row, for example: https://github.com/user-attachments/assets/e533cfc1-b712-498a-b9d9-3b3a2b4a700a"; /> This change mitigates this issue by shortening the gaps in reported metrics. The collected metrics won't be withhold during stabilization period either. ## Brief change log - Removing the logic to report no metrics during stabilization - Adjusted the logging to better understand the stabilization/metric window periods - Update the timestamp format to contain millis (unit tests operate with millis) ``` 2025-02-17 16:56:13,948 o.a.f.a.ScalingMetricCollector [INFO ] Stabilizing... until 1969-12-31 16:00:00.100. 1 samples collected 2025-02-17 16:56:13,952 o.a.f.a.ScalingMetricCollector [INFO ] Stabilizing... until 1969-12-31 16:00:00.100. 2 samples collected 2025-02-17 16:56:13,957 o.a.f.a.ScalingMetricCollector [INFO ] Metric window is not full until 1969-12-31 16:00:00.250. 3 samples collected 2025-02-17 16:56:13,958 o.a.f.a.ScalingMetricCollector [INFO ] Metric window is not full until 1969-12-31 16:00:00.250. 4 samples collected 2025-02-17 16:56:13,960 o.a.f.a.ScalingMetricCollector [INFO ] Metric window is now full. Dropped 3 samples before 1969-12-31 16:00:00.160, keeping 2. ``` ## Verifying this change Updated existing unit tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no - Core observer or reconciler logic that is regularly executed: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-36409) Publish some autoscaler metrics during stabilisation period
[ https://issues.apache.org/jira/browse/FLINK-36409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-36409: --- Labels: pull-request-available (was: ) > Publish some autoscaler metrics during stabilisation period > --- > > Key: FLINK-36409 > URL: https://issues.apache.org/jira/browse/FLINK-36409 > Project: Flink > Issue Type: Bug > Components: Autoscaler, Kubernetes Operator >Affects Versions: kubernetes-operator-1.9.0 >Reporter: Gyula Fora >Assignee: Matyas Orhidi >Priority: Major > Labels: pull-request-available > > Currently autoscaler metrics are not published at all during stabilisation > period. This makes sense for most metrics as they rely on evaluation logic > but for some such as PARALLELISM we should always publish them so users have > metric continuity. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-37152][build] Update Flink version to 1.20 [flink-cdc]
lvyanquan commented on PR #3868: URL: https://github.com/apache/flink-cdc/pull/3868#issuecomment-2667370988 > Well, exchanged the `mongodb7` failure to a `mysql` one.. :) Will check on it tomorrow I think that the failure of mysql is unrelated to the change here, because this error also occurs on the master branch, I will also try to identify the cause of this problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37333][runtime] Use ForwardForUnspecifiedPartitioner when adaptive broadcast join takes effect. [flink]
zhuzhurk commented on code in PR #26163:
URL: https://github.com/apache/flink/pull/26163#discussion_r1960865281
##
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/strategy/AdaptiveBroadcastJoinOptimizationStrategy.java:
##
@@ -162,7 +162,22 @@ protected void tryOptimizeAdaptiveJoin(
broadcastThreshold,
leftInputSize,
rightInputSize);
+} else {
+LOG.info(
+"Modified stream edges for the join node [{}]
failed.",
Review Comment:
maybe "Modification to stream edges for the join node [{}] failed. Keep the
join node as is."
This is to prevent users from misinterpreting the situation as an unexpected
error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-37313] Fix the problem of reading binlog before the high and low watermarks during the snapshot process [flink-cdc]
lvyanquan commented on PR #3920: URL: https://github.com/apache/flink-cdc/pull/3920#issuecomment-2667375370 Hi, @gongzexin. I think that the problem that you met was resolved in https://github.com/apache/flink-cdc/pull/3902 as https://github.com/apache/flink-cdc/pull/3902#issuecomment-2639121416 describes, can you check for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-36203) Have some way to control the polling frequency of GetRecords from SplitFetcher
[ https://issues.apache.org/jira/browse/FLINK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-36203: --- Labels: pull-request-available (was: ) > Have some way to control the polling frequency of GetRecords from SplitFetcher > -- > > Key: FLINK-36203 > URL: https://issues.apache.org/jira/browse/FLINK-36203 > Project: Flink > Issue Type: Improvement > Components: Connectors / DynamoDB >Reporter: Abhi Gupta >Priority: Minor > Labels: pull-request-available > > There is no polling frequency of GetRecords from the connector today. We > should have some way to control how fast / slow we poll from GetRecords -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-37044) Support connectors and formats as plugins
[ https://issues.apache.org/jira/browse/FLINK-37044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928266#comment-17928266 ] Karan Makhija commented on FLINK-37044: --- Thank you. I will check it out. > Support connectors and formats as plugins > - > > Key: FLINK-37044 > URL: https://issues.apache.org/jira/browse/FLINK-37044 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.20.0, 1.19.1, 2.0-preview >Reporter: Karan Makhija >Priority: Minor > > According to the documentation, Plugin based approach works with file systems > and metric reporters. It would help to extend it to Formats and Connectors. > Relevant links: > [https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/filesystems/plugins/] > https://issues.apache.org/jira/browse/FLINK-11952 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-37349) Add timeout for test case that call waitForSinkSize method to avoid Infinite waiting
Yanquan Lv created FLINK-37349:
--
Summary: Add timeout for test case that call waitForSinkSize
method to avoid Infinite waiting
Key: FLINK-37349
URL: https://issues.apache.org/jira/browse/FLINK-37349
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Affects Versions: cdc-3.3.0
Reporter: Yanquan Lv
Fix For: cdc-3.4.0
Add timeout for test case that call waitForSinkSize method:
{code:java}
protected static void waitForSinkSize(String sinkName, int expectedSize)
throws InterruptedException {
while (sinkSize(sinkName) < expectedSize) {
Thread.sleep(100);
}
} {code}
In abnormal case, this will lead to infinite waiting until CI time out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-36253) Wordcount on Docker test failed due to mismatch output hash
[
https://issues.apache.org/jira/browse/FLINK-36253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo closed FLINK-36253.
--
Resolution: Invalid
> Wordcount on Docker test failed due to mismatch output hash
> ---
>
> Key: FLINK-36253
> URL: https://issues.apache.org/jira/browse/FLINK-36253
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Affects Versions: 1.18.1
>Reporter: Weijie Guo
>Priority: Blocker
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=61837&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=43ba8ce7-ebbf-57cd-9163-444305d74117&l=5488
> {code:java}
> Sep 03 00:58:52 FAIL WordCount: Output hash mismatch. Got
> d41d8cd98f00b204e9800998ecf8427e, expected 0e5bd0a3dd7d5a7110aa85ff70adb54b.
> Sep 03 00:58:52 head hexdump of actual:
> head: cannot open
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-46667176796/out/docker_wc_out*'
> for reading: No such file or directory
> Sep 03 00:58:52 Stopping job timeout watchdog (with pid=174905)
> Sep 03 00:58:52 [FAIL] Test script contains errors.
> Sep 03 00:58:52 Checking for errors...
> Sep 03 00:58:52 No errors in log files.
> Sep 03 00:58:52 Checking for exceptions...
> Sep 03 00:58:52 No exceptions in log files.
> Sep 03 00:58:52 Checking for non-empty .out files...
> grep: /home/vsts/work/_temp/debug_files/flink-logs/*.out: No such file or
> directory
> Sep 03 00:58:52 No non-empty .out files.
> Sep 03 00:58:52
> Sep 03 00:58:52 [FAIL] 'Wordcount on Docker test (custom fs plugin)' failed
> after 1 minutes and 5 seconds! Test exited with exit code 1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-37073) Cross-team verification for "Adaptive optimization for batch jobs"
[
https://issues.apache.org/jira/browse/FLINK-37073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927892#comment-17927892
]
Yi Zhang edited comment on FLINK-37073 at 2/19/25 6:59 AM:
---
Verification results:
The results match the description.
Verification steps:
# Start the Flink cluster, construct the test case with data, and submit the
batch job from sql client.
# From the Flink UI, the job vertices of completed and running nodes can be
observed. There is a 'show pending operators' button in the top-left corner,
which, when selected, displays the stream nodes that have not yet run, as shown
in Attachment. The verification results match the description.
Verification case：
```
SET 'execution.runtime-mode' = 'batch';
CREATE TABLE dim1 (
a int NOT NULL,
b int NOT NULL
);
CREATE TABLE IF NOT EXISTS fact1 (
a int NOT NULL,
c int
);
CREATE TEMPORARY TABLE sink (
a int NOT NULL,
b int NOT NULL,
c int
)
WITH ('connector' = 'blackhole');
INSERT INTO sink
SELECT /*+ SHUFFLE_HASH(dim1) */ dim1.a AS a, b, c
FROM dim1, fact1_index
WHERE dim1.a = fact1_index.a AND dim1.b = 0;
```
was (Author: JIRAUSER307600):
Verification Steps:
# Start the Flink cluster, construct the test case with data, and submit the
batch job from sql client.
# From the Flink UI, the job vertices of completed and running nodes can be
observed. There is a 'show pending operators' button in the top-left corner,
which, when selected, displays the stream nodes that have not yet run, as shown
in Attachment. The verification results match the description.
Verification case：
```
SET 'execution.runtime-mode' = 'batch';
CREATE TABLE dim1 (
a int NOT NULL,
b int NOT NULL
);
CREATE TABLE IF NOT EXISTS fact1 (
a int NOT NULL,
c int
);
CREATE TEMPORARY TABLE sink (
a int NOT NULL,
b int NOT NULL,
c int
)
WITH ('connector' = 'blackhole');
INSERT INTO sink
SELECT /*+ SHUFFLE_HASH(dim1) */ dim1.a AS a, b, c
FROM dim1, fact1_index
WHERE dim1.a = fact1_index.a AND dim1.b = 0;
```
> Cross-team verification for "Adaptive optimization for batch jobs"
> --
>
> Key: FLINK-37073
> URL: https://issues.apache.org/jira/browse/FLINK-37073
> Project: Flink
> Issue Type: Sub-task
>Reporter: Xintong Song
>Assignee: Yi Zhang
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: adaptive-image.png
>
>
> In Flink 2.0, we introduced adaptive batch execution. In this mode, the job
> graph is incrementally generated, and the web UI has been adapted
> accordingly. We need to follow the steps below to verify:
> # Submit a multi-jobVertex batch job.
> # Check if the job graph displayed in the web UI unfolds as the job vertices
> are created, and if not all job vertices are created, there is a 'show
> pending operators' button in the top-left corner of the job overview page.
> Clicking this button should display the job graph along with the remaining
> stream nodes that have not yet been transfer to job vertices.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-36549][formats] Fix using the ignore-parse-errors parameter in Debezium/Canal/Maxwell/Ogg JSON results in unexpected data loss [flink]
xiaoyu19950911 commented on PR #26172: URL: https://github.com/apache/flink/pull/26172#issuecomment-2667674623 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37204] Add missing StarRocks connector options for 1.2.10. [flink-cdc]
lvyanquan commented on code in PR #3883:
URL: https://github.com/apache/flink-cdc/pull/3883#discussion_r1961067067
##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactory.java:
##
@@ -167,6 +172,7 @@ public Set> optionalOptions() {
Set> optionalOptions = new HashSet<>();
optionalOptions.add(StarRocksDataSinkOptions.SINK_LABEL_PREFIX);
optionalOptions.add(StarRocksDataSinkOptions.SINK_CONNECT_TIMEOUT);
+optionalOptions.add(StarRocksDataSinkOptions.SINK_SOCKET_TIMEOUT);
Review Comment:
It's better to verify this option in StarRocksDataSinkFactoryTest.
##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkOptions.java:
##
@@ -73,6 +73,14 @@ public class StarRocksDataSinkOptions {
.defaultValue(3)
.withDescription("Timeout in millisecond for connecting to
the `load-url`.");
+public static final ConfigOption SINK_SOCKET_TIMEOUT =
+ConfigOptions.key("sink.socket.timeout-ms")
Review Comment:
Please add this option to document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-37204] Add missing StarRocks connector options for 1.2.10. [flink-cdc]
lvyanquan commented on code in PR #3883:
URL: https://github.com/apache/flink-cdc/pull/3883#discussion_r1961069924
##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkOptions.java:
##
@@ -73,6 +73,14 @@ public class StarRocksDataSinkOptions {
.defaultValue(3)
.withDescription("Timeout in millisecond for connecting to
the `load-url`.");
+public static final ConfigOption SINK_SOCKET_TIMEOUT =
+ConfigOptions.key("sink.socket.timeout-ms")
+.intType()
+.defaultValue(-1)
+.withDescription(
+"Supported since 1.2.10. The time duration for
which the HTTP client waits for data."
Review Comment:
We should not expose the version of StarRocks sdk to user, instead, we can
expose the version of FlinkCDC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [BP-2.0][FLINK-37335][state/forst] Handle the first full checkpoint for ForSt IncrementalSnapshotStrategy. [flink]
AlexYinHan opened a new pull request, #26176: URL: https://github.com/apache/flink/pull/26176 ## What is the purpose of the change Fix the case when ForSt encounters the first full checkpoint with IncrementalCheckpoint is enabled. ## Brief change log - ```ForStIncrementalSnapshotStrategy``` skips throwing an exception if it encounters FORWARD sharingFilesStrategy but there are no previous snapshots to reuse. ## Verifying this change This change added tests and can be verified as follows: - ForStIncrementalSnapshotStrategyTest#testIncrementalCheckpointWithFirstBeingFull ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
lvyanquan commented on code in PR #3925: URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961112111 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java: ## @@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore) "col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()), "col2")); +addedColumns.add( +AddColumnEvent.before( +Column.physicalColumn( +"col4_first_before", + org.apache.flink.cdc.common.types.DataTypes.STRING()), +"col1")); Review Comment: col4_first? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-37350) Memory leak caused by skewed join optimization strategy
Lei Yang created FLINK-37350: Summary: Memory leak caused by skewed join optimization strategy Key: FLINK-37350 URL: https://issues.apache.org/jira/browse/FLINK-37350 Project: Flink Issue Type: Bug Components: Table SQL / Planner Reporter: Lei Yang Fix For: 2.0.0, 2.0.1 When the AdaptiveBroadcastJoinOptimizationStrategy is in effect, it leads to the statistics in the AdaptiveSkewedJoinOptimizationStrategy to not be released properly, which causes memory leaks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [BP-2.0][FLINK-37335][state/forst] Handle the first full checkpoint for ForSt IncrementalSnapshotStrategy. [flink]
flinkbot commented on PR #26176: URL: https://github.com/apache/flink/pull/26176#issuecomment-2667721635 ## CI report: * b0f3d9134edd14af7052d852c4717386146ebb5f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35360] support Flink cdc pipeline Yarn application mode [flink-cdc]
Mrart commented on code in PR #3643:
URL: https://github.com/apache/flink-cdc/pull/3643#discussion_r1961103490
##
flink-cdc-composer/pom.xml:
##
@@ -61,6 +61,12 @@ limitations under the License.
flink-kubernetes
${flink.version}
+
+org.apache.flink
+flink-yarn
+${flink.version}
+provided
Review Comment:
> [flink-yarn](https://issues.apache.org/jira/browse/FLINK-yarn) is only
required at compile time and is provided by the flink environment at runtime
Yes, flink-yarn doesn't need to be compiled if it's in flink lib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
MOBIN-F commented on code in PR #3925: URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961136374 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java: ## @@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore) "col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()), "col2")); +addedColumns.add( +AddColumnEvent.before( +Column.physicalColumn( +"col4_first_before", + org.apache.flink.cdc.common.types.DataTypes.STRING()), +"col1")); Review Comment: yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35599] Introduce JDBC pipeline sink connector [flink-cdc]
Mrart commented on PR #3916: URL: https://github.com/apache/flink-cdc/pull/3916#issuecomment-2667786041 Why not integrate with flink-cdc-connector？It has implemented pgsql/orcale/mysql. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-36203][Connectors/DynamoDB] Added polling delay between empty polls in getRecords calls [flink-connector-aws]
gguptp opened a new pull request, #190: URL: https://github.com/apache/flink-connector-aws/pull/190 ## Purpose of the change DDB Streams customers are charged on number of getrecords calls and given the connector maintains a queue of shards and if a task manager has only a few shards in the queue, it can cause a shard being called (1000/(x*y)) times in a second where x = number of shards in the queue, y = average getrecords latency. We should ideally be having some delay in case the last poll was an empty poll so that the customer gets charged less ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change added tests and can be verified as follows: - *Added unit tests* - *Manually verified by running the Kinesis connector on a local Flink cluster.* ## Significant changes *(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)* - [ ] Dependencies have been added or upgraded - [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`) - [ ] Serializers have been changed - [ x ] New feature has been introduced - If yes, how is this documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37320] [Observer] FINISHED finite streaming jobs incorrectly being set to RECONCILING [flink-kubernetes-operator]
gyfora commented on code in PR #944:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/944#discussion_r1960993002
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java:
##
@@ -152,7 +152,10 @@ protected void
observeJmDeployment(FlinkResourceContext ctx) {
}
deploymentStatus.setJobManagerDeploymentStatus(JobManagerDeploymentStatus.MISSING);
-deploymentStatus.getJobStatus().setState(JobStatus.RECONCILING);
+
+if (!ReconciliationUtils.isJobInTerminalState(deploymentStatus)) {
+deploymentStatus.getJobStatus().setState(JobStatus.RECONCILING);
Review Comment:
Can you please add a unit test to guard this change against accidental
future regressions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-37073) Cross-team verification for "Adaptive optimization for batch jobs"
[ https://issues.apache.org/jira/browse/FLINK-37073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Lee closed FLINK-37073. -- Resolution: Done > Cross-team verification for "Adaptive optimization for batch jobs" > -- > > Key: FLINK-37073 > URL: https://issues.apache.org/jira/browse/FLINK-37073 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Yi Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: adaptive-image.png > > > In Flink 2.0, we introduced adaptive batch execution. In this mode, the job > graph is incrementally generated, and the web UI has been adapted > accordingly. We need to follow the steps below to verify: > # Submit a multi-jobVertex batch job. > # Check if the job graph displayed in the web UI unfolds as the job vertices > are created, and if not all job vertices are created, there is a 'show > pending operators' button in the top-left corner of the job overview page. > Clicking this button should display the job graph along with the remaining > stream nodes that have not yet been transfer to job vertices. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-37073) Cross-team verification for "Adaptive optimization for batch jobs"
[ https://issues.apache.org/jira/browse/FLINK-37073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928271#comment-17928271 ] Junrui Lee commented on FLINK-37073: Thanks [~yizh] for verification~ > Cross-team verification for "Adaptive optimization for batch jobs" > -- > > Key: FLINK-37073 > URL: https://issues.apache.org/jira/browse/FLINK-37073 > Project: Flink > Issue Type: Sub-task >Reporter: Xintong Song >Assignee: Yi Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: adaptive-image.png > > > In Flink 2.0, we introduced adaptive batch execution. In this mode, the job > graph is incrementally generated, and the web UI has been adapted > accordingly. We need to follow the steps below to verify: > # Submit a multi-jobVertex batch job. > # Check if the job graph displayed in the web UI unfolds as the job vertices > are created, and if not all job vertices are created, there is a 'show > pending operators' button in the top-left corner of the job overview page. > Clicking this button should display the job graph along with the remaining > stream nodes that have not yet been transfer to job vertices. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35810) OOM in AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
[
https://issues.apache.org/jira/browse/FLINK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-35810:
-
Fix Version/s: 2.0.0
> OOM in
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> -
>
> Key: FLINK-35810
> URL: https://issues.apache.org/jira/browse/FLINK-35810
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.0-preview
>Reporter: Rui Fan
>Assignee: Weijie Guo
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 2.0.0
>
>
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> fails
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159]
>
> I always saw an OOM before this test failure.
> {code:java}
> Jul 25 04:58:33 java.lang.OutOfMemoryError: Java heap space
> Jul 25 04:58:33 Dumping heap to java_pid23540.hprof ...
> Jul 25 04:58:34 Heap dump file created [36427163 bytes in 0.226 secs]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35810) OOM in AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
[
https://issues.apache.org/jira/browse/FLINK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-35810:
-
Affects Version/s: (was: 2.0.0)
(was: 2.1.0)
> OOM in
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> -
>
> Key: FLINK-35810
> URL: https://issues.apache.org/jira/browse/FLINK-35810
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.0-preview
>Reporter: Rui Fan
>Assignee: Weijie Guo
>Priority: Blocker
> Labels: pull-request-available, test-stability
>
> AsyncWaitOperatorTest.testProcessingTimeRepeatedCompleteUnorderedWithRetry
> fails
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60837&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=10159]
>
> I always saw an OOM before this test failure.
> {code:java}
> Jul 25 04:58:33 java.lang.OutOfMemoryError: Java heap space
> Jul 25 04:58:33 Dumping heap to java_pid23540.hprof ...
> Jul 25 04:58:34 Heap dump file created [36427163 bytes in 0.226 secs]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-37335) ForSt fails to generate the first checkpoint in IncrementalCP and NO_CLAIM mode.
[
https://issues.apache.org/jira/browse/FLINK-37335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song reassigned FLINK-37335:
Assignee: Han Yin
> ForSt fails to generate the first checkpoint in IncrementalCP and NO_CLAIM
> mode.
>
>
> Key: FLINK-37335
> URL: https://issues.apache.org/jira/browse/FLINK-37335
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 2.0.0
>Reporter: Han Yin
>Assignee: Han Yin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> The {{ForStIncrementalSnapshotStrategy}} throws an exception if it encounters
> a {{sharingFilesStrategy}} of {{FORWARD}} when generating asynchronous
> snapshots. This behavior is designed to fail early in unexpected situations
> where full checkpoints are required for the incremental snapshot strategy.
> However, this incorrectly handles the following scenario: When incremental
> checkpoints are enabled and the job is restored in {{NO_CLAIM}} mode, the
> first checkpoint should be a full checkpoint, meaning the
> {{sharingFilesStrategy}} would be {{{}FORWARD{}}}.
> To address this issue, we let {{ForStIncrementalSnapshotStrategy}} skip
> throwing an exception if it encounters {{FORWARD}} but there are no previous
> snapshots to reuse.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-36867) flink on k8s operator could not create TM
[
https://issues.apache.org/jira/browse/FLINK-36867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-36867.
Resolution: Invalid
> flink on k8s operator could not create TM
> -
>
> Key: FLINK-36867
> URL: https://issues.apache.org/jira/browse/FLINK-36867
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: 1.8.0
> Environment: Client Version: version.Info\{Major:"1", Minor:"24",
> GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a",
> GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2",
> Compiler:"gc", Platform:"linux/amd64"}
> Kustomize Version: v4.5.4
> Server Version: version.Info\{Major:"1", Minor:"24", GitVersion:"v1.24.1",
> GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean",
> BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc",
> Platform:"linux/amd64"}
>
> flink 1.16.3
> operator 1.8.0
>Reporter: 周龙华
>Priority: Blocker
>
> 创建JobManager 成功，但是在提交ResourceManager 创建TaskManager时，一直不成功。报错如下
> 2024-12-08 13:55:20,302 INFO
> org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled
> external resources: []
> 2024-12-08 13:55:20,302 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key
> 'kubernetes.service-account' instead of key
> 'kubernetes.taskmanager.service-account'
> 2024-12-08 13:55:20,303 INFO
> org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Creating
> new TaskManager pod with name
> flink-ess-recall-mqtt-to-kafka-job-taskmanager-1-154 and resource <2048,0.5>.
> 2024-12-08 13:55:30,305 WARN
> org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Could not
> create pod flink-ess-recall-mqtt-to-kafka-job-taskmanager-1-154, exception:
> java.util.concurrent.CompletionException:
> io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
> 2024-12-08 13:55:30,306 WARN
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Failed requesting worker with resource spec WorkerResourceSpec
> \{cpuCores=0.5, taskHeapSize=1013.760mb (1063004400 bytes), taskOffHeapSize=0
> bytes, networkMemSize=158.720mb (166429984 bytes), managedMemSize=158.720mb
> (166429984 bytes), numSlots=2}, current pending count: 0
> java.util.concurrent.CompletionException:
> io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
> at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
> ~[?:?]
> at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
> ~[?:?]
> at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
> at java.lang.Thread.run(Unknown Source) ~[?:?]
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error
> has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:63)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$createTaskManagerPod$1(Fabric8FlinkKubeClient.java:163)
> ~[flink-dist-1.16.1.jar:1.16.1]
> ... 4 more
> Caused by: java.net.SocketTimeoutException: timeout
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.java:678)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.java:686)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.java:154)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.java:136)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterce
[jira] [Updated] (FLINK-35900) PyFlink: update version of Apache Beam dependency
[ https://issues.apache.org/jira/browse/FLINK-35900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-35900: - Issue Type: Improvement (was: Bug) > PyFlink: update version of Apache Beam dependency > - > > Key: FLINK-35900 > URL: https://issues.apache.org/jira/browse/FLINK-35900 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.19.1 >Reporter: Lydia L >Priority: Major > > PyFlink relies on Apache Beam numPy and it is up to date with numPy and other various python libraries. So it > would be helpful to update the Apache Beam version used in PyFlink. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35900) PyFlink: update version of Apache Beam dependency
[ https://issues.apache.org/jira/browse/FLINK-35900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-35900: - Priority: Major (was: Blocker) > PyFlink: update version of Apache Beam dependency > - > > Key: FLINK-35900 > URL: https://issues.apache.org/jira/browse/FLINK-35900 > Project: Flink > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Lydia L >Priority: Major > > PyFlink relies on Apache Beam numPy and it is up to date with numPy and other various python libraries. So it > would be helpful to update the Apache Beam version used in PyFlink. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-37254) Flink CDC OceanBase Connector Throws Exception for Column Default Value ‘DEFAULT ‘0’
[
https://issues.apache.org/jira/browse/FLINK-37254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928280#comment-17928280
]
caixiaowei commented on FLINK-37254:
Hello, may I ask when this bug will be fixed?
> Flink CDC OceanBase Connector Throws Exception for Column Default Value
> ‘DEFAULT ‘0’
>
>
> Key: FLINK-37254
> URL: https://issues.apache.org/jira/browse/FLINK-37254
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: cdc-3.1.0
> Environment: flink 1.18.1
> flink cdc 3.1.0
> oceanbase 5.7.25-OceanBase_CE-v4.1.0.2
>
>Reporter: caixiaowei
>Assignee: He Wang
>Priority: Blocker
> Labels: pull-request-available
> Attachments: 1.png
>
>
> When the table schema contains the following field:
> {color:#00}`tenant_id`{color}
> {color:#80}bigint{color}({color:#ff}20{color})
> {color:#80}unsigned{color} {color:#80}NOT{color}
> {color:#80}NULL{color} {color:#80}DEFAULT{color}
> {color:#008000}'0'{color} {color:#00}COMMENT{color}
> {color:#008000}'Tenant ID'{color},
> The following exception is thrown：
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy
> at
> org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:180)
> at
> org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:107)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:277)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:268)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:261)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:787)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:764)
> at
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:488)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309)
> at
> org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> at
> org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307)
> at
> org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222)
> at
> org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85)
> at
> org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168)
> at
> org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
> at
> org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
> at
> org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
> at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
> at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
> at
> org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
> at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
> at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
> at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280)
> at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241)
> at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
> at
> java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
> at java.util.concurrent.ForkJoinPool.runWorker(Fork
[jira] [Created] (FLINK-37348) [pipeline-paimon]fix paimon ArrayIndexOutOfBoundsException when add column before first_column
MOBIN created FLINK-37348:
-
Summary: [pipeline-paimon]fix paimon
ArrayIndexOutOfBoundsException when add column before first_column
Key: FLINK-37348
URL: https://issues.apache.org/jira/browse/FLINK-37348
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.3.0
Reporter: MOBIN
Repeat step：
# ALTER TABLE table_name add COLUMN new_first_column bigint first
# throw ArrayIndexOutOfBoundsException
log:
{code:java}
2025-02-19 11:32:16,586 INFO
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
All sink subtask have flushed for table dw_app.cdc_sink19. Start to apply
schema change request:
SchemaChangeRequest{tableId=dw_app.cdc_sink19,
schemaChangeEvent=AddColumnEvent{tableId=dw_app.cdc_sink19,
addedColumns=[ColumnWithPosition{column=`new_first_column` BIGINT,
position=BEFORE, existedColumnName=first_column}]}, subTaskId=0}
that extracts to:
AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column_EVOLVE4,
addedColumns=[ColumnWithPosition{column=`new_first_column` BIGINT,
position=BEFORE, existedColumnName=first_column}]}
2025-02-19 11:32:16,716 ERROR
org.apache.flink.cdc.runtime.operators.schema.common.SchemaRegistry [] - An
exception was triggered from Schema change applying task. Job will fail now.
org.apache.flink.util.FlinkRuntimeException: Failed to apply schema change
event.
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.lambda$startSchemaChangesEvolve$0(SchemaCoordinator.java:290)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:418) ~[?:1.8.0_65]
at java.util.ArrayList.get(ArrayList.java:431) ~[?:1.8.0_65]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumnWithBeforePosition(PaimonMetadataApplier.java:275)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumnEventWithPosition(PaimonMetadataApplier.java:237)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumn(PaimonMetadataApplier.java:206)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.lambda$applySchemaChange$0(PaimonMetadataApplier.java:127)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.common.event.visitor.SchemaChangeEventVisitor.visit(SchemaChangeEventVisitor.java:47)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applySchemaChange(PaimonMetadataApplier.java:124)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.applyAndUpdateEvolvedSchemaChange(SchemaCoordinator.java:434)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.applySchemaChange(SchemaCoordinator.java:401)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.lambda$startSchemaChangesEvolve$0(SchemaCoordinator.java:288)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
... 5 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
MOBIN-F opened a new pull request, #3925:
URL: https://github.com/apache/flink-cdc/pull/3925
Repeat step：
1. ALTER TABLE table_name add COLUMN new_first_column bigint first
2. throw ArrayIndexOutOfBoundsException(Because the [add column first] event
will be converted into an [add column before] event,
ArrayIndexOutOfBoundsException is thrown when getting the existingColumnIndex)
log:
```
2025-02-19 11:32:16,586 INFO
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
All sink subtask have flushed for table dw_app.cdc_sink19. Start to apply
schema change request:
SchemaChangeRequest{tableId=dw_app.cdc_sink19,
schemaChangeEvent=AddColumnEvent{tableId=dw_app.cdc_sink19,
addedColumns=[ColumnWithPosition{column=`new_first_column` BIGINT,
position=BEFORE, existedColumnName=first_column}]}, subTaskId=0}
that extracts to:
AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column_EVOLVE4,
addedColumns=[ColumnWithPosition{column=`new_first_column` BIGINT,
position=BEFORE, existedColumnName=first_column}]}
2025-02-19 11:32:16,716 ERROR
org.apache.flink.cdc.runtime.operators.schema.common.SchemaRegistry [] - An
exception was triggered from Schema change applying task. Job will fail now.
org.apache.flink.util.FlinkRuntimeException: Failed to apply schema change
event.
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.lambda$startSchemaChangesEvolve$0(SchemaCoordinator.java:290)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:418) ~[?:1.8.0_65]
at java.util.ArrayList.get(ArrayList.java:431) ~[?:1.8.0_65]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumnWithBeforePosition(PaimonMetadataApplier.java:275)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumnEventWithPosition(PaimonMetadataApplier.java:237)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumn(PaimonMetadataApplier.java:206)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.lambda$applySchemaChange$0(PaimonMetadataApplier.java:127)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.common.event.visitor.SchemaChangeEventVisitor.visit(SchemaChangeEventVisitor.java:47)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applySchemaChange(PaimonMetadataApplier.java:124)
~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.applyAndUpdateEvolvedSchemaChange(SchemaCoordinator.java:434)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.applySchemaChange(SchemaCoordinator.java:401)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
at
org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.lambda$startSchemaChangesEvolve$0(SchemaCoordinator.java:288)
~[flink-cdc-dist-3.3.0.jar:3.3.0]
... 5 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-37348) [pipeline-paimon]fix paimon ArrayIndexOutOfBoundsException when add column before first_column
[
https://issues.apache.org/jira/browse/FLINK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-37348:
---
Labels: pull-request-available (was: )
> [pipeline-paimon]fix paimon ArrayIndexOutOfBoundsException when add column
> before first_column
> --
>
> Key: FLINK-37348
> URL: https://issues.apache.org/jira/browse/FLINK-37348
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: cdc-3.3.0
>Reporter: MOBIN
>Priority: Major
> Labels: pull-request-available
>
> Repeat step：
> # ALTER TABLE table_name add COLUMN new_first_column bigint first
> # throw ArrayIndexOutOfBoundsException
> log:
> {code:java}
> 2025-02-19 11:32:16,586 INFO
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] -
> All sink subtask have flushed for table dw_app.cdc_sink19. Start to apply
> schema change request:
> SchemaChangeRequest{tableId=dw_app.cdc_sink19,
> schemaChangeEvent=AddColumnEvent{tableId=dw_app.cdc_sink19,
> addedColumns=[ColumnWithPosition{column=`new_first_column` BIGINT,
> position=BEFORE, existedColumnName=first_column}]}, subTaskId=0}
> that extracts to:
> AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column_EVOLVE4,
> addedColumns=[ColumnWithPosition{column=`new_first_column` BIGINT,
> position=BEFORE, existedColumnName=first_column}]}
> 2025-02-19 11:32:16,716 ERROR
> org.apache.flink.cdc.runtime.operators.schema.common.SchemaRegistry [] - An
> exception was triggered from Schema change applying task. Job will fail now.
> org.apache.flink.util.FlinkRuntimeException: Failed to apply schema change
> event.
> at
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.lambda$startSchemaChangesEvolve$0(SchemaCoordinator.java:290)
> ~[flink-cdc-dist-3.3.0.jar:3.3.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[?:1.8.0_65]
> at java.util.ArrayList.get(ArrayList.java:431) ~[?:1.8.0_65]
> at
> org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumnWithBeforePosition(PaimonMetadataApplier.java:275)
> ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
> at
> org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumnEventWithPosition(PaimonMetadataApplier.java:237)
> ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
> at
> org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumn(PaimonMetadataApplier.java:206)
> ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
> at
> org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.lambda$applySchemaChange$0(PaimonMetadataApplier.java:127)
> ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
> at
> org.apache.flink.cdc.common.event.visitor.SchemaChangeEventVisitor.visit(SchemaChangeEventVisitor.java:47)
> ~[flink-cdc-dist-3.3.0.jar:3.3.0]
> at
> org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applySchemaChange(PaimonMetadataApplier.java:124)
> ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
> at
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.applyAndUpdateEvolvedSchemaChange(SchemaCoordinator.java:434)
> ~[flink-cdc-dist-3.3.0.jar:3.3.0]
> at
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.applySchemaChange(SchemaCoordinator.java:401)
> ~[flink-cdc-dist-3.3.0.jar:3.3.0]
> at
> org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.lambda$startSchemaChangesEvolve$0(SchemaCoordinator.java:288)
> ~[flink-cdc-dist-3.3.0.jar:3.3.0]
> ... 5 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-37313) [BUG] MySQL snapshot read phase is unresponsive for a long time (even up to several hours)
[
https://issues.apache.org/jira/browse/FLINK-37313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928241#comment-17928241
]
Yanquan Lv commented on FLINK-37313:
Hi, [~zexin]. could you check that whether
[https://github.com/apache/flink-cdc/pull/3902] has solved you problem?
> [BUG] MySQL snapshot read phase is unresponsive for a long time (even up to
> several hours)
> --
>
> Key: FLINK-37313
> URL: https://issues.apache.org/jira/browse/FLINK-37313
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: 3.0.0
> Environment: mysql: 8.0.33
> flink-connector-mysql-cdc: 3.3.0
>Reporter: zexin gong
>Assignee: zexin gong
>Priority: Major
> Labels: pull-request-available
> Attachments: 读取binlog.jpg, 读取结果集.jpg
>
>
> The split reading time during the MySQL snapshot phase is abnormal. The
> specific situation is as follows:
> 1. The {*}snapshot reading is very fast{*}, about a few seconds:
> !读取结果集.jpg!
> 2. The time to *read the high and low watermarks of the snapshot is abnormal*
> (even up to several hours):
> !读取binlog.jpg!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-36998) The RowKind of data about globalUpsertResult is different in KeyedUpsertingSinkFunction while being inserted or being restored
[ https://issues.apache.org/jira/browse/FLINK-36998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928230#comment-17928230 ] lincoln lee commented on FLINK-36998: - Fixed in 2.0: bbfbe33f6bc265af3264774bc4600b850af17661 > The RowKind of data about globalUpsertResult is different in > KeyedUpsertingSinkFunction while being inserted or being restored > -- > > Key: FLINK-36998 > URL: https://issues.apache.org/jira/browse/FLINK-36998 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: xuyang >Assignee: xuyang >Priority: Minor > Labels: pull-request-available > > Take a row data with RowKind +U as an example: > When inserting this data into sink by #invoke, the row kind is set to +I. > However when this data is restored from globalRawResult, the row kind is +U. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-36998][table-planner] Keep same behavior about the RowKind of data in globalUpsertResult for KeyedUpsertingSinkFunction while being inserted or being restore [flink]
lincoln-lil merged PR #26169: URL: https://github.com/apache/flink/pull/26169 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37201] [cdc] [transform] FlinkCDC Pipeline transform arithmetic functions support parameters of null and more numerical types [flink-cdc]
leonardBang merged PR #3881: URL: https://github.com/apache/flink-cdc/pull/3881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-37201) FlinkCDC Pipeline transform arithmetic functions support parameters of null and more numerical types.
[ https://issues.apache.org/jira/browse/FLINK-37201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-37201: --- Fix Version/s: cdc-3.4.0 > FlinkCDC Pipeline transform arithmetic functions support parameters of null > and more numerical types. > - > > Key: FLINK-37201 > URL: https://issues.apache.org/jira/browse/FLINK-37201 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.3.0 >Reporter: Wenkai Qi >Assignee: Wenkai Qi >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.4.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > FlinkCDC Pipeline transform arithmetic functions can not support parameters > of null and more numerical types. For example: ceil, floor, round and abs. > When the input parameter is null or a numerical type such as int, the program > will have an error. > I have fixed it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-37201) FlinkCDC Pipeline transform arithmetic functions support parameters of null and more numerical types.
[ https://issues.apache.org/jira/browse/FLINK-37201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu resolved FLINK-37201. Resolution: Implemented Implemented via master: 83ae36a854ffe37b5ea16a9cb0494b8651123f02 > FlinkCDC Pipeline transform arithmetic functions support parameters of null > and more numerical types. > - > > Key: FLINK-37201 > URL: https://issues.apache.org/jira/browse/FLINK-37201 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.3.0 >Reporter: Wenkai Qi >Assignee: Wenkai Qi >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.4.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > FlinkCDC Pipeline transform arithmetic functions can not support parameters > of null and more numerical types. For example: ceil, floor, round and abs. > When the input parameter is null or a numerical type such as int, the program > will have an error. > I have fixed it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-37347) [docs ]Support any columns as chunk key column (postgres, orcale, db2, sqlserver)
Lee SeungMin created FLINK-37347: Summary: [docs ]Support any columns as chunk key column (postgres, orcale, db2, sqlserver) Key: FLINK-37347 URL: https://issues.apache.org/jira/browse/FLINK-37347 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Lee SeungMin update document for https://issues.apache.org/jira/browse/FLINK-37332?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Flink%20CDC%22 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [hotfix][runtime] Remove the unused SchedulerBase#transitionToScheduled method [flink]
RocMarshal opened a new pull request, #26175: URL: https://github.com/apache/flink/pull/26175 ## What is the purpose of the change [hotfix][runtime] Remove the unused SchedulerBase#transitionToScheduled method ## Brief change log [hotfix][runtime] Remove the unused SchedulerBase#transitionToScheduled method ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35810][test] increase unittest Xmx to debug FLINK-35810 [flink]
flinkbot commented on PR #26174: URL: https://github.com/apache/flink/pull/26174#issuecomment-2667454792 ## CI report: * daaed1ed67dd8afea067c3dca689dca13a3ef611 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-37347) [docs ]Support any columns as chunk key column (postgres, orcale, db2, sqlserver)
[ https://issues.apache.org/jira/browse/FLINK-37347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-37347: --- Labels: pull-request-available (was: ) > [docs ]Support any columns as chunk key column (postgres, orcale, db2, > sqlserver) > - > > Key: FLINK-37347 > URL: https://issues.apache.org/jira/browse/FLINK-37347 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Lee SeungMin >Priority: Major > Labels: pull-request-available > > update document for > https://issues.apache.org/jira/browse/FLINK-37332?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Flink%20CDC%22 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-37347][DOCS] Support any column as chunk key column (postgres, orcale, db2, sqlserver) [flink-cdc]
SML0127 opened a new pull request, #3924: URL: https://github.com/apache/flink-cdc/pull/3924 updates documentation for https://github.com/apache/flink-cdc/pull/3922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37332] Support any column as chunk key column (postgres, orcale, db2, sqlserver) [flink-cdc]
SML0127 commented on PR #3922: URL: https://github.com/apache/flink-cdc/pull/3922#issuecomment-2667460016 > I suggest separating the pull requests for code and documentation to avoid the term 'snapshot' appearing in the documents. The pull request for the documentation can be merged after the release of version 3.4. Hi, @gtk96 I seperated pr for docs (https://github.com/apache/flink-cdc/pull/3924) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix][runtime] Remove the unused SchedulerBase#transitionToScheduled method [flink]
flinkbot commented on PR #26175: URL: https://github.com/apache/flink/pull/26175#issuecomment-2667467529 ## CI report: * f1ea6361313af6844927811cb7f18be3cc1e9e0f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
MOBIN-F commented on code in PR #3925: URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961119229 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java: ## @@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore) "col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()), "col2")); +addedColumns.add( +AddColumnEvent.before( +Column.physicalColumn( +"col4_first_before", + org.apache.flink.cdc.common.types.DataTypes.STRING()), +"col1")); Review Comment: col4_first already exists, so I added the _before suffix to distinguish it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
lvyanquan commented on code in PR #3925: URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961126616 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java: ## @@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore) "col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()), "col2")); +addedColumns.add( +AddColumnEvent.before( +Column.physicalColumn( +"col4_first_before", + org.apache.flink.cdc.common.types.DataTypes.STRING()), +"col1")); Review Comment: The first column here is already col4_first, right? If col1 is used, then col4_first_fefore should be the second column. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.19][FLINK-36740] [WebFrontend] Update frontend dependencies to address vulnerabilities [flink]
mehdid93 closed pull request #25914: [BP-1.19][FLINK-36740] [WebFrontend] Update frontend dependencies to address vulnerabilities URL: https://github.com/apache/flink/pull/25914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Release flink-shaded 20.0 [flink-web]
snuyanzin opened a new pull request, #774: URL: https://github.com/apache/flink-web/pull/774 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-37320) FINISHED finite streaming jobs incorrectly being set to RECONCILING
[ https://issues.apache.org/jira/browse/FLINK-37320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Castelli updated FLINK-37320: -- Description: Hello, I believe I've found a bug within the observation logic for finite streaming jobs. This is a follow-up to: [https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k]. *For finite streaming jobs:* # The job finishes successfully and the job status changes to FINISHED # TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the JM deployments and clears HA configmap data # On the next loop, the observer sees MISSING JM and changes the job status from FINISHED to RECONCILING The job had reached a terminal state. It shouldn't have been set back to RECONCILING. This leads to an operator error later when a recovery attempt is triggered. The recovery is triggered because the JM is MISSING, the status is RECONCILING, spec shows RUNNING, and HA enabled. The recovery fails with validateHaMetadataExists throwing UpgradeFailureException. At that point the deployment gets stuck in a loop with status RECONCILING and UpgradeFailureException thrown on each cycle. I've attached operator logs showing this. *Proposed solution:* I think the fix would be to wrap [AbstractFlinkDeploymentObserver.observeJmDeployment|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155] in an if-statement that checks the job is not in a terminal state. Happy to discuss and/or put up the 2 line code change PR. was: Hello, I believe I've found bugs within the observation logic for both finite streaming and batch jobs. This is a follow-up to: [https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k]. *For finite streaming jobs:* # The job finishes successfully and the job status changes to FINISHED # TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the JM deployments and clears HA configmap data # On the next loop, the observer sees MISSING JM and changes the job status from FINISHED to RECONCILING The job had reached a terminal state. It shouldn't have been set back to RECONCILING. This leads to an operator error later when a recovery attempt is triggered. The recovery is triggered because the JM is MISSING, the status is RECONCILING, spec shows RUNNING, and HA enabled. The recovery fails with validateHaMetadataExists throwing UpgradeFailureException. At that point the deployment gets stuck in a loop with status RECONCILING and UpgradeFailureException thrown on each cycle. I've attached operator logs showing this. *Proposed solution:* I think the fix would be to wrap [AbstractFlinkDeploymentObserver.observeJmDeployment|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155] in an if-statement that checks the job is not in a terminal state. Happy to discuss and/or put up the 2 line code change PR. > FINISHED finite streaming jobs incorrectly being set to RECONCILING > --- > > Key: FLINK-37320 > URL: https://issues.apache.org/jira/browse/FLINK-37320 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.10.0 > Environment: I've attached the flinkdeployment CR and operator-config > I used to locally replicate. >Reporter: Luca Castelli >Priority: Minor > Attachments: operator-config.yaml, > operator-log-finite-streaming-job.log, test-finite-streaming-job.yaml > > > Hello, > I believe I've found a bug within the observation logic for finite streaming > jobs. This is a follow-up to: > [https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k]. > *For finite streaming jobs:* > # The job finishes successfully and the job status changes to FINISHED > # TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the > JM deployments and clears HA configmap data > # On the next loop, the observer sees MISSING JM and changes the job status > from FINISHED to RECONCILING > The job had reached a terminal state. It shouldn't have been set back to > RECONCILING. > This leads to an operator error later when a recovery attempt is triggered. > The recovery is triggered because the JM is MISSING, the status is > RECONCILING, spec shows RUNNING, and HA enabled. The recovery fails with > validateHaMetadataExists throwing UpgradeFailureException. > At that point the deployment gets stuck in a loop with status RECONCILING and > UpgradeFailureException thrown on each cycle. I've attached operator logs > showing th
[jira] [Updated] (FLINK-37320) FINISHED finite streaming jobs incorrectly being set to RECONCILING
[ https://issues.apache.org/jira/browse/FLINK-37320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Castelli updated FLINK-37320: -- Summary: FINISHED finite streaming jobs incorrectly being set to RECONCILING (was: FINISHED jobs incorrectly being set to RECONCILING) > FINISHED finite streaming jobs incorrectly being set to RECONCILING > --- > > Key: FLINK-37320 > URL: https://issues.apache.org/jira/browse/FLINK-37320 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.10.0 > Environment: I've attached the flinkdeployment CR and operator-config > I used to locally replicate. >Reporter: Luca Castelli >Priority: Minor > Attachments: operator-config.yaml, > operator-log-finite-streaming-job.log, test-finite-streaming-job.yaml > > > Hello, > I believe I've found bugs within the observation logic for both finite > streaming and batch jobs. This is a follow-up to: > [https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k]. > *For finite streaming jobs:* > # The job finishes successfully and the job status changes to FINISHED > # TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the > JM deployments and clears HA configmap data > # On the next loop, the observer sees MISSING JM and changes the job status > from FINISHED to RECONCILING > The job had reached a terminal state. It shouldn't have been set back to > RECONCILING. > This leads to an operator error later when a recovery attempt is triggered. > The recovery is triggered because the JM is MISSING, the status is > RECONCILING, spec shows RUNNING, and HA enabled. The recovery fails with > validateHaMetadataExists throwing UpgradeFailureException. > At that point the deployment gets stuck in a loop with status RECONCILING and > UpgradeFailureException thrown on each cycle. I've attached operator logs > showing this. > *Proposed solution:* I think the fix would be to wrap > [AbstractFlinkDeploymentObserver.observeJmDeployment|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155] > in an if-statement that checks the job is not in a terminal state. Happy to > discuss and/or put up the 2 line code change PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-37320) FINISHED jobs incorrectly being set to RECONCILING
[
https://issues.apache.org/jira/browse/FLINK-37320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luca Castelli updated FLINK-37320:
--
Description:
Hello,
I believe I've found bugs within the observation logic for both finite
streaming and batch jobs. This is a follow-up to:
[https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k].
*For finite streaming jobs:*
# The job finishes successfully and the job status changes to FINISHED
# TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the JM
deployments and clears HA configmap data
# On the next loop, the observer sees MISSING JM and changes the job status
from FINISHED to RECONCILING
The job had reached a terminal state. It shouldn't have been set back to
RECONCILING.
This leads to an operator error later when a recovery attempt is triggered. The
recovery is triggered because the JM is MISSING, the status is RECONCILING,
spec shows RUNNING, and HA enabled. The recovery fails with
validateHaMetadataExists throwing UpgradeFailureException.
At that point the deployment gets stuck in a loop with status RECONCILING and
UpgradeFailureException thrown on each cycle. I've attached operator logs
showing this.
*Proposed solution:* I think the fix would be to wrap
[AbstractFlinkDeploymentObserver.observeJmDeployment|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155]
in an if-statement that checks the job is not in a terminal state. Happy to
discuss and/or put up the 2 line code change PR.
was:
Hello,
I believe I've found bugs within the observation logic for both finite
streaming and batch jobs. This is a follow-up to:
[https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k].
*For finite streaming jobs:*
# The job finishes successfully and the job status changes to FINISHED
# TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the JM
deployments and clears HA configmap data
# On the next loop, the observer sees MISSING JM and changes the job status
from FINISHED to RECONCILING
The job had reached a terminal state. It shouldn't have been set back to
RECONCILING.
This leads to an operator error later when a recovery attempt is triggered. The
recovery is triggered because the JM is MISSING, the status is RECONCILING,
spec shows RUNNING, and HA enabled. The recovery fails with
validateHaMetadataExists throwing UpgradeFailureException.
At that point the deployment gets stuck in a loop with status RECONCILING and
UpgradeFailureException thrown on each cycle. I've attached operator logs
showing this.
*Proposed solution:* I think the fix would be to wrap
[AbstractFlinkDeploymentObserver.observeJmDeployment|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155]
in an if-statement that checks the job is not in a terminal state. Happy to
discuss and/or put up the 2 line code change PR.
*For batch jobs:*
The root error (full logs attached):
{code:java}
Caused by: org.apache.flink.runtime.rest.util.RestClientException:
[org.apache.flink.runtime.rest.handler.RestHandlerException: Checkpointing has
not been enabled.
at
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:107){code}
# The job finishes successfully and the job status changes to FINISHED in
memory
# When observing the FlinkCluster, in
[AbstractFlinkService.getCheckpointInfo|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L570],
the operator successfully connects to the Flink REST API
# It tries to get checkpoint statistics via CheckpointingStatisticsHandler,
but the handler throws an error because checkpointing is not enabled. This is
actually an expected response since it's a BATCH job
E.g.
[http://localhost:8081/jobs/job-id/checkpoints|http://localhost:8081/jobs/d12843a2a326d5268e550012e620cdca/checkpoints]
{code:java}
{ "errors": [ "org.apache.flink.runtime.rest.handler.RestHandlerException:
Checkpointing has not been enabled.\n\tat
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:107)\n\tat
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)\n\tat
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)\n\tat
or
[jira] [Updated] (FLINK-37320) FINISHED jobs incorrectly being set to RECONCILING
[ https://issues.apache.org/jira/browse/FLINK-37320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Castelli updated FLINK-37320: -- Attachment: (was: test-batch-job.yaml) > FINISHED jobs incorrectly being set to RECONCILING > -- > > Key: FLINK-37320 > URL: https://issues.apache.org/jira/browse/FLINK-37320 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.10.0 > Environment: I've attached the flinkdeployment CR and operator-config > I used to locally replicate. >Reporter: Luca Castelli >Priority: Minor > Attachments: operator-config.yaml, > operator-log-finite-streaming-job.log, test-finite-streaming-job.yaml > > > Hello, > I believe I've found bugs within the observation logic for both finite > streaming and batch jobs. This is a follow-up to: > [https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k]. > *For finite streaming jobs:* > # The job finishes successfully and the job status changes to FINISHED > # TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the > JM deployments and clears HA configmap data > # On the next loop, the observer sees MISSING JM and changes the job status > from FINISHED to RECONCILING > The job had reached a terminal state. It shouldn't have been set back to > RECONCILING. > This leads to an operator error later when a recovery attempt is triggered. > The recovery is triggered because the JM is MISSING, the status is > RECONCILING, spec shows RUNNING, and HA enabled. The recovery fails with > validateHaMetadataExists throwing UpgradeFailureException. > At that point the deployment gets stuck in a loop with status RECONCILING and > UpgradeFailureException thrown on each cycle. I've attached operator logs > showing this. > *Proposed solution:* I think the fix would be to wrap > [AbstractFlinkDeploymentObserver.observeJmDeployment|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155] > in an if-statement that checks the job is not in a terminal state. Happy to > discuss and/or put up the 2 line code change PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-37344) setup_hugo.sh should install the hugo binary only
[ https://issues.apache.org/jira/browse/FLINK-37344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels resolved FLINK-37344. Resolution: Fixed > setup_hugo.sh should install the hugo binary only > - > > Key: FLINK-37344 > URL: https://issues.apache.org/jira/browse/FLINK-37344 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-37344][doc] setup_hugo.sh now installs the binary only [flink]
mxm merged PR #26170: URL: https://github.com/apache/flink/pull/26170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-37320) FINISHED finite streaming jobs incorrectly being set to RECONCILING
[ https://issues.apache.org/jira/browse/FLINK-37320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-37320: --- Labels: pull-request-available (was: ) > FINISHED finite streaming jobs incorrectly being set to RECONCILING > --- > > Key: FLINK-37320 > URL: https://issues.apache.org/jira/browse/FLINK-37320 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.10.0 > Environment: I've attached the flinkdeployment CR and operator-config > I used to locally replicate. >Reporter: Luca Castelli >Priority: Minor > Labels: pull-request-available > Attachments: operator-config.yaml, > operator-log-finite-streaming-job.log, test-finite-streaming-job.yaml > > > Hello, > I believe I've found a bug within the observation logic for finite streaming > jobs. This is a follow-up to: > [https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k]. > *For finite streaming jobs:* > # The job finishes successfully and the job status changes to FINISHED > # TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the > JM deployments and clears HA configmap data > # On the next loop, the observer sees MISSING JM and changes the job status > from FINISHED to RECONCILING > The job had reached a terminal state. It shouldn't have been set back to > RECONCILING. > This leads to an operator error later when a recovery attempt is triggered. > The recovery is triggered because the JM is MISSING, the status is > RECONCILING, spec shows RUNNING, and HA enabled. The recovery fails with > validateHaMetadataExists throwing UpgradeFailureException. > At that point the deployment gets stuck in a loop with status RECONCILING and > UpgradeFailureException thrown on each cycle. I've attached operator logs > showing this. > *Proposed solution:* I think the fix would be to wrap > [AbstractFlinkDeploymentObserver.observeJmDeployment|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155] > in an if-statement that checks the job is not in a terminal state. Happy to > discuss and/or put up the 2 line code change PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-37320] [Observer] FINISHED finite streaming jobs incorrectly being set to RECONCILING [flink-kubernetes-operator]
luca-p-castelli opened a new pull request, #944: URL: https://github.com/apache/flink-kubernetes-operator/pull/944 ## What is the purpose of the change This pull request fixes a bug where the observer tries changing the JOB STATUS of a FINISHED finite/bounded streaming job back to RECONCILING if the JM is MISSING. ## Brief change log Wraps [AbstractFlinkDeploymentObserver.observeJmDeployment](https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155) in an if-statement that only executes if the job is NOT in a terminal state. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): No - The public API, i.e., is any changes to the `CustomResourceDescriptors`: No - Core observer or reconciler logic that is regularly executed: Yes ## Documentation - Does this pull request introduce a new feature? No - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-37320) [Observer] FINISHED finite streaming jobs incorrectly being set to RECONCILING
[ https://issues.apache.org/jira/browse/FLINK-37320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Castelli updated FLINK-37320: -- Summary: [Observer] FINISHED finite streaming jobs incorrectly being set to RECONCILING (was: FINISHED finite streaming jobs incorrectly being set to RECONCILING) > [Observer] FINISHED finite streaming jobs incorrectly being set to RECONCILING > -- > > Key: FLINK-37320 > URL: https://issues.apache.org/jira/browse/FLINK-37320 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.10.0 > Environment: I've attached the flinkdeployment CR and operator-config > I used to locally replicate. >Reporter: Luca Castelli >Priority: Minor > Labels: pull-request-available > Attachments: operator-config.yaml, > operator-log-finite-streaming-job.log, test-finite-streaming-job.yaml > > > Hello, > I believe I've found a bug within the observation logic for finite streaming > jobs. This is a follow-up to: > [https://lists.apache.org/thread/xvsk4fmlqln092cdolvox4dgko0pw81k]. > *For finite streaming jobs:* > # The job finishes successfully and the job status changes to FINISHED > # TTL (kubernetes.operator.jm-deployment.shutdown-ttl) cleanup removes the > JM deployments and clears HA configmap data > # On the next loop, the observer sees MISSING JM and changes the job status > from FINISHED to RECONCILING > The job had reached a terminal state. It shouldn't have been set back to > RECONCILING. > This leads to an operator error later when a recovery attempt is triggered. > The recovery is triggered because the JM is MISSING, the status is > RECONCILING, spec shows RUNNING, and HA enabled. The recovery fails with > validateHaMetadataExists throwing UpgradeFailureException. > At that point the deployment gets stuck in a loop with status RECONCILING and > UpgradeFailureException thrown on each cycle. I've attached operator logs > showing this. > *Proposed solution:* I think the fix would be to wrap > [AbstractFlinkDeploymentObserver.observeJmDeployment|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L155] > in an if-statement that checks the job is not in a terminal state. Happy to > discuss and/or put up the 2 line code change PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-37344) setup_hugo.sh should install the hugo binary only
[
https://issues.apache.org/jira/browse/FLINK-37344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928169#comment-17928169
]
Gabor Somogyi commented on FLINK-37344:
---
[{{9a1e986}}|https://github.com/apache/flink/commit/9a1e9868bddb26898450c9b2410d036ad516ba0b]
on master
> setup_hugo.sh should install the hugo binary only
> -
>
> Key: FLINK-37344
> URL: https://issues.apache.org/jira/browse/FLINK-37344
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Gabor Somogyi
>Assignee: Gabor Somogyi
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [BP-1.19][FLINK-36689][Runtime/Web Frontend] Update ng-zorro-antd to v18 [flink]
mehdid93 closed pull request #25913: [BP-1.19][FLINK-36689][Runtime/Web Frontend] Update ng-zorro-antd to v18 URL: https://github.com/apache/flink/pull/25913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.19][FLINK-34194] Update CI to Ubuntu 22.04 (Jammy) [flink]
mehdid93 closed pull request #25911: [BP-1.19][FLINK-34194] Update CI to Ubuntu 22.04 (Jammy) URL: https://github.com/apache/flink/pull/25911 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.19][FLINK-36739] Update the NodeJS to v22.11.0 (LTS) [flink]
mehdid93 closed pull request #25912: [BP-1.19][FLINK-36739] Update the NodeJS to v22.11.0 (LTS) URL: https://github.com/apache/flink/pull/25912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23886) An exception is thrown out when recover job timers from checkpoint file
[ https://issues.apache.org/jira/browse/FLINK-23886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928137#comment-17928137 ] Tommy Schnabel commented on FLINK-23886: Value overflow makes sense! We were not able to get that specific checkpoint to restore. We had to fork flink and make a change so that any issue parsing timers were dropped. We added logs when this happened and it was <100 timers that were corrupted. I attached the patch we made to an above comment > An exception is thrown out when recover job timers from checkpoint file > --- > > Key: FLINK-23886 > URL: https://issues.apache.org/jira/browse/FLINK-23886 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.10.0, 1.11.3, 1.13.2, 1.13.5, 1.14.4 >Reporter: Jing Zhang >Assignee: Yuan Mei >Priority: Major > Attachments: image-2021-08-25-16-38-04-023.png, > image-2021-08-25-16-38-12-308.png, image-2021-08-25-17-06-29-806.png, > image-2021-08-25-17-07-38-327.png, image-2024-07-08-16-55-53-243.png, > segment-drop-corrupted-timer-state.diff > > > A user report the bug in the [mailist. > |http://mail-archives.apache.org/mod_mbox/flink-user/202108.mbox/%3ccakmsf43j14nkjmgjuy4dh5qn2vbjtw4tfh4pmmuyvcvfhgf...@mail.gmail.com%3E]I > paste the content here. > Setup Specifics: > Version: 1.6.2 > RocksDB Map State > Timers stored in rocksdb > > When we have this job running for long periods of time like > 30 days, if > for some reason the job restarts, we encounter "Error while deserializing the > element". Is this a known issue fixed in later versions? I see some changes > to code for FLINK-10175, but we don't use any queryable state > > Below is the stack trace > > org.apache.flink.util.FlinkRuntimeException: Error while deserializing the > element. > at > org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.deserializeElement(RocksDBCachingPriorityQueueSet.java:389) > at > org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.peek(RocksDBCachingPriorityQueueSet.java:146) > at > org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.peek(RocksDBCachingPriorityQueueSet.java:56) > at > org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue$InternalPriorityQueueComparator.comparePriority(KeyGroupPartitionedPriorityQueue.java:274) > at > org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue$InternalPriorityQueueComparator.comparePriority(KeyGroupPartitionedPriorityQueue.java:261) > at > org.apache.flink.runtime.state.heap.HeapPriorityQueue.isElementPriorityLessThen(HeapPriorityQueue.java:164) > at > org.apache.flink.runtime.state.heap.HeapPriorityQueue.siftUp(HeapPriorityQueue.java:121) > at > org.apache.flink.runtime.state.heap.HeapPriorityQueue.addInternal(HeapPriorityQueue.java:85) > at > org.apache.flink.runtime.state.heap.AbstractHeapPriorityQueue.add(AbstractHeapPriorityQueue.java:73) > at > org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue.(KeyGroupPartitionedPriorityQueue.java:89) > at > org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBPriorityQueueSetFactory.create(RocksDBKeyedStateBackend.java:2792) > at > org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.create(RocksDBKeyedStateBackend.java:450) > at > org.apache.flink.streaming.api.operators.InternalTimeServiceManager.createTimerPriorityQueue(InternalTimeServiceManager.java:121) > at > org.apache.flink.streaming.api.operators.InternalTimeServiceManager.registerOrGetTimerService(InternalTimeServiceManager.java:106) > at > org.apache.flink.streaming.api.operators.InternalTimeServiceManager.getInternalTimerService(InternalTimeServiceManager.java:87) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.getInternalTimerService(AbstractStreamOperator.java:764) > at > org.apache.flink.streaming.api.operators.KeyedProcessOperator.open(KeyedProcessOperator.java:61) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.EOFException > at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) > at org.apache.flink.types.StringValue.readString(StringValue.java:769) > at > org.apache.flink.api.common.typeutils.base.StringSerializer.deserialize(StringSerializer.java:69) > at > org.apache.flink.api.common.typeutils.base.StringSerializer.deserialize(StringSerializer.java:28) > at > org.apache.flink.api.java.typeutils.runtime.RowSeriali
Re: [PR] [FLINK-33703] Use maven 3.8.6 for release profile [flink-shaded]
snuyanzin commented on PR #131: URL: https://github.com/apache/flink-shaded/pull/131#issuecomment-2666253912 Addressed feedback also noticed that latest jackson 2.18.2 requires maven-shade-plugin 3.6.0 since it has a version folder for java 22. And maven-shade-plugin 3.6.0 requires at least maven 3.6.3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-33703) Use maven 3.8.6 for releasing of flink-shaded
[
https://issues.apache.org/jira/browse/FLINK-33703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin resolved FLINK-33703.
-
Resolution: Fixed
> Use maven 3.8.6 for releasing of flink-shaded
> -
>
> Key: FLINK-33703
> URL: https://issues.apache.org/jira/browse/FLINK-33703
> Project: Flink
> Issue Type: Technical Debt
> Components: BuildSystem / Shaded
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Major
> Labels: pull-request-available
> Fix For: shaded-20.0
>
>
> Currently there is maven-enforcer-plugin configuration (for release only)
> {noformat}
>
>
> (,3.3)
>
> {noformat}
> which seems to be outdated and for ci 3.8.6 is used
> We should keep them in sync and use 3.8.6 for both
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33703) Use maven 3.8.6 for releasing of flink-shaded
[
https://issues.apache.org/jira/browse/FLINK-33703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928150#comment-17928150
]
Sergey Nuyanzin commented on FLINK-33703:
-
Merged as
[8b3972eb102c42fd247b9e2d66d50a7959ffd5b7|https://github.com/apache/flink-shaded/commit/8b3972eb102c42fd247b9e2d66d50a7959ffd5b7]
> Use maven 3.8.6 for releasing of flink-shaded
> -
>
> Key: FLINK-33703
> URL: https://issues.apache.org/jira/browse/FLINK-33703
> Project: Flink
> Issue Type: Technical Debt
> Components: BuildSystem / Shaded
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Major
> Labels: pull-request-available
>
> Currently there is maven-enforcer-plugin configuration (for release only)
> {noformat}
>
>
> (,3.3)
>
> {noformat}
> which seems to be outdated and for ci 3.8.6 is used
> We should keep them in sync and use 3.8.6 for both
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33703) Use maven 3.8.6 for releasing of flink-shaded
[
https://issues.apache.org/jira/browse/FLINK-33703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-33703:
Fix Version/s: shaded-20.0
> Use maven 3.8.6 for releasing of flink-shaded
> -
>
> Key: FLINK-33703
> URL: https://issues.apache.org/jira/browse/FLINK-33703
> Project: Flink
> Issue Type: Technical Debt
> Components: BuildSystem / Shaded
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Major
> Labels: pull-request-available
> Fix For: shaded-20.0
>
>
> Currently there is maven-enforcer-plugin configuration (for release only)
> {noformat}
>
>
> (,3.3)
>
> {noformat}
> which seems to be outdated and for ci 3.8.6 is used
> We should keep them in sync and use 3.8.6 for both
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-30371] Fix the problem of JdbcOutputFormat database connection leak [flink-connector-jdbc]
KevDi commented on PR #75: URL: https://github.com/apache/flink-connector-jdbc/pull/75#issuecomment-2666504445 @fapaul i can try to take a look at it tomorrow. Question would be if it is still enough to do the fix like i provided it here or if the solution from @tamirsagi here https://github.com/apache/flink-connector-jdbc/pull/5#issuecomment-2515475756 would also work. I'm not sure about it because the error seems to come when the ` flush` get an exception. It is then thrown upwords without a chance to close the connection. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-37152][build] Update Flink version to 1.20 [flink-cdc]
ferenc-csaky commented on PR #3868: URL: https://github.com/apache/flink-cdc/pull/3868#issuecomment-2666506471 Well, exchanged the `mongodb7` failure to a `mysql` one.. :) Will check on it tomorrow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-37344) setup_hugo.sh should install the hugo binary only
Gabor Somogyi created FLINK-37344: - Summary: setup_hugo.sh should install the hugo binary only Key: FLINK-37344 URL: https://issues.apache.org/jira/browse/FLINK-37344 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Gabor Somogyi Assignee: Gabor Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] Bump org.apache.hive:hive-exec from 3.1.3 to 4.0.1 in /flink-sql-connector-hive-3.1.3 [flink-connector-hive]
dependabot[bot] opened a new pull request, #31: URL: https://github.com/apache/flink-connector-hive/pull/31 Bumps org.apache.hive:hive-exec from 3.1.3 to 4.0.1. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/flink-connector-hive/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-21912][pyflink] Introduce ResolvedSchema in Python Table API [flink]
autophagy commented on code in PR #26121: URL: https://github.com/apache/flink/pull/26121#discussion_r1959749951 ## flink-python/pyflink/table/catalog.py: ## @@ -1391,3 +1398,418 @@ def of(catalog_name: str, configuration: Configuration, comment: str = None): j_catalog_descriptor = gateway.jvm.org.apache.flink.table.catalog.CatalogDescriptor.of( catalog_name, configuration._j_configuration, comment) return CatalogDescriptor(j_catalog_descriptor) + + +class Column(metaclass=ABCMeta): Review Comment: In Python, the `metaclass` argument defines how the underlying class should be created. In this case we're using `ABCMeta` from the [Abstract Base Classes ](https://docs.python.org/3/library/abc.html) module, which in this context is used to enforce abstract method implementation in subclasses and prevent instantiation of the abstract class `Column` (trying to instantiate it with `Column()` will return an error like `TypeError: Can't instantiate abstract class Column without an implementation for abstract method 'with_comment'`, etc). It's similar to defining a `abstract class` in Java. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-21912][pyflink] Introduce ResolvedSchema in Python Table API [flink]
twalthr commented on code in PR #26121:
URL: https://github.com/apache/flink/pull/26121#discussion_r1959584163
##
flink-python/pyflink/table/catalog.py:
##
@@ -1391,3 +1398,418 @@ def of(catalog_name: str, configuration: Configuration,
comment: str = None):
j_catalog_descriptor =
gateway.jvm.org.apache.flink.table.catalog.CatalogDescriptor.of(
catalog_name, configuration._j_configuration, comment)
return CatalogDescriptor(j_catalog_descriptor)
+
+
+class Column(metaclass=ABCMeta):
Review Comment:
For my Python knowledge, what is `metaclass=ABCMeta`?
##
flink-python/pyflink/table/catalog.py:
##
@@ -1391,3 +1398,418 @@ def of(catalog_name: str, configuration: Configuration,
comment: str = None):
j_catalog_descriptor =
gateway.jvm.org.apache.flink.table.catalog.CatalogDescriptor.of(
catalog_name, configuration._j_configuration, comment)
return CatalogDescriptor(j_catalog_descriptor)
+
+
+class Column(metaclass=ABCMeta):
+"""
+Representation of a column in a :class:`~pyflink.table.ResolvedSchema`.
+
+A table column describes either a :class:`PhysicalColumn`,
:class:`ComputedColumn`, or
+:class:`MetadataColumn`.
+"""
+
+def __init__(self, j_column):
+self._j_column = j_column
+
+def __eq__(self, other):
+return self.__class__ == other.__class__ and
self._j_column.equals(other._j_column)
+
+def __hash__(self):
+return self._j_column.hashCode()
+
+def __str__(self):
+return self._j_column.toString()
+
+@staticmethod
+def _from_j_column(j_column) -> Optional["Column"]:
+"""
+Returns a non-abstract column, either a :class:`PhysicalColumn`, a
:class:`ComputedColumn`,
+or a :class:`MetadataColumn` from an
org.apache.flink.table.catalog.Column.
+"""
+if j_column is None:
+return None
+gateway = get_gateway()
+JColumn = gateway.jvm.org.apache.flink.table.catalog.Column
+JPhysicalColumn =
gateway.jvm.org.apache.flink.table.catalog.Column.PhysicalColumn
+JComputedColumn =
gateway.jvm.org.apache.flink.table.catalog.Column.ComputedColumn
+JMetadataColumn =
gateway.jvm.org.apache.flink.table.catalog.Column.MetadataColumn
+j_clz = j_column.getClass()
+
+if not get_java_class(JColumn).isAssignableFrom(j_clz):
+raise TypeError("The input %s is not an instance of Column." %
j_column)
+
+if
get_java_class(JPhysicalColumn).isAssignableFrom(j_column.getClass()):
+return PhysicalColumn(j_physical_column=j_column.getClass())
+elif
get_java_class(JComputedColumn).isAssignableFrom(j_column.getClass()):
+return MetaDataColumn(j_metadata_column=j_column.getClass())
+elif
get_java_class(JMetadataColumn).isAssignableFrom(j_column.getClass()):
+return MetaDataColumn(j_metadata_column=j_column.getClass())
+else:
+return None
+
+@staticmethod
+def physical(name: str, data_type: DataType) -> "PhysicalColumn":
+"""
+Creates a regular table column that represents physical data.
+"""
+gateway = get_gateway()
+j_data_type = _to_java_data_type(data_type)
+j_physical_column =
gateway.jvm.org.apache.flink.table.catalog.Column.physical(
+name, j_data_type
+)
+return PhysicalColumn(j_physical_column)
+
+@staticmethod
+def computed(name: str, resolved_expression: ResolvedExpression) ->
"ComputedColumn":
+"""
+Creates a computed column that is computed from the given
+:class:`~pyflink.table.ResolvedExpression`.
+"""
+gateway = get_gateway()
+j_resolved_expression = resolved_expression
+j_computed_column =
gateway.jvm.org.apache.flink.table.catalog.Column.computed(
+name, j_resolved_expression
+)
+return ComputedColumn(j_computed_column)
+
+@staticmethod
+def metadata(
+name: str, data_type: DataType, metadata_key: Optional[str],
is_virtual: bool
+) -> "MetaDataColumn":
+"""
+Creates a metadata column from metadata of the given column name or
from metadata of the
+given key (if not null).
+
+Allows to specify whether the column is virtual or not.
+"""
+gateway = get_gateway()
+j_data_type = _to_java_data_type(data_type)
+j_metadata_column =
gateway.jvm.org.apache.flink.table.catalog.Column.metadata(
+name, j_data_type, metadata_key, is_virtual
+)
+return MetaDataColumn(j_metadata_column)
+
+@abstractmethod
+def with_comment(self, comment: Optional[str]):
+"""
+Add the comment to the column and return the new object.
+"""
+pass
+
+@abstractmethod
+def is_physical(self) -> bool:
+"""
+Returns whether the given column
[jira] [Commented] (FLINK-36549) Using the ignore-parse-errors parameter in Debezium/Canal/Maxwell/Ogg JSON results in unexpected data loss.
[
https://issues.apache.org/jira/browse/FLINK-36549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928053#comment-17928053
]
Benchao Li commented on FLINK-36549:
[~xiaoyu1995] Thanks for volunteering to fix this. could you also create PRs
for 1.20 and 1.19 release branch?
> Using the ignore-parse-errors parameter in Debezium/Canal/Maxwell/Ogg JSON
> results in unexpected data loss.
> ---
>
> Key: FLINK-36549
> URL: https://issues.apache.org/jira/browse/FLINK-36549
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.17.2, 1.18.1, 1.19.1
>Reporter: jiangyu
>Assignee: xiaoyu
>Priority: Critical
> Labels: pull-request-available
>
> In Debezium/Canal/Maxwell/Ogg JSON, setting {{ignore-parse-errors}} would
> cause data loss if an operator chained with a format-related operator
> encounters an exception. The reason is that in the deserialization
> implementation of Debezium/Canal/Maxwell/Ogg JSON, enabling the
> {{ignore-parse-errors}} parameter skips exceptions related to the format's
> emitted data. For example, in Canal's JSON code, enabling the
> {{ignore-parse-errors}} parameter catches and skips exceptions for {{emitRow}}
> {code:java}
> @Override
> public void deserialize(@Nullable byte[] message, Collector out)
> throws IOException {
> if (message == null || message.length == 0) {
> return;
> }
> try {
> final JsonNode root = jsonDeserializer.deserializeToJsonNode(message);
> if (database != null) {
> if (!databasePattern
> .matcher(root.get(ReadableMetadata.DATABASE.key).asText())
> .matches()) {
> return;
> }
> }
> if (table != null) {
> if (!tablePattern
> .matcher(root.get(ReadableMetadata.TABLE.key).asText())
> .matches()) {
> return;
> }
> }
> final GenericRowData row = (GenericRowData)
> jsonDeserializer.convertToRowData(root);
> String type = row.getString(2).toString(); // "type" field
> if (OP_INSERT.equals(type)) {
> // "data" field is an array of row, contains inserted rows
> ArrayData data = row.getArray(0);
> for (int i = 0; i < data.size(); i++) {
> GenericRowData insert = (GenericRowData) data.getRow(i,
> fieldCount);
> insert.setRowKind(RowKind.INSERT);
> emitRow(row, insert, out);
> }
> } else if (OP_UPDATE.equals(type)) {
> // "data" field is an array of row, contains new rows
> ArrayData data = row.getArray(0);
> // "old" field is an array of row, contains old values
> ArrayData old = row.getArray(1);
> for (int i = 0; i < data.size(); i++) {
> // the underlying JSON deserialization schema always produce
> GenericRowData.
> GenericRowData after = (GenericRowData) data.getRow(i,
> fieldCount);
> GenericRowData before = (GenericRowData) old.getRow(i,
> fieldCount);
> final JsonNode oldField = root.get(FIELD_OLD);
> for (int f = 0; f < fieldCount; f++) {
> if (before.isNullAt(f) &&
> oldField.findValue(fieldNames.get(f)) == null) {
> // fields in "old" (before) means the fields are
> changed
> // fields not in "old" (before) means the fields are
> not changed
> // so we just copy the not changed fields into before
> before.setField(f, after.getField(f));
> }
> }
> before.setRowKind(RowKind.UPDATE_BEFORE);
> after.setRowKind(RowKind.UPDATE_AFTER);
> emitRow(row, before, out);
> emitRow(row, after, out);
> }
> } else if (OP_DELETE.equals(type)) {
> // "data" field is an array of row, contains deleted rows
> ArrayData data = row.getArray(0);
> for (int i = 0; i < data.size(); i++) {
> GenericRowData insert = (GenericRowData) data.getRow(i,
> fieldCount);
> insert.setRowKind(RowKind.DELETE);
> emitRow(row, insert, out);
> }
> } else if (OP_CREATE.equals(type)) {
> // "data" field is null and "type" is "CREATE" which means
> // this is a DDL change event, and we should skip it.
> return;
> } else {
> if (!ignoreParseErrors) {
>
Re: [PR] [FLINK-37336][python] Introduce TablePipeline descriptor to Python Table API [flink]
autophagy commented on code in PR #26167: URL: https://github.com/apache/flink/pull/26167#discussion_r1959596338 ## flink-python/pyflink/table/table.py: ## @@ -1077,6 +1078,89 @@ def explain(self, *extra_details: ExplainDetail) -> str: j_extra_details = to_j_explain_detail_arr(extra_details) return self._j_table.explain(TEXT, j_extra_details) +def insert_into( +self, table_path_or_descriptor: Union[str, TableDescriptor], overwrite: bool = False +) -> TablePipeline: +""" +When ``target_path_or_descriptor`` is a table path: + +Declares that the pipeline defined by the given :class:`Table` (backed by a Review Comment: Is it not the docs for `Table.insertInto`? https://nightlies.apache.org/flink/flink-docs-release-2.0/api/java/org/apache/flink/table/api/Table.html#insertInto(java.lang.String) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-36549) Using the ignore-parse-errors parameter in Debezium/Canal/Maxwell/Ogg JSON results in unexpected data loss.
[
https://issues.apache.org/jira/browse/FLINK-36549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928054#comment-17928054
]
xiaoyu commented on FLINK-36549:
[~libenchao] Sure, I can create PRs for the 1.20 and 1.19 release branches as
well. I'll get started on those and let you know once they're ready for review.
> Using the ignore-parse-errors parameter in Debezium/Canal/Maxwell/Ogg JSON
> results in unexpected data loss.
> ---
>
> Key: FLINK-36549
> URL: https://issues.apache.org/jira/browse/FLINK-36549
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.17.2, 1.18.1, 1.19.1
>Reporter: jiangyu
>Assignee: xiaoyu
>Priority: Critical
> Labels: pull-request-available
>
> In Debezium/Canal/Maxwell/Ogg JSON, setting {{ignore-parse-errors}} would
> cause data loss if an operator chained with a format-related operator
> encounters an exception. The reason is that in the deserialization
> implementation of Debezium/Canal/Maxwell/Ogg JSON, enabling the
> {{ignore-parse-errors}} parameter skips exceptions related to the format's
> emitted data. For example, in Canal's JSON code, enabling the
> {{ignore-parse-errors}} parameter catches and skips exceptions for {{emitRow}}
> {code:java}
> @Override
> public void deserialize(@Nullable byte[] message, Collector out)
> throws IOException {
> if (message == null || message.length == 0) {
> return;
> }
> try {
> final JsonNode root = jsonDeserializer.deserializeToJsonNode(message);
> if (database != null) {
> if (!databasePattern
> .matcher(root.get(ReadableMetadata.DATABASE.key).asText())
> .matches()) {
> return;
> }
> }
> if (table != null) {
> if (!tablePattern
> .matcher(root.get(ReadableMetadata.TABLE.key).asText())
> .matches()) {
> return;
> }
> }
> final GenericRowData row = (GenericRowData)
> jsonDeserializer.convertToRowData(root);
> String type = row.getString(2).toString(); // "type" field
> if (OP_INSERT.equals(type)) {
> // "data" field is an array of row, contains inserted rows
> ArrayData data = row.getArray(0);
> for (int i = 0; i < data.size(); i++) {
> GenericRowData insert = (GenericRowData) data.getRow(i,
> fieldCount);
> insert.setRowKind(RowKind.INSERT);
> emitRow(row, insert, out);
> }
> } else if (OP_UPDATE.equals(type)) {
> // "data" field is an array of row, contains new rows
> ArrayData data = row.getArray(0);
> // "old" field is an array of row, contains old values
> ArrayData old = row.getArray(1);
> for (int i = 0; i < data.size(); i++) {
> // the underlying JSON deserialization schema always produce
> GenericRowData.
> GenericRowData after = (GenericRowData) data.getRow(i,
> fieldCount);
> GenericRowData before = (GenericRowData) old.getRow(i,
> fieldCount);
> final JsonNode oldField = root.get(FIELD_OLD);
> for (int f = 0; f < fieldCount; f++) {
> if (before.isNullAt(f) &&
> oldField.findValue(fieldNames.get(f)) == null) {
> // fields in "old" (before) means the fields are
> changed
> // fields not in "old" (before) means the fields are
> not changed
> // so we just copy the not changed fields into before
> before.setField(f, after.getField(f));
> }
> }
> before.setRowKind(RowKind.UPDATE_BEFORE);
> after.setRowKind(RowKind.UPDATE_AFTER);
> emitRow(row, before, out);
> emitRow(row, after, out);
> }
> } else if (OP_DELETE.equals(type)) {
> // "data" field is an array of row, contains deleted rows
> ArrayData data = row.getArray(0);
> for (int i = 0; i < data.size(); i++) {
> GenericRowData insert = (GenericRowData) data.getRow(i,
> fieldCount);
> insert.setRowKind(RowKind.DELETE);
> emitRow(row, insert, out);
> }
> } else if (OP_CREATE.equals(type)) {
> // "data" field is null and "type" is "CREATE" which means
> // this is a DDL change event, and we should skip it.
> return;
> } else {
>
[jira] [Updated] (FLINK-37342) Add support for smile format in flink-shaded
[ https://issues.apache.org/jira/browse/FLINK-37342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-37342: --- Labels: pull-request-available (was: ) > Add support for smile format in flink-shaded > > > Key: FLINK-37342 > URL: https://issues.apache.org/jira/browse/FLINK-37342 > Project: Flink > Issue Type: Sub-task >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-37342][table] Add `smile` format [flink-shaded]
snuyanzin merged PR #144: URL: https://github.com/apache/flink-shaded/pull/144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-37342) Add support for smile format in flink-shaded
[ https://issues.apache.org/jira/browse/FLINK-37342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin resolved FLINK-37342. - Fix Version/s: shaded-20.0 Resolution: Fixed > Add support for smile format in flink-shaded > > > Key: FLINK-37342 > URL: https://issues.apache.org/jira/browse/FLINK-37342 > Project: Flink > Issue Type: Sub-task >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > Fix For: shaded-20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-37336][python] Introduce TablePipeline descriptor to Python Table API [flink]
autophagy commented on code in PR #26167:
URL: https://github.com/apache/flink/pull/26167#discussion_r1959612343
##
flink-python/pyflink/table/catalog.py:
##
@@ -1391,3 +1391,87 @@ def of(catalog_name: str, configuration: Configuration,
comment: str = None):
j_catalog_descriptor =
gateway.jvm.org.apache.flink.table.catalog.CatalogDescriptor.of(
catalog_name, configuration._j_configuration, comment)
return CatalogDescriptor(j_catalog_descriptor)
+
+
+class ObjectIdentifier(object):
+"""
+Identifies an object in a catalog. It allows to identify objects such as
tables, views,
+function, or types in a catalog. An identifier must be fully qualified. It
is the
+responsibility of the catalog manager to resolve an identifier to an
object.
+
+While Path :class:`ObjectPath` is used within the same catalog, instances
of this class can be
+used across catalogs.
+
+Two objects are considered equal if they share the same object identifier
in a stable session
+context.
+"""
+
+_UNKNOWN = ""
+
+def __init__(self, j_object_identifier):
+self._j_object_identifier = j_object_identifier
+
+def __str__(self):
+return self._j_object_identifier.toString()
+
+def __hash__(self):
+return self._j_object_identifier.hashCode()
+
+def __eq__(self, other):
+return isinstance(other, self.__class__) and
self._j_object_identifier.equals(
+other._j_object_identifier
+)
+
+@staticmethod
+def of(catalog_name: str, database_name: str, object_name: str) ->
"ObjectIdentifier":
+assert catalog_name is not None, "Catalog name must not be null."
+assert database_name is not None, "Database name must not be null."
+assert object_name is not None, "Object name must not be null."
+
+if catalog_name == ObjectIdentifier._UNKNOWN or database_name ==
ObjectIdentifier._UNKNOWN:
+raise ValueError(f"Catalog or database cannot be named
{ObjectIdentifier._UNKNOWN}")
+else:
+gateway = get_gateway()
+j_object_identifier =
gateway.jvm.org.apache.flink.table.catalog.ObjectIdentifier.of(
+catalog_name, database_name, object_name
+)
+return ObjectIdentifier(j_object_identifier=j_object_identifier)
+
+def get_catalog_name(self) -> str:
+return self._j_object_identifier.getCatalogName()
+
+def get_database_name(self) -> str:
+return self._j_object_identifier.getDatabaseName()
+
+def get_object_name(self) -> str:
+return self._j_object_identifier.getObjectName()
+
+def to_object_path(self) -> ObjectPath:
+"""
+Convert this :class:`ObjectIdentifier` to :class:`ObjectPath`.
+
+Throws a TableException if the identifier cannot be converted.
+"""
+j_object_path = self._j_object_identifier.toObjectPath()
+return ObjectPath(j_object_path=j_object_path)
+
+def to_list(self) -> List[str]:
+"""
+List of the component names of this object identifier.
+"""
+return self._j_object_identifier.toList()
+
+def as_serializable_string(self) -> str:
+"""
+Returns a string that fully serializes this instance. The serialized
string can be used for
+transmitting or persisting an object identifier.
+
+Throws a TableException if the identifier cannot be serialized.
Review Comment:
Yes, the underlying Java exception will be returned (albeit wrapped in a
py4j exception). Similar patterns are used elsewhere:
https://github.com/apache/flink/blob/master/flink-python/pyflink/table/catalog.py#L208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-37342) Add support for smile format in flink-shaded
[ https://issues.apache.org/jira/browse/FLINK-37342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928055#comment-17928055 ] Sergey Nuyanzin commented on FLINK-37342: - Merged as [f76d7c13f0877814a08ca477fdb5f8676a882df0|https://github.com/apache/flink-shaded/commit/f76d7c13f0877814a08ca477fdb5f8676a882df0] > Add support for smile format in flink-shaded > > > Key: FLINK-37342 > URL: https://issues.apache.org/jira/browse/FLINK-37342 > Project: Flink > Issue Type: Sub-task >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35622][runtime] Filter out noisy 'Coordinator of operator xxx does not exist' exceptions in CollectResultFetcher [flink]
bgeng777 opened a new pull request, #26171: URL: https://github.com/apache/flink/pull/26171 ## What is the purpose of the change When running large-scale flink batch jobs with the collect sink(e.g. running tpc-ds tests), the Flink JobManager frequently logs "Coordinator of operator does not exist or the job vertex this operator belongs to is not initialized." exceptions when using the collect() method. This exception is caused by the collect sink attempting to fetch data from the corresponding operator coordinator on the JM based on the operator ID. However, batch jobs do not initialize all job vertices at the beginning, and instead, initialize them in a sequential manner. As a result, when a job vertex has not been initialized yet, the corresponding coordinator cannot be found, leading to the printing of this message. ## Brief change log - Add `CoordinatorNotExistException` that extends `FlinkException` - Use `CoordinatorNotExistException` to improve relevant exception - Filter out `CoordinatorNotExistException` in `CollectResultFetcher` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35622) Filter out noisy "Coordinator of operator xxxx does not exist" exceptions in batch mode
[ https://issues.apache.org/jira/browse/FLINK-35622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35622: --- Labels: pull-request-available (was: ) > Filter out noisy "Coordinator of operator does not exist" exceptions in > batch mode > --- > > Key: FLINK-35622 > URL: https://issues.apache.org/jira/browse/FLINK-35622 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Junrui Li >Priority: Major > Labels: pull-request-available > > In batch mode, the Flink JobManager logs frequently print "Coordinator of > operator does not exist or the job vertex this operator belongs to is > not initialized." exceptions when using the collect() method. > This exception is caused by the collect sink attempting to fetch data from > the corresponding operator coordinator on the JM based on the operator ID. > However, batch jobs do not initialize all job vertices at the beginning, and > instead, initialize them in a sequential manner. If a job vertex has not been > initialized yet, the corresponding coordinator cannot be found, leading to > the printing of this message. > These exceptions are harmless and do not affect the job execution, but they > can clutter the logs and make it difficult to find relevant information, > especially for large-scale batch jobs. > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-37152][build] Update Flink version to 1.20 [flink-cdc]
ferenc-csaky commented on PR #3868: URL: https://github.com/apache/flink-cdc/pull/3868#issuecomment-2665508415 @lvyanquan seemed to be some intermittent issue, the CI I ran today succeeded at that step, but since in the meantime Flink 1.20.1 got released IMO it makes sense to update to that right away, so I updated `1.20.0` to `1.20.1` in all relevant places, except the CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
