[jira] [Resolved] (FLINK-37532) Job recovery always starts from initialSavepointPath when JobManager HA is disabled in case of jobManger failure

2025-03-23 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-37532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora resolved FLINK-37532. Resolution: Not A Problem > Job recovery always starts from initialSavepointPath when JobManager H

Re: [PR] [FLINK-36684][cdc-connector][mysql] Support read changelog as append only mode [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on PR #3708: URL: https://github.com/apache/flink-cdc/pull/3708#issuecomment-2746993395 Add please rebase to solve the conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [FLINK-36931][cdc] FlinkCDC YAML supports batch mode [flink-cdc]

2025-03-23 Thread via GitHub
aiwenmo commented on code in PR #3812: URL: https://github.com/apache/flink-cdc/pull/3812#discussion_r2009513615 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/common/TableIdRouter.java: ## @@ -95,4 +96,23 @@ private TableId resolveReplacement(

Re: [PR] [FLINK-36931][cdc] FlinkCDC YAML supports batch mode [flink-cdc]

2025-03-23 Thread via GitHub
aiwenmo commented on code in PR #3812: URL: https://github.com/apache/flink-cdc/pull/3812#discussion_r2009509031 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/common/TableIdRouter.java: ## @@ -76,7 +77,7 @@ public List route(TableId sourceTable

Re: [PR] [FLINK-36931][cdc] FlinkCDC YAML supports batch mode [flink-cdc]

2025-03-23 Thread via GitHub
aiwenmo commented on code in PR #3812: URL: https://github.com/apache/flink-cdc/pull/3812#discussion_r2009509031 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/common/TableIdRouter.java: ## @@ -76,7 +77,7 @@ public List route(TableId sourceTable

Re: [PR] [FLINK-37507] Fix MySQL CDC accidentally captures common-prefix database when `scan.binlog.newly-added-table` is enabled [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on PR #3957: URL: https://github.com/apache/flink-cdc/pull/3957#issuecomment-2746968883 PTAL @leonardBang. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] [FLINK-34558][state] Support tracking state size [flink]

2025-03-23 Thread via GitHub
hejufang commented on PR #25837: URL: https://github.com/apache/flink/pull/25837#issuecomment-2746773866 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [FLINK-37222][table] Don't resolve the CatalogView to QueryOperationCatalogView [flink]

2025-03-23 Thread via GitHub
fsk119 merged PR #26323: URL: https://github.com/apache/flink/pull/26323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache

Re: [PR] [hotfix] Short circuit judgment for create blob storage dir [flink]

2025-03-23 Thread via GitHub
flinkbot commented on PR #26338: URL: https://github.com/apache/flink/pull/26338#issuecomment-2746914354 ## CI report: * 1d19be629ec14ee2a6bed28aa33a8c0bd3b68329 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

Re: [PR] [FLINK-36931][cdc] FlinkCDC YAML supports batch mode [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on code in PR #3812: URL: https://github.com/apache/flink-cdc/pull/3812#discussion_r2009421299 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/common/TableIdRouter.java: ## @@ -95,4 +96,23 @@ private TableId resolveReplacement

Re: [PR] [FLINK-34558][state] Support tracking state size [flink]

2025-03-23 Thread via GitHub
hejufang commented on PR #25837: URL: https://github.com/apache/flink/pull/25837#issuecomment-2746233755 @fredia Thank you for your suggestions. I have updated the relevant descriptions in the metrics.md document. Additionally, I believe the overhead of serializing is quite low, as the

Re: [PR] [hotfix] Rename VoidBlobStore to NoOperationBlobStore [flink]

2025-03-23 Thread via GitHub
flinkbot commented on PR #26337: URL: https://github.com/apache/flink/pull/26337#issuecomment-2746847407 ## CI report: * 23ee73365b4371447ef7a2175215ba6a40f0364e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[PR] [hotfix] Short circuit judgment for create blob storage dir [flink]

2025-03-23 Thread via GitHub
beliefer opened a new pull request, #26338: URL: https://github.com/apache/flink/pull/26338 ## What is the purpose of the change This PR aims to short circuit judgment for create blob storage dir According to https://github.com/apache/flink/blob/df058589affe786f5dcbf253cd3769502c28

Re: [PR] [FLINK-37507] Fix MySQL CDC accidentally captures common-prefix database when `scan.binlog.newly-added-table` is enabled [flink-cdc]

2025-03-23 Thread via GitHub
yuxiqian commented on code in PR #3957: URL: https://github.com/apache/flink-cdc/pull/3957#discussion_r2009433702 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java

Re: [PR] [WIP][FLINK-37453][tests] Port Ruby migration tests to Java [flink-cdc]

2025-03-23 Thread via GitHub
yuxiqian commented on PR #3965: URL: https://github.com/apache/flink-cdc/pull/3965#issuecomment-2746843623 CI passed in my forked repo, mark it ready for review. This PR is aimed at migration framework with minimal changes for reviewability. More E2e test cases could be added later to

Re: [PR] [FLINK-37507] Fix MySQL CDC accidentally captures common-prefix database when `scan.binlog.newly-added-table` is enabled [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on code in PR #3957: URL: https://github.com/apache/flink-cdc/pull/3957#discussion_r2009429769 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.jav

Re: [PR] [FLINK-37531] Respect the design of chain programming for RpcServiceBuilder [flink]

2025-03-23 Thread via GitHub
beliefer commented on PR #26334: URL: https://github.com/apache/flink/pull/26334#issuecomment-2746715794 @xintongsong Thank you for your explanation. I will close this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[PR] [hotfix] Rename VoidBlobStore to NoOperationBlobStore [flink]

2025-03-23 Thread via GitHub
beliefer opened a new pull request, #26337: URL: https://github.com/apache/flink/pull/26337 ## What is the purpose of the change This PR aims to rename `VoidBlobStore` to `NoOperationBlobStore`. ## Brief change log Rename `VoidBlobStore` to `NoOperationBlobStore`.

Re: [PR] [FLINK-35670][cdc-connector][postgres]Flink cdc pipeline support postgres source [flink-cdc]

2025-03-23 Thread via GitHub
AppleJunJiang commented on PR #3442: URL: https://github.com/apache/flink-cdc/pull/3442#issuecomment-2746803586 为啥这么久了还没有合并? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [hotfix] [base] Fix the serialization error occurring with table names containing dot [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on PR #2705: URL: https://github.com/apache/flink-cdc/pull/2705#issuecomment-2746801489 Met the same problem and it‘s indeed helpful. Could @edmond-kk help to rebase to master? -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] [FLINK-37506][transform] Print more descriptive error messages when TransformOperator throws an exception [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on PR #3960: URL: https://github.com/apache/flink-cdc/pull/3960#issuecomment-2746797888 PTAL @leonardBang. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[jira] [Updated] (FLINK-37375) Checkpoint supports the Operator to customize asynchronous snapshot state

2025-03-23 Thread Zakelly Lan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-37375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zakelly Lan updated FLINK-37375: Issue Type: New Feature (was: Bug) > Checkpoint supports the Operator to customize asynchronous s

[jira] [Commented] (FLINK-37375) Checkpoint supports the Operator to customize asynchronous snapshot state

2025-03-23 Thread Zakelly Lan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-37375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17937775#comment-17937775 ] Zakelly Lan commented on FLINK-37375: - [~hejufang001] But it still required to be fi

Re: [PR] [FLINK-37128][pipeline-connector/mysql] Binlog reader skip `shouldEmit` filter when snapshot reader skip backfill [flink-cdc]

2025-03-23 Thread via GitHub
loserwang1024 commented on PR #3874: URL: https://github.com/apache/flink-cdc/pull/3874#issuecomment-2746720174 This idea looks good to me, which will help imporve a lot. I have some advice: 1. we can do this before add stream split. When adding stream split, we can do what org.apache.f

Re: [PR] [FLINK-37531] Respect the design of chain programming for RpcServiceBuilder [flink]

2025-03-23 Thread via GitHub
beliefer closed pull request #26334: [FLINK-37531] Respect the design of chain programming for RpcServiceBuilder URL: https://github.com/apache/flink/pull/26334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[PR] [FLINK-37531] Respect the design of chain programming for RpcServiceBuilder [flink]

2025-03-23 Thread via GitHub
beliefer opened a new pull request, #26334: URL: https://github.com/apache/flink/pull/26334 ## What is the purpose of the change This PR aims to respect the design of chain programming for `RpcServiceBuilder`. According to the design of `RpcServiceBuilder`, the implementation shoul

Re: [PR] [FLINK-37510] Avoid duplicate calculations for SlidingEventTimeWindows [flink]

2025-03-23 Thread via GitHub
beliefer commented on PR #26319: URL: https://github.com/apache/flink/pull/26319#issuecomment-2746720571 > Probably I missed something however can you please describe what is the issue here and how we could check that it is gone after this commit? It's obviously. I have added the desc

Re: [PR] [FLINK-37506][transform] Print more descriptive error messages when TransformOperator throws an exception [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on PR #3960: URL: https://github.com/apache/flink-cdc/pull/3960#issuecomment-2746679194 > Seems `DataSinkWriterOperator` does not manipulate data change events in BinaryRecordData level. Will printing corresponding TableId suffice? Yeah, corresponding TableId and S

[jira] [Created] (FLINK-37537) Duplicate key when job failover in SchemaOperator

2025-03-23 Thread linqigeng (Jira)
linqigeng created FLINK-37537: - Summary: Duplicate key when job failover in SchemaOperator Key: FLINK-37537 URL: https://issues.apache.org/jira/browse/FLINK-37537 Project: Flink Issue Type: Bug

[jira] [Commented] (FLINK-37532) Job recovery always starts from initialSavepointPath when JobManager HA is disabled in case of jobManger failure

2025-03-23 Thread Jun He (Jira)
[ https://issues.apache.org/jira/browse/FLINK-37532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17937764#comment-17937764 ] Jun He commented on FLINK-37532: [~gyfora] Thanks for your reply. I'll enable JM HA. >

Re: [PR] [FLINK-37222][table] Don't resolve the CatalogView to QueryOperationC… [flink]

2025-03-23 Thread via GitHub
fsk119 merged PR #26324: URL: https://github.com/apache/flink/pull/26324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache

[jira] [Comment Edited] (FLINK-37222) Table planner exception when sql client submit job

2025-03-23 Thread Shengkai Fang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-37222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936990#comment-17936990 ] Shengkai Fang edited comment on FLINK-37222 at 3/24/25 1:23 AM: --

[jira] [Closed] (FLINK-37222) Table planner exception when sql client submit job

2025-03-23 Thread Shengkai Fang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-37222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang closed FLINK-37222. - Resolution: Fixed > Table planner exception when sql client submit job > ---

Re: [PR] [FLINK-34558][state] Support tracking state size [flink]

2025-03-23 Thread via GitHub
hejufang commented on code in PR #25837: URL: https://github.com/apache/flink/pull/25837#discussion_r2009127739 ## flink-runtime/src/main/java/org/apache/flink/runtime/state/metrics/MetricsTrackingMapState.java: ## @@ -33,133 +37,223 @@ * @param Type of the user entry key of

Re: [PR] [FLINK-37506][transform] Print more descriptive error messages when TransformOperator throws an exception [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on code in PR #3960: URL: https://github.com/apache/flink-cdc/pull/3960#discussion_r2009049446 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java: ## @@ -274,6 +275,28 @@ public void close() throws Ex

Re: [PR] [FLINK-37511][rest] Use Jackson serialization in JobPlanInfo.Plan [flink]

2025-03-23 Thread via GitHub
Efrat19 commented on PR #26320: URL: https://github.com/apache/flink/pull/26320#issuecomment-2746105572 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [FLINK-37506][transform] Print more descriptive error messages when TransformOperator throws an exception [flink-cdc]

2025-03-23 Thread via GitHub
lvyanquan commented on PR #3960: URL: https://github.com/apache/flink-cdc/pull/3960#issuecomment-2746102551 What's more, could we add same optimization in DataSinkWriterOperator#processElement method? -- This is an automated message from the Apache Git Service. To respond to the message,