[PR] [FLINK-35868][mongo] Add MongoDB 6.0 & 7.0 tests [flink-cdc]

2024-07-23 Thread via GitHub
yuxiqian opened a new pull request, #3489: URL: https://github.com/apache/flink-cdc/pull/3489 This closes FLINK-35868. It allows Mongo CDC test cases running on 6.0.16 & 7.0.12 (and legacy 5.0.2). Notice: since JUnit doesn't allow parameterized `@ClassRule` or static fields, so

Re: [PR] [FLINK-35868][mongo] Add MongoDB 6.0 & 7.0 tests [flink-cdc]

2024-07-23 Thread via GitHub
yuxiqian commented on PR #3489: URL: https://github.com/apache/flink-cdc/pull/3489#issuecomment-2244493624 @leonardBang @Jiabao-Sun PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[jira] [Updated] (FLINK-35868) Bump Mongo driver version to support Mongo 7.0+

2024-07-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35868: --- Labels: pull-request-available (was: ) > Bump Mongo driver version to support Mongo 7.0+ >

[jira] [Created] (FLINK-35876) mysql cdc may change from snapshot mode to binlog mode failed when cpu load of mysql server is 100%

2024-07-23 Thread Youle (Jira)
Youle created FLINK-35876: - Summary: mysql cdc may change from snapshot mode to binlog mode failed when cpu load of mysql server is 100% Key: FLINK-35876 URL: https://issues.apache.org/jira/browse/FLINK-35876

[jira] [Updated] (FLINK-35876) mysql cdc may change from snapshot mode to binlog mode failed when cpu load of mysql server is 100%

2024-07-23 Thread Youle (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Youle updated FLINK-35876: -- Attachment: image-2024-07-23-15-47-10-376.png Description: I use mysql cdc to read data from mysql to wri

[jira] [Updated] (FLINK-35876) mysql cdc may change from snapshot mode to binlog mode failed when cpu load of mysql server is 100%

2024-07-23 Thread Youle (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Youle updated FLINK-35876: -- Description: I use mysql cdc to read data from mysql to write to doris, mysql cdc  may change from sanpshot m

[jira] [Updated] (FLINK-35876) mysql cdc may change from snapshot mode to binlog mode failed when cpu load of mysql server is 100%

2024-07-23 Thread Youle (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Youle updated FLINK-35876: -- Description: I use mysql cdc to read data from mysql to write to doris, mysql cdc  may change from sanpshot m

[jira] [Created] (FLINK-35877) Shade protobuf in flink

2024-07-23 Thread zhuanshenbsj1 (Jira)
zhuanshenbsj1 created FLINK-35877: - Summary: Shade protobuf in flink Key: FLINK-35877 URL: https://issues.apache.org/jira/browse/FLINK-35877 Project: Flink Issue Type: Improvement C

[jira] [Updated] (FLINK-35876) mysql cdc may change from snapshot mode to binlog mode failed when cpu load of mysql server is 100%

2024-07-23 Thread Youle (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Youle updated FLINK-35876: -- Component/s: Flink CDC Affects Version/s: cdc-3.1.0 > mysql cdc may change from snapshot mode to bin

[PR] [FLINK-35877] Shade protobuf in flink [flink]

2024-07-23 Thread via GitHub
zhuanshenbsj1 opened a new pull request, #25112: URL: https://github.com/apache/flink/pull/25112 ## What is the purpose of the change Shade the classes in protobuf to avoid class conflict. ## Brief change log *(for example:)* - *The TaskInfo is stored in

[jira] [Updated] (FLINK-35877) Shade protobuf in flink

2024-07-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35877: --- Labels: pull-request-available (was: ) > Shade protobuf in flink > ---

[jira] [Created] (FLINK-35880) CLONE - [Release-1.20] Stage source and binary releases on dist.apache.org

2024-07-23 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-35880: -- Summary: CLONE - [Release-1.20] Stage source and binary releases on dist.apache.org Key: FLINK-35880 URL: https://issues.apache.org/jira/browse/FLINK-35880 Project: Flink

[jira] [Updated] (FLINK-35881) [1.20-rc2] Propose a pull request for website updates

2024-07-23 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35881: --- Summary: [1.20-rc2] Propose a pull request for website updates (was: CLONE - [Release-1.20] Propose

[jira] [Created] (FLINK-35881) CLONE - [Release-1.20] Propose a pull request for website updates

2024-07-23 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-35881: -- Summary: CLONE - [Release-1.20] Propose a pull request for website updates Key: FLINK-35881 URL: https://issues.apache.org/jira/browse/FLINK-35881 Project: Flink

[jira] [Created] (FLINK-35878) Build Release Candidate: 1.20.0-rc2

2024-07-23 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-35878: -- Summary: Build Release Candidate: 1.20.0-rc2 Key: FLINK-35878 URL: https://issues.apache.org/jira/browse/FLINK-35878 Project: Flink Issue Type: New Feature A

[jira] [Created] (FLINK-35879) CLONE - [Release-1.20] Build and stage Java and Python artifacts

2024-07-23 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-35879: -- Summary: CLONE - [Release-1.20] Build and stage Java and Python artifacts Key: FLINK-35879 URL: https://issues.apache.org/jira/browse/FLINK-35879 Project: Flink

[jira] [Created] (FLINK-35882) CLONE - [Release-1.20] Vote on the release candidate

2024-07-23 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-35882: -- Summary: CLONE - [Release-1.20] Vote on the release candidate Key: FLINK-35882 URL: https://issues.apache.org/jira/browse/FLINK-35882 Project: Flink Issue Type:

[jira] [Updated] (FLINK-35880) [1.20-rc2] Stage source and binary releases on dist.apache.org

2024-07-23 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35880: --- Summary: [1.20-rc2] Stage source and binary releases on dist.apache.org (was: CLONE - [Release-1.20

[jira] [Updated] (FLINK-35881) [1.20-rc2] Propose a pull request for website updates

2024-07-23 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35881: --- Affects Version/s: 1.20.0 (was: 1.17.0) > [1.20-rc2] Propose a pull reque

[jira] [Updated] (FLINK-35879) [1.20-rc2] Build and stage Java and Python artifacts

2024-07-23 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35879: --- Summary: [1.20-rc2] Build and stage Java and Python artifacts (was: CLONE - [Release-1.20] Build an

[jira] [Updated] (FLINK-35882) [1.20-rc2] Vote on the release candidate

2024-07-23 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35882: --- Summary: [1.20-rc2] Vote on the release candidate (was: CLONE - [Release-1.20] Vote on the release

[jira] [Updated] (FLINK-35882) [1.20-rc2] Vote on the release candidate

2024-07-23 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35882: --- Affects Version/s: 1.20.0 (was: 1.17.0) > [1.20-rc2] Vote on the release

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-07-23 Thread via GitHub
hlteoh37 commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1687672586 ## flink-connector-aws/flink-connector-sqs/src/test/java/org/apache/flink/connector/sqs/sink/SqsSinkBuilderTest.java: ## @@ -25,14 +25,6 @@ /** Covers cons

Re: [PR] [FLINK-35877] Shade protobuf in flink [flink]

2024-07-23 Thread via GitHub
flinkbot commented on PR #25112: URL: https://github.com/apache/flink/pull/25112#issuecomment-2244623653 ## CI report: * 8a98ca6989b8a1192f72c4d0fd58529e99f9bfd5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

Re: [PR] [FLINK-35737] Prevent Memory Leak by Closing MemoryExecutionGraphInfoStore on MiniCluster Shutdown [flink]

2024-07-23 Thread via GitHub
fengjiajie commented on PR #25009: URL: https://github.com/apache/flink/pull/25009#issuecomment-2244648203 Hi @Samrat002, I was wondering if you had a chance to take another look at this PR. Please let me know if any further changes are needed. Thanks! -- This is an automated message from

[jira] [Assigned] (FLINK-35868) Bump Mongo driver version to support Mongo 7.0+

2024-07-23 Thread Jiabao Sun (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun reassigned FLINK-35868: -- Assignee: yux > Bump Mongo driver version to support Mongo 7.0+ > ---

Re: [PR] [FLINK-35623] Bump mongo-driver version from 4.7.2 to 5.1.1 to support MongoDB 7.0 [flink-connector-mongodb]

2024-07-23 Thread via GitHub
Jiabao-Sun commented on PR #36: URL: https://github.com/apache/flink-connector-mongodb/pull/36#issuecomment-2244667146 Hi @yux, could you help review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [FLINK-35868][cdc-connector][mongodb] Add MongoDB 6.0 & 7.0 tests [flink-cdc]

2024-07-23 Thread via GitHub
Jiabao-Sun commented on PR #3489: URL: https://github.com/apache/flink-cdc/pull/3489#issuecomment-2244672599 Shall we bump the driver version from 4.7.1 to 5.1.1 as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[jira] [Commented] (FLINK-35877) Shade protobuf in flink

2024-07-23 Thread Martijn Visser (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868001#comment-17868001 ] Martijn Visser commented on FLINK-35877: This is tagged for Shaded, but it's not

Re: [PR] [FLINK-35868][cdc-connector][mongodb] Bump dependency version to support MongoDB 7.0 [flink-cdc]

2024-07-23 Thread via GitHub
yuxiqian commented on PR #3489: URL: https://github.com/apache/flink-cdc/pull/3489#issuecomment-2244705856 > Shall we bump the driver version from 4.7.1 to 5.1.1 as well? Done, bumped `mongo-kafka` version, too. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] [FLINK-35623] Bump mongo-driver version from 4.7.2 to 5.1.1 to support MongoDB 7.0 [flink-connector-mongodb]

2024-07-23 Thread via GitHub
Jiabao-Sun merged PR #36: URL: https://github.com/apache/flink-connector-mongodb/pull/36 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr

[jira] [Resolved] (FLINK-35623) Bump mongo-driver version from 4.7.2 to 5.1.1 to support MongoDB 7.0

2024-07-23 Thread Jiabao Sun (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun resolved FLINK-35623. Resolution: Implemented main: a7551187d904ed819db085fc36c2cf735913ed5e > Bump mongo-driver versio

Re: [PR] [FLINK-33386][runtime] Support tasks balancing at slot level for Default Scheduler [flink]

2024-07-23 Thread via GitHub
1996fanrui commented on code in PR #23635: URL: https://github.com/apache/flink/pull/23635#discussion_r1687687114 ## flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/LocalInputPreferredSlotSharingStrategyTest.java: ## @@ -69,46 +69,29 @@ class LocalInputPreferredSl

Re: [PR] [FLINK-35877] Shade protobuf in flink [flink]

2024-07-23 Thread via GitHub
zhuanshenbsj1 commented on PR #25112: URL: https://github.com/apache/flink/pull/25112#issuecomment-2244784197 > Which part of the Flink runtime/system relies on Protobuf, that would justify shading Protobuf? This is mainly to avoid the dependency conflict between the protobuf used in

[jira] [Updated] (FLINK-33977) Adaptive scheduler may not minimize the number of TMs during downscaling

2024-07-23 Thread RocMarshal (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-33977: --- Attachment: screenshot-1.png > Adaptive scheduler may not minimize the number of TMs during downscal

[jira] [Commented] (FLINK-33977) Adaptive scheduler may not minimize the number of TMs during downscaling

2024-07-23 Thread RocMarshal (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868012#comment-17868012 ] RocMarshal commented on FLINK-33977: Thanks [~Zhanghao Chen] for reporting it. Thi

Re: [PR] [FLINK-35874][cdc-connector][mysql] Check pureBinlogPhaseTables set before call getBinlogPosition method [flink-cdc]

2024-07-23 Thread via GitHub
qiaozongmi commented on PR #3488: URL: https://github.com/apache/flink-cdc/pull/3488#issuecomment-2244820245 @leonardBang PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[jira] [Updated] (FLINK-35874) Check pureBinlogPhaseTables set before call getBinlogPosition method in BinlogSplitReader

2024-07-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35874: --- Labels: pull-request-available (was: ) > Check pureBinlogPhaseTables set before call getBin

[PR] [FLINK-33874][runtime] Support resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler [flink]

2024-07-23 Thread via GitHub
RocMarshal opened a new pull request, #25113: URL: https://github.com/apache/flink/pull/25113 ## What is the purpose of the change Support resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler ## Brief change log - intro

[jira] [Updated] (FLINK-33874) Support resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler

2024-07-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33874: --- Labels: pull-request-available (was: ) > Support resource request wait mechanism at Default

Re: [PR] [FLINK-33874][runtime] Support resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler [flink]

2024-07-23 Thread via GitHub
flinkbot commented on PR #25113: URL: https://github.com/apache/flink/pull/25113#issuecomment-2244834549 ## CI report: * 7e539f5fc908f38eebba16f9480bcf3f5d5009c3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[jira] [Commented] (FLINK-35883) Wildcard projection inserts column at wrong place

2024-07-23 Thread yux (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868022#comment-17868022 ] yux commented on FLINK-35883: - I'd like to investigate this based on FLINK-35272. > Wildcar

[jira] [Created] (FLINK-35883) Wildcard projection inserts column at wrong place

2024-07-23 Thread yux (Jira)
yux created FLINK-35883: --- Summary: Wildcard projection inserts column at wrong place Key: FLINK-35883 URL: https://issues.apache.org/jira/browse/FLINK-35883 Project: Flink Issue Type: Bug Com

Re: [PR] [FLINK-35272][cdc][runtime] Pipeline Transform job supports omitting / renaming calculation column [flink-cdc]

2024-07-23 Thread via GitHub
leonardBang commented on code in PR #3285: URL: https://github.com/apache/flink-cdc/pull/3285#discussion_r1687881675 ## flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/ddl/data_types_test.sql: ## @@ -0,0 +1,28 @@ +-- Licensed to the Apache Software Foundation

Re: [PR] [FLINK-35868][cdc-connector][mongodb] Bump dependency version to support MongoDB 7.0 [flink-cdc]

2024-07-23 Thread via GitHub
leonardBang commented on code in PR #3489: URL: https://github.com/apache/flink-cdc/pull/3489#discussion_r1687920162 ## flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/pom.xml: ## @@ -69,7 +69,7 @@ limitations under the License. o

[jira] [Created] (FLINK-35884) Support

2024-07-23 Thread JunboWang (Jira)
JunboWang created FLINK-35884: - Summary: Support Key: FLINK-35884 URL: https://issues.apache.org/jira/browse/FLINK-35884 Project: Flink Issue Type: Improvement Components: Flink CDC

[jira] [Updated] (FLINK-35884) Pipeline connector MySQL support snapshot.chunk.key-column

2024-07-23 Thread JunboWang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JunboWang updated FLINK-35884: -- Summary: Pipeline connector MySQL support snapshot.chunk.key-column (was: Support) > Pipeline connec

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
ferenc-csaky commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1687799832 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/KubernetesOperatorConfigOptions.java: ## @@ -426,6 +438,14

[jira] [Commented] (FLINK-33977) Adaptive scheduler may not minimize the number of TMs during downscaling

2024-07-23 Thread Zhanghao Chen (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868038#comment-17868038 ] Zhanghao Chen commented on FLINK-33977: --- Hi [~RocMarshal], there's no progress on

Re: [PR] [FLINK-35868][cdc-connector][mongodb] Bump dependency version to support MongoDB 7.0 [flink-cdc]

2024-07-23 Thread via GitHub
yuxiqian commented on PR #3489: URL: https://github.com/apache/flink-cdc/pull/3489#issuecomment-2245079611 I'll investigate this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[jira] [Updated] (FLINK-35884) Pipeline connector MySQL support snapshot.chunk.key-column

2024-07-23 Thread JunboWang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JunboWang updated FLINK-35884: -- Description: flink-connector-mysql-cdc `MySqlSourceOptions` supports specifying the parameter scan.inc

[jira] [Updated] (FLINK-35884) Pipeline connector MySQL support scan.incremental.snapshot.chunk.key-column

2024-07-23 Thread JunboWang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JunboWang updated FLINK-35884: -- Summary: Pipeline connector MySQL support scan.incremental.snapshot.chunk.key-column (was: Pipeline c

[jira] [Updated] (FLINK-35884) Pipeline MySQL connector support scan.incremental.snapshot.chunk.key-column

2024-07-23 Thread JunboWang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JunboWang updated FLINK-35884: -- Summary: Pipeline MySQL connector support scan.incremental.snapshot.chunk.key-column (was: Pipeline c

[PR] [FLINK-35884] MySQL pipeline support snapshot chunk key-column [flink-cdc]

2024-07-23 Thread via GitHub
beryllw opened a new pull request, #3490: URL: https://github.com/apache/flink-cdc/pull/3490 flink-connector-mysql-cdc `MySqlSourceOptions` supports specifying the parameter scan.incremental.snapshot.chunk.key-column to divide chunks, pipeline connector should also support. -- This is an

[jira] [Updated] (FLINK-35884) Pipeline MySQL connector supports setting chunk column key

2024-07-23 Thread JunboWang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JunboWang updated FLINK-35884: -- Summary: Pipeline MySQL connector supports setting chunk column key (was: Pipeline MySQL connector su

[jira] [Updated] (FLINK-35884) Pipeline MySQL connector supports setting chunk column key

2024-07-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35884: --- Labels: pull-request-available (was: ) > Pipeline MySQL connector supports setting chunk co

Re: [PR] [FLINK-35242] Supports per-SE type configuration & "lenient" evolution behavior [flink-cdc]

2024-07-23 Thread via GitHub
leonardBang commented on code in PR #3339: URL: https://github.com/apache/flink-cdc/pull/3339#discussion_r1687943729 ## flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/DataSink.java: ## @@ -30,5 +33,5 @@ public interface DataSink { EventSinkProvider getEvent

Re: [PR] [FLINK-35857] fix redeploy failed deployment without latest checkpoint [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
gyfora commented on code in PR #855: URL: https://github.com/apache/flink-kubernetes-operator/pull/855#discussion_r1687973449 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java: ## @@ -289,6 +289,8 @@

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-07-23 Thread via GitHub
hlteoh37 commented on PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#issuecomment-2245153099 @19priyadhingra The tests seems to be failing. Can we please take a look? Also - it would be good if we squash the commits! -- This is an automated message from the Ap

[jira] [Created] (FLINK-35885) SlicingWindowOperator should not process watermark with proctime

2024-07-23 Thread Baozhu Zhao (Jira)
Baozhu Zhao created FLINK-35885: --- Summary: SlicingWindowOperator should not process watermark with proctime Key: FLINK-35885 URL: https://issues.apache.org/jira/browse/FLINK-35885 Project: Flink

[jira] [Commented] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-07-23 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868069#comment-17868069 ] Gyula Fora commented on FLINK-35285: Hey [~trystan] sorry for the late reply. I see

Re: [PR] [FLINK-35776] Simplify job status handling [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
gyfora merged PR #851: URL: https://github.com/apache/flink-kubernetes-operator/pull/851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688087051 ## flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/spec/JobSpec.java: ## @@ -90,8 +100,8 @@ public class JobSp

[jira] [Closed] (FLINK-35776) Simplify job observe logic

2024-07-23 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-35776. -- Resolution: Fixed merged to main 15f648ce46537d6a1df3a87cfd653a3f855d0dcd > Simplify job observe logi

Re: [PR] [FLINK-32682] Make it possible to use query time based time functions in streaming mode [flink]

2024-07-23 Thread via GitHub
dawidwys closed pull request #23083: [FLINK-32682] Make it possible to use query time based time functions in streaming mode URL: https://github.com/apache/flink/pull/23083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[jira] [Updated] (FLINK-35885) SlicingWindowOperator should not process watermark with proctime

2024-07-23 Thread Baozhu Zhao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Baozhu Zhao updated FLINK-35885: Description: We have discovered an unexpected case where abnormal data with a count of 0 occurs w

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688100790 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/KubernetesOperatorConfigOptions.java: ## @@ -426,6 +438,14 @

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688104509 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkStateSnapshotContext.java: ## @@ -0,0 +1,112 @@ +/*

Re: [PR] [FLINK-35292] Set dummy savepoint path during last-state upgrade [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
gyfora commented on code in PR #849: URL: https://github.com/apache/flink-kubernetes-operator/pull/849#discussion_r1688103240 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java: ## @@ -265,7 +274,7 @@

[jira] [Closed] (FLINK-35292) Set dummy savepoint path during last-state upgrade

2024-07-23 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-35292. -- Resolution: Fixed merged to main 207b149f9b556d68c4ab98d16cdde2f7820659c0 > Set dummy savepoint path

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688126716 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkStateSnapshotController.java: ## @@ -0,0 +1,216 @@

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688136504 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/SnapshotObserver.java: ## @@ -273,15 +280,19 @@ private vo

Re: [PR] [FLINK-35857] fix redeploy failed deployment without latest checkpoint [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
chenyuzhi459 commented on code in PR #855: URL: https://github.com/apache/flink-kubernetes-operator/pull/855#discussion_r1688138227 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java: ## @@ -289,6 +28

Re: [PR] [FLINK-35857] fix redeploy failed deployment without latest checkpoint [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
gyfora commented on code in PR #855: URL: https://github.com/apache/flink-kubernetes-operator/pull/855#discussion_r1688196270 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractJobReconciler.java: ## @@ -289,6 +289,8 @@

Re: [PR] [FLINK-35272][cdc][runtime] Pipeline Transform job supports omitting / renaming calculation column [flink-cdc]

2024-07-23 Thread via GitHub
yuxiqian commented on PR #3285: URL: https://github.com/apache/flink-cdc/pull/3285#issuecomment-2245470485 Done, rebased due to some conflicts with https://github.com/apache/flink-cdc/commit/26ff6d2a081181f3df7aa49d65d804c57c634122. Will add `CAST ... AS` tests after #3357 got merged. --

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688221106 ## flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkStateSnapshotState.java: ## @@ -0,0 +1,39 @@ +/

[jira] [Commented] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-07-23 Thread Trystan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868093#comment-17868093 ] Trystan commented on FLINK-35285: -   {noformat} As long as your job parallelism is very

[jira] [Comment Edited] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-07-23 Thread Trystan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868093#comment-17868093 ] Trystan edited comment on FLINK-35285 at 7/23/24 3:00 PM: -- {nof

[PR] [FLINK-22748][connector-kafka] Allow dynamic target topic selection in SQL Kafka sinks [flink-connector-kafka]

2024-07-23 Thread via GitHub
klam-shop opened a new pull request, #109: URL: https://github.com/apache/flink-connector-kafka/pull/109 ## What is the purpose of the change Allows writing to different Kafka topics based on the `topic` metadata column value in SQL, and updates the Table API's `KafkaDynamicSink` to accep

Re: [PR] [FLINK-22748][connector-kafka] Allow dynamic target topic selection in SQL Kafka sinks [flink]

2024-07-23 Thread via GitHub
klam-shop commented on PR #16142: URL: https://github.com/apache/flink/pull/16142#issuecomment-2245500551 Hi I've taken some time to build upon [Nicholas Jiang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=nicholasjiang) 's PR and port it to the new kafka connector repo:

[jira] [Comment Edited] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-07-23 Thread Trystan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868093#comment-17868093 ] Trystan edited comment on FLINK-35285 at 7/23/24 3:02 PM: -- {nof

[jira] [Comment Edited] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-07-23 Thread Trystan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868093#comment-17868093 ] Trystan edited comment on FLINK-35285 at 7/23/24 3:03 PM: -- {nof

[jira] [Commented] (FLINK-22748) Allow dynamic target topic selection in SQL Kafka sinks

2024-07-23 Thread Kevin Lam (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868094#comment-17868094 ] Kevin Lam commented on FLINK-22748: --- Hi I've taken some time to build upon [~nicholasj

[jira] [Comment Edited] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-07-23 Thread Trystan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868093#comment-17868093 ] Trystan edited comment on FLINK-35285 at 7/23/24 3:03 PM: -- {nof

Re: [PR] [FLINK-22748][connector-kafka] Allow dynamic target topic selection in SQL Kafka sinks [flink-connector-kafka]

2024-07-23 Thread via GitHub
klam-shop commented on code in PR #109: URL: https://github.com/apache/flink-connector-kafka/pull/109#discussion_r1688236979 ## flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaRecordSerializationSchema.java: ## @@ -144,14 +147,3

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688262282 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/controller/FlinkStateSnapshotContext.java: ## @@ -0,0 +1,112 @@ +/*

[jira] [Created] (FLINK-35886) Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked

2024-07-23 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-35886: -- Summary: Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked Key: FLINK-35886 URL: https://issues.apache.org/jira/browse/FLINK-35886

[jira] [Assigned] (FLINK-35886) Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked

2024-07-23 Thread Piotr Nowojski (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski reassigned FLINK-35886: -- Assignee: Piotr Nowojski > Incorrect watermark idleness timeout accounting when subta

[jira] [Commented] (FLINK-35886) Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked

2024-07-23 Thread Piotr Nowojski (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868098#comment-17868098 ] Piotr Nowojski commented on FLINK-35886: I think this problem requires a public

Re: [PR] [FLINK-35272][cdc][runtime] Pipeline Transform job supports omitting / renaming calculation column [flink-cdc]

2024-07-23 Thread via GitHub
yuxiqian commented on code in PR #3285: URL: https://github.com/apache/flink-cdc/pull/3285#discussion_r1688271544 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformFilterProcessor.java: ## Review Comment: I modified expression eval

Re: [PR] [FLINK-35272][cdc][runtime] Pipeline Transform job supports omitting / renaming calculation column [flink-cdc]

2024-07-23 Thread via GitHub
yuxiqian commented on code in PR #3285: URL: https://github.com/apache/flink-cdc/pull/3285#discussion_r1688271544 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformFilterProcessor.java: ## Review Comment: I modified expression eval

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-07-23 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1688281873 ## flink-connector-aws/flink-connector-sqs/src/test/java/org/apache/flink/connector/sqs/sink/SqsSinkBuilderTest.java: ## @@ -25,14 +25,6 @@ /** Cover

[jira] [Comment Edited] (FLINK-35886) Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked

2024-07-23 Thread Piotr Nowojski (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868098#comment-17868098 ] Piotr Nowojski edited comment on FLINK-35886 at 7/23/24 3:45 PM: -

[jira] [Created] (FLINK-35887) Null Pointer Exception in TypeExtractor.isRecord when trying to provide type info for interface

2024-07-23 Thread Jacob Jona Fahlenkamp (Jira)
Jacob Jona Fahlenkamp created FLINK-35887: - Summary: Null Pointer Exception in TypeExtractor.isRecord when trying to provide type info for interface Key: FLINK-35887 URL: https://issues.apache.org/jira/bro

[jira] [Updated] (FLINK-35887) Null Pointer Exception in TypeExtractor.isRecord when trying to provide type info for interface

2024-07-23 Thread Jacob Jona Fahlenkamp (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Jona Fahlenkamp updated FLINK-35887: -- Priority: Major (was: Minor) > Null Pointer Exception in TypeExtractor.isReco

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688463075 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/listener/AuditUtils.java: ## @@ -53,6 +62,16 @@ private static Stri

Re: [PR] [FLINK-35265] Implement FlinkStateSnapshot custom resource [flink-kubernetes-operator]

2024-07-23 Thread via GitHub
mateczagany commented on code in PR #821: URL: https://github.com/apache/flink-kubernetes-operator/pull/821#discussion_r1688472333 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java: ## @@ -716,6 +695,48 @@ public C

[jira] [Commented] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-07-23 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868168#comment-17868168 ] Gyula Fora commented on FLINK-35285: {noformat} I would argue that a current paralle

[jira] [Commented] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-07-23 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868169#comment-17868169 ] Gyula Fora commented on FLINK-35285: It's much better to not scale down than to unde

[PR] [DRAFT][FLINK-34440][formats][protobuf-confluent] Protobuf confluent dynamic format [flink]

2024-07-23 Thread via GitHub
dmariassy opened a new pull request, #25114: URL: https://github.com/apache/flink/pull/25114 **This is a DRAFT PR**. ## TODO - Add missing boilerplate (e.g. config) - Add debezium support - Test the format in Shopify Flink jobs - Docs (might be a separate PR) ##

  1   2   >