Re: [PR] [FLINK-35910][table] Add the built-in function BTRIM [flink]
dylanhz commented on PR #25127: URL: https://github.com/apache/flink/pull/25127#issuecomment-2268322441 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35884][mysql-pipeline-connector] MySQL pipeline support snapshot chunk key-column [flink-cdc]
beryllw commented on PR #3490: URL: https://github.com/apache/flink-cdc/pull/3490#issuecomment-2268338432 [compile_and_test](https://github.com/apache/flink-cdc/actions/runs/10195953793/job/28335622434?pr=3490#logs) ``` Error: Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 122.202 s <<< FAILURE! - in org.apache.flink.cdc.connectors.starrocks.sink.StarRocksMetadataApplierITCase Error: org.apache.flink.cdc.connectors.starrocks.sink.StarRocksMetadataApplierITCase Time elapsed: 122.202 s <<< ERROR! java.util.concurrent.CompletionException: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image starrocks/allin1-ubuntu:3.2.6 ``` [data_stream_migration_test](https://github.com/apache/flink-cdc/actions/runs/10195953793/job/28335732338?pr=3490#logs) and [pipeline_migration_test](https://github.com/apache/flink-cdc/actions/runs/10195953793/job/28335732490?pr=3490#logs) ``` line 1: docker-compose: command not found ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35976) StreamPhysicalOverAggregate should handle column name confliction
lincoln lee created FLINK-35976:
---
Summary: StreamPhysicalOverAggregate should handle column name
confliction
Key: FLINK-35976
URL: https://issues.apache.org/jira/browse/FLINK-35976
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.19.1, 1.20.0
Reporter: lincoln lee
Assignee: lincoln lee
Fix For: 2.0.0
Duplicate column name exception occurred when use a nested over aggregate query,
e.g., a repro case:
{code}
@Test
def testNestedOverAgg(): Unit = {
util.addTable(s"""
|CREATE TEMPORARY TABLE src (
| a STRING,
| b STRING,
| ts TIMESTAMP_LTZ(3),
| watermark FOR ts as ts
|) WITH (
| 'connector' = 'values'
|)
|""".stripMargin)
util.verifyExecPlan(s"""
|SELECT *
|FROM (
| SELECT
| *, count(*) OVER (PARTITION BY a ORDER BY ts) AS c2
| FROM (
| SELECT
| *, count(*) OVER (PARTITION BY a,b ORDER BY ts) AS c1
| FROM src
| )
|)
|""".stripMargin)
}
{code}
{code}
org.apache.flink.table.api.ValidationException: Field names must be unique.
Found duplicates: [w0$o0]
at org.apache.flink.table.types.logical.RowType.validateFields(RowType.java:273)
at org.apache.flink.table.types.logical.RowType.(RowType.java:158)
at org.apache.flink.table.types.logical.RowType.of(RowType.java:298)
at org.apache.flink.table.types.logical.RowType.of(RowType.java:290)
at
org.apache.flink.table.planner.calcite.FlinkTypeFactory$.toLogicalRowType(FlinkTypeFactory.scala:678)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregate.translateToExecNode(StreamPhysicalOverAggregate.scala:57)
at
org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode(FlinkPhysicalRel.scala:53)
at
org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode$(FlinkPhysicalRel.scala:52)
at
org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregateBase.translateToExecNode(StreamPhysicalOverAggregateBase.scala:35)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:74)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:54)
at
org.apache.flink.table.planner.delegation.PlannerBase.translateToExecNodeGraph(PlannerBase.scala:407)
at
org.apache.flink.table.planner.utils.TableTestUtilBase.assertPlanEquals(TableTestBase.scala:1076)
at
org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:920)
at
org.apache.flink.table.planner.utils.TableTestUtilBase.verifyExecPlan(TableTestBase.scala:675)
at
org.apache.flink.table.planner.plan.stream.sql.agg.OverAggregateTest.testNestedOverAgg(OverAggregateTest.scala:460)
{code}
This is a similar case In https://issues.apache.org/jira/browse/FLINK-22121,
but missed the fixing in streaming over agg scenario.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35976) StreamPhysicalOverAggregate should handle column name confliction
[
https://issues.apache.org/jira/browse/FLINK-35976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35976:
---
Labels: pull-request-available (was: )
> StreamPhysicalOverAggregate should handle column name confliction
> -
>
> Key: FLINK-35976
> URL: https://issues.apache.org/jira/browse/FLINK-35976
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.20.0, 1.19.1
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> Duplicate column name exception occurred when use a nested over aggregate
> query,
> e.g., a repro case:
> {code}
> @Test
> def testNestedOverAgg(): Unit = {
> util.addTable(s"""
> |CREATE TEMPORARY TABLE src (
> | a STRING,
> | b STRING,
> | ts TIMESTAMP_LTZ(3),
> | watermark FOR ts as ts
> |) WITH (
> | 'connector' = 'values'
> |)
> |""".stripMargin)
> util.verifyExecPlan(s"""
> |SELECT *
> |FROM (
> | SELECT
> | *, count(*) OVER (PARTITION BY a ORDER BY ts) AS c2
> | FROM (
> | SELECT
> | *, count(*) OVER (PARTITION BY a,b ORDER BY ts) AS c1
> | FROM src
> | )
> |)
> |""".stripMargin)
> }
> {code}
>
> {code}
> org.apache.flink.table.api.ValidationException: Field names must be unique.
> Found duplicates: [w0$o0]
>
> at
> org.apache.flink.table.types.logical.RowType.validateFields(RowType.java:273)
> at org.apache.flink.table.types.logical.RowType.(RowType.java:158)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:298)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:290)
> at
> org.apache.flink.table.planner.calcite.FlinkTypeFactory$.toLogicalRowType(FlinkTypeFactory.scala:678)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregate.translateToExecNode(StreamPhysicalOverAggregate.scala:57)
> at
> org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode(FlinkPhysicalRel.scala:53)
> at
> org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode$(FlinkPhysicalRel.scala:52)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregateBase.translateToExecNode(StreamPhysicalOverAggregateBase.scala:35)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:74)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:54)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translateToExecNodeGraph(PlannerBase.scala:407)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.assertPlanEquals(TableTestBase.scala:1076)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:920)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.verifyExecPlan(TableTestBase.scala:675)
> at
> org.apache.flink.table.planner.plan.stream.sql.agg.OverAggregateTest.testNestedOverAgg(OverAggregateTest.scala:460)
> {code}
>
> This is a similar case In https://issues.apache.org/jira/browse/FLINK-22121,
> but missed the fixing in streaming over agg scenario.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35974) PyFlink YARN per-job on Docker test failed because docker-compose command not found
[
https://issues.apache.org/jira/browse/FLINK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-35974:
---
Priority: Critical (was: Major)
> PyFlink YARN per-job on Docker test failed because docker-compose command not
> found
> ---
>
> Key: FLINK-35974
> URL: https://issues.apache.org/jira/browse/FLINK-35974
> Project: Flink
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Weijie Guo
>Priority: Critical
>
> {code:java}
> Aug 02 09:20:55
> ==
> Aug 02 09:20:55 Running 'PyFlink YARN per-job on Docker test'
> Aug 02 09:20:55
> ==
> Aug 02 09:20:55 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-55842252037
> Aug 02 09:20:56 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-2.0-SNAPSHOT-bin/flink-2.0-SNAPSHOT
> Aug 02 09:20:57 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-2.0-SNAPSHOT-bin/flink-2.0-SNAPSHOT
> Aug 02 09:20:57 Docker version 26.1.3, build b72abbb
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/common_docker.sh:
> line 24: docker-compose: command not found
> Aug 02 09:20:57 [FAIL] Test script contains errors.
> Aug 02 09:20:57 Checking of logs skipped.
> Aug 02 09:20:57
> Aug 02 09:20:57 [FAIL] 'PyFlink YARN per-job on Docker test' failed after 0
> minutes and 1 seconds! Test exited with exit code 1
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=61174&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=0f3adb59-eefa-51c6-2858-3654d9e0749d&l=9544
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35974) PyFlink YARN per-job on Docker test failed because docker-compose command not found
[
https://issues.apache.org/jira/browse/FLINK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870963#comment-17870963
]
Weijie Guo commented on FLINK-35974:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=61206&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=0f3adb59-eefa-51c6-2858-3654d9e0749d&l=9811
> PyFlink YARN per-job on Docker test failed because docker-compose command not
> found
> ---
>
> Key: FLINK-35974
> URL: https://issues.apache.org/jira/browse/FLINK-35974
> Project: Flink
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Weijie Guo
>Priority: Major
>
> {code:java}
> Aug 02 09:20:55
> ==
> Aug 02 09:20:55 Running 'PyFlink YARN per-job on Docker test'
> Aug 02 09:20:55
> ==
> Aug 02 09:20:55 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-55842252037
> Aug 02 09:20:56 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-2.0-SNAPSHOT-bin/flink-2.0-SNAPSHOT
> Aug 02 09:20:57 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-2.0-SNAPSHOT-bin/flink-2.0-SNAPSHOT
> Aug 02 09:20:57 Docker version 26.1.3, build b72abbb
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/common_docker.sh:
> line 24: docker-compose: command not found
> Aug 02 09:20:57 [FAIL] Test script contains errors.
> Aug 02 09:20:57 Checking of logs skipped.
> Aug 02 09:20:57
> Aug 02 09:20:57 [FAIL] 'PyFlink YARN per-job on Docker test' failed after 0
> minutes and 1 seconds! Test exited with exit code 1
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=61174&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=0f3adb59-eefa-51c6-2858-3654d9e0749d&l=9544
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-35976][table-planner] Fix column name conflicts in StreamPhysicalOverAggregate [flink]
lincoln-lil opened a new pull request, #25152: URL: https://github.com/apache/flink/pull/25152 ## What is the purpose of the change [FLINK-22121](https://issues.apache.org/jira/browse/FLINK-22121) fixed a similar column name confliction both for rank and batch over agg scenarios, but lost the `StreamPhysicalOverAggregate`, this pr aims to fix it. ## Brief change log Consistent output rowtype of `StreamPhysicalOverAggregate` with `BatchPhysicalOverAggregate` Reuse related utility methods ## Verifying this change Both stream and batch sql's `OverAggregateTest` added new test cases ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with @Public(Evolving): (no) - The serializers: (no ) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35974) PyFlink YARN per-job on Docker test failed because docker-compose command not found
[
https://issues.apache.org/jira/browse/FLINK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-35974:
---
Priority: Blocker (was: Critical)
> PyFlink YARN per-job on Docker test failed because docker-compose command not
> found
> ---
>
> Key: FLINK-35974
> URL: https://issues.apache.org/jira/browse/FLINK-35974
> Project: Flink
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Weijie Guo
>Priority: Blocker
>
> {code:java}
> Aug 02 09:20:55
> ==
> Aug 02 09:20:55 Running 'PyFlink YARN per-job on Docker test'
> Aug 02 09:20:55
> ==
> Aug 02 09:20:55 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-55842252037
> Aug 02 09:20:56 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-2.0-SNAPSHOT-bin/flink-2.0-SNAPSHOT
> Aug 02 09:20:57 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-2.0-SNAPSHOT-bin/flink-2.0-SNAPSHOT
> Aug 02 09:20:57 Docker version 26.1.3, build b72abbb
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/common_docker.sh:
> line 24: docker-compose: command not found
> Aug 02 09:20:57 [FAIL] Test script contains errors.
> Aug 02 09:20:57 Checking of logs skipped.
> Aug 02 09:20:57
> Aug 02 09:20:57 [FAIL] 'PyFlink YARN per-job on Docker test' failed after 0
> minutes and 1 seconds! Test exited with exit code 1
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=61174&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=0f3adb59-eefa-51c6-2858-3654d9e0749d&l=9544
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35976][table-planner] Fix column name conflicts in StreamPhysicalOverAggregate [flink]
flinkbot commented on PR #25152: URL: https://github.com/apache/flink/pull/25152#issuecomment-2268372484 ## CI report: * a9cddd19b65eae41b6dcacff03c02bfa6340 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35976) StreamPhysicalOverAggregate should handle column name confliction
[
https://issues.apache.org/jira/browse/FLINK-35976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870971#comment-17870971
]
Feng Jin commented on FLINK-35976:
--
A similar issue.
[https://issues.apache.org/jira/projects/FLINK/issues/FLINK-27519|https://issues.apache.org/jira/projects/FLINK/issues/FLINK-35976]
> StreamPhysicalOverAggregate should handle column name confliction
> -
>
> Key: FLINK-35976
> URL: https://issues.apache.org/jira/browse/FLINK-35976
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.20.0, 1.19.1
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> Duplicate column name exception occurred when use a nested over aggregate
> query,
> e.g., a repro case:
> {code}
> @Test
> def testNestedOverAgg(): Unit = {
> util.addTable(s"""
> |CREATE TEMPORARY TABLE src (
> | a STRING,
> | b STRING,
> | ts TIMESTAMP_LTZ(3),
> | watermark FOR ts as ts
> |) WITH (
> | 'connector' = 'values'
> |)
> |""".stripMargin)
> util.verifyExecPlan(s"""
> |SELECT *
> |FROM (
> | SELECT
> | *, count(*) OVER (PARTITION BY a ORDER BY ts) AS c2
> | FROM (
> | SELECT
> | *, count(*) OVER (PARTITION BY a,b ORDER BY ts) AS c1
> | FROM src
> | )
> |)
> |""".stripMargin)
> }
> {code}
>
> {code}
> org.apache.flink.table.api.ValidationException: Field names must be unique.
> Found duplicates: [w0$o0]
>
> at
> org.apache.flink.table.types.logical.RowType.validateFields(RowType.java:273)
> at org.apache.flink.table.types.logical.RowType.(RowType.java:158)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:298)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:290)
> at
> org.apache.flink.table.planner.calcite.FlinkTypeFactory$.toLogicalRowType(FlinkTypeFactory.scala:678)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregate.translateToExecNode(StreamPhysicalOverAggregate.scala:57)
> at
> org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode(FlinkPhysicalRel.scala:53)
> at
> org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode$(FlinkPhysicalRel.scala:52)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregateBase.translateToExecNode(StreamPhysicalOverAggregateBase.scala:35)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:74)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:54)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translateToExecNodeGraph(PlannerBase.scala:407)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.assertPlanEquals(TableTestBase.scala:1076)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:920)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.verifyExecPlan(TableTestBase.scala:675)
> at
> org.apache.flink.table.planner.plan.stream.sql.agg.OverAggregateTest.testNestedOverAgg(OverAggregateTest.scala:460)
> {code}
>
> This is a similar case In https://issues.apache.org/jira/browse/FLINK-22121,
> but missed the fixing in streaming over agg scenario.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35976) StreamPhysicalOverAggregate should handle column name confliction
[
https://issues.apache.org/jira/browse/FLINK-35976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870971#comment-17870971
]
Feng Jin edited comment on FLINK-35976 at 8/5/24 7:46 AM:
--
A similar issue.
https://issues.apache.org/jira/browse/FLINK-27519
was (Author: hackergin):
A similar issue.
[https://issues.apache.org/jira/projects/FLINK/issues/FLINK-27519|https://issues.apache.org/jira/projects/FLINK/issues/FLINK-35976]
> StreamPhysicalOverAggregate should handle column name confliction
> -
>
> Key: FLINK-35976
> URL: https://issues.apache.org/jira/browse/FLINK-35976
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.20.0, 1.19.1
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> Duplicate column name exception occurred when use a nested over aggregate
> query,
> e.g., a repro case:
> {code}
> @Test
> def testNestedOverAgg(): Unit = {
> util.addTable(s"""
> |CREATE TEMPORARY TABLE src (
> | a STRING,
> | b STRING,
> | ts TIMESTAMP_LTZ(3),
> | watermark FOR ts as ts
> |) WITH (
> | 'connector' = 'values'
> |)
> |""".stripMargin)
> util.verifyExecPlan(s"""
> |SELECT *
> |FROM (
> | SELECT
> | *, count(*) OVER (PARTITION BY a ORDER BY ts) AS c2
> | FROM (
> | SELECT
> | *, count(*) OVER (PARTITION BY a,b ORDER BY ts) AS c1
> | FROM src
> | )
> |)
> |""".stripMargin)
> }
> {code}
>
> {code}
> org.apache.flink.table.api.ValidationException: Field names must be unique.
> Found duplicates: [w0$o0]
>
> at
> org.apache.flink.table.types.logical.RowType.validateFields(RowType.java:273)
> at org.apache.flink.table.types.logical.RowType.(RowType.java:158)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:298)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:290)
> at
> org.apache.flink.table.planner.calcite.FlinkTypeFactory$.toLogicalRowType(FlinkTypeFactory.scala:678)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregate.translateToExecNode(StreamPhysicalOverAggregate.scala:57)
> at
> org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode(FlinkPhysicalRel.scala:53)
> at
> org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode$(FlinkPhysicalRel.scala:52)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregateBase.translateToExecNode(StreamPhysicalOverAggregateBase.scala:35)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:74)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:54)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translateToExecNodeGraph(PlannerBase.scala:407)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.assertPlanEquals(TableTestBase.scala:1076)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:920)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.verifyExecPlan(TableTestBase.scala:675)
> at
> org.apache.flink.table.planner.plan.stream.sql.agg.OverAggregateTest.testNestedOverAgg(OverAggregateTest.scala:460)
> {code}
>
> This is a similar case In https://issues.apache.org/jira/browse/FLINK-22121,
> but missed the fixing in streaming over agg scenario.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35143][cdc-connector][mysql] Expose newly added tables capture in mysql pipeline connector. [flink-cdc]
qg-lin commented on PR #3411: URL: https://github.com/apache/flink-cdc/pull/3411#issuecomment-2268411831 > @qg-lin Please rebase the master branch and let's pass the CI. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31980] Implement support for EFO in Kinesis consumer [flink-connector-aws]
hlteoh37 commented on PR #76: URL: https://github.com/apache/flink-connector-aws/pull/76#issuecomment-2268475739 @Sunninsky Hey! We are currently working on implementing EFO support for the new source connector. Estimate maybe end of the quarter? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31980] Implement support for EFO in Kinesis consumer [flink-connector-aws]
Sunninsky commented on PR #76: URL: https://github.com/apache/flink-connector-aws/pull/76#issuecomment-2268478660 > @Sunninsky Hey! We are currently working on implementing EFO support for the new source connector. Estimate maybe end of the quarter? That's very cool, thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34215] FLIP-377: Support fine-grained configuration to control filter push down for JDBC Connector [flink-connector-jdbc]
eskabetxe commented on PR #95: URL: https://github.com/apache/flink-connector-jdbc/pull/95#issuecomment-2268502868 @1996fanrui can you check please.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35976) StreamPhysicalOverAggregate should handle column name confliction
[
https://issues.apache.org/jira/browse/FLINK-35976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870989#comment-17870989
]
lincoln lee commented on FLINK-35976:
-
Thanks [~hackergin] for your tips! Let's move the disscussion there.
> StreamPhysicalOverAggregate should handle column name confliction
> -
>
> Key: FLINK-35976
> URL: https://issues.apache.org/jira/browse/FLINK-35976
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.20.0, 1.19.1
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> Duplicate column name exception occurred when use a nested over aggregate
> query,
> e.g., a repro case:
> {code}
> @Test
> def testNestedOverAgg(): Unit = {
> util.addTable(s"""
> |CREATE TEMPORARY TABLE src (
> | a STRING,
> | b STRING,
> | ts TIMESTAMP_LTZ(3),
> | watermark FOR ts as ts
> |) WITH (
> | 'connector' = 'values'
> |)
> |""".stripMargin)
> util.verifyExecPlan(s"""
> |SELECT *
> |FROM (
> | SELECT
> | *, count(*) OVER (PARTITION BY a ORDER BY ts) AS c2
> | FROM (
> | SELECT
> | *, count(*) OVER (PARTITION BY a,b ORDER BY ts) AS c1
> | FROM src
> | )
> |)
> |""".stripMargin)
> }
> {code}
>
> {code}
> org.apache.flink.table.api.ValidationException: Field names must be unique.
> Found duplicates: [w0$o0]
>
> at
> org.apache.flink.table.types.logical.RowType.validateFields(RowType.java:273)
> at org.apache.flink.table.types.logical.RowType.(RowType.java:158)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:298)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:290)
> at
> org.apache.flink.table.planner.calcite.FlinkTypeFactory$.toLogicalRowType(FlinkTypeFactory.scala:678)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregate.translateToExecNode(StreamPhysicalOverAggregate.scala:57)
> at
> org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode(FlinkPhysicalRel.scala:53)
> at
> org.apache.flink.table.planner.plan.nodes.physical.FlinkPhysicalRel.translateToExecNode$(FlinkPhysicalRel.scala:52)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregateBase.translateToExecNode(StreamPhysicalOverAggregateBase.scala:35)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:74)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:54)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translateToExecNodeGraph(PlannerBase.scala:407)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.assertPlanEquals(TableTestBase.scala:1076)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:920)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.verifyExecPlan(TableTestBase.scala:675)
> at
> org.apache.flink.table.planner.plan.stream.sql.agg.OverAggregateTest.testNestedOverAgg(OverAggregateTest.scala:460)
> {code}
>
> This is a similar case In https://issues.apache.org/jira/browse/FLINK-22121,
> but missed the fixing in streaming over agg scenario.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [hotfix] Fix StarRocks FE startup failure due to insufficient disk space available [flink-cdc]
banmoy commented on PR #3508: URL: https://github.com/apache/flink-cdc/pull/3508#issuecomment-2268532722 @yuxiqian Thanks for the effort. Do you have the FE log with the exception? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] Fix StarRocks FE startup failure due to insufficient disk space available [flink-cdc]
yuxiqian commented on PR #3508:
URL: https://github.com/apache/flink-cdc/pull/3508#issuecomment-2268537879
FE log
```
2024-08-05T07:10:43.9361779Z 968 [testcontainers-lifecycle-0] INFO
org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded
org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from
~/.testcontainers.properties, will try it first
2024-08-05T07:10:43.9365864Z 1211 [testcontainers-lifecycle-0] INFO
org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker
environment with local Unix socket (unix:///var/run/docker.sock)
2024-08-05T07:10:43.9368448Z 1212 [testcontainers-lifecycle-0] INFO
org.testcontainers.DockerClientFactory - Docker host IP address is localhost
2024-08-05T07:10:43.9379835Z 2024-08-05 07:08:47.735Z ERROR (main|1)
[MetaHelper.checkMetaDir():173] Free capacity left for meta dir:
/data/deploy/starrocks/fe/meta is less than 5GB
2024-08-05T07:10:43.9381764Z 2024-08-05 07:08:47.741Z ERROR (main|1)
[StarRocksFE.start():193] StarRocksFE start failed
2024-08-05T07:10:43.9382840Z com.starrocks.common.InvalidMetaDirException:
null
2024-08-05T07:10:43.9384161Z at
com.starrocks.leader.MetaHelper.checkMetaDir(MetaHelper.java:175)
~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9385881Z at
com.starrocks.StarRocksFE.start(StarRocksFE.java:128) ~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9387283Z at
com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9389128Z 2024-08-05 07:08:50.671Z ERROR (main|1)
[MetaHelper.checkMetaDir():173] Free capacity left for meta dir:
/data/deploy/starrocks/fe/meta is less than 5GB
2024-08-05T07:10:43.9390824Z 2024-08-05 07:08:50.676Z ERROR (main|1)
[StarRocksFE.start():193] StarRocksFE start failed
2024-08-05T07:10:43.9391800Z com.starrocks.common.InvalidMetaDirException:
null
2024-08-05T07:10:43.9392942Z at
com.starrocks.leader.MetaHelper.checkMetaDir(MetaHelper.java:175)
~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9394353Z at
com.starrocks.StarRocksFE.start(StarRocksFE.java:128) ~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9395698Z at
com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9397604Z 2024-08-05 07:08:54.774Z ERROR (main|1)
[MetaHelper.checkMetaDir():173] Free capacity left for meta dir:
/data/deploy/starrocks/fe/meta is less than 5GB
2024-08-05T07:10:43.9399446Z 2024-08-05 07:08:54.780Z ERROR (main|1)
[StarRocksFE.start():193] StarRocksFE start failed
2024-08-05T07:10:43.9400527Z com.starrocks.common.InvalidMetaDirException:
null
2024-08-05T07:10:43.9402057Z at
com.starrocks.leader.MetaHelper.checkMetaDir(MetaHelper.java:175)
~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9403602Z at
com.starrocks.StarRocksFE.start(StarRocksFE.java:128) ~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9405008Z at
com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9406949Z 2024-08-05 07:08:59.705Z ERROR (main|1)
[MetaHelper.checkMetaDir():173] Free capacity left for meta dir:
/data/deploy/starrocks/fe/meta is less than 5GB
2024-08-05T07:10:43.9408874Z 2024-08-05 07:08:59.711Z ERROR (main|1)
[StarRocksFE.start():193] StarRocksFE start failed
2024-08-05T07:10:43.9409993Z com.starrocks.common.InvalidMetaDirException:
null
2024-08-05T07:10:43.9411318Z at
com.starrocks.leader.MetaHelper.checkMetaDir(MetaHelper.java:175)
~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9412865Z at
com.starrocks.StarRocksFE.start(StarRocksFE.java:128) ~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9414249Z at
com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
2024-08-05T07:10:43.9414961Z
```
StdOut log
2024-08-05T07:10:43.9607673Z 160852 [testcontainers-lifecycle-0] ERROR
tc.starrocks/allin1-ubuntu:3.2.6 - Log output from the failed container:
2024-08-05T07:10:43.9608467Z
2024-08-05T07:10:43.9609174Z * _
___ __
2024-08-05T07:10:43.9610288Z * |\ \|\___ ___\\ __ \|\ __ \|\
__ \|\ __ \|\ \|\ \|\ \ |\ \
2024-08-05T07:10:43.9611339Z * \ \ \___|\|___ \ \_\ \ \|\ \ \ \|\ \ \
\|\ \ \ \|\ \ \ \___|\ \ \/ /|\ \ \___|_
2024-08-05T07:10:43.9612008Z * \ \_ \ \ \ \ \ \ __ \ \ _ _\
\ _ _\ \ \\\ \ \ \\ \ ___ \ \_ \
2024-08-05T07:10:43.9612646Z *\||\ \ \ \ \ \ \ \ \ \ \ \\ \\
\ \\ \\ \ \\\ \ \ \\ \ \\ \ \||\ \
2024-08-05T07:10:43.9613280Z * \_\ \ \ \__\ \ \__\ \__\ \__\\
_\\ \__\\ _\\ \___\ \___\ \__\\ \__\\_\ \
2024-08-05T07
[jira] [Commented] (FLINK-27519) Fix duplicates names when there are multiple levels of over window aggregate
[
https://issues.apache.org/jira/browse/FLINK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870999#comment-17870999
]
lincoln lee commented on FLINK-27519:
-
Thanks [~hackergin] for the tips and the discussion here! Looks like this issue
has been on hold for a while.
I recently encountered this issue in an internal version and had a quick fix
after some investigating (since I didn't search for the jira directly, but it
looks like everyone is basically on the same page). So [~hackergin] [~paul8263]
could some of you help review the pr
[#25152|https://github.com/apache/flink/pull/25152]?
> Fix duplicates names when there are multiple levels of over window aggregate
>
>
> Key: FLINK-27519
> URL: https://issues.apache.org/jira/browse/FLINK-27519
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.0
>Reporter: Feng Jin
>Priority: Major
>
> A similar issue like
> [FLINK-22121|https://issues.apache.org/jira/browse/FLINK-22121]
> And can be reproduced by adding this unit test
> org.apache.flink.table.planner.plan.stream.sql.agg.GroupWindowTest#testWindowAggregateWithAnotherWindowAggregate
> {code:java}
> //代码占位符
> @Test
> def testWindowAggregateWithAnotherWindowAggregate(): Unit = {
> val sql =
> """
> |SELECT CAST(pv AS INT) AS pv, CAST(uv AS INT) AS uv FROM (
> | SELECT *, count(distinct(c)) over (partition by a order by b desc)
> AS uv
> | FROM (
> |SELECT *, count(*) over (partition by a, c order by b desc) AS pv
> |FROM MyTable
> | )
> |)
> |""".stripMargin
> util.verifyExecPlan(sql)
> } {code}
> The error message :
>
>
> {code:java}
> //代码占位符
> org.apache.flink.table.api.ValidationException: Field names must be unique.
> Found duplicates: [w0$o0] at
> org.apache.flink.table.types.logical.RowType.validateFields(RowType.java:273)
> at org.apache.flink.table.types.logical.RowType.(RowType.java:158)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:298)
> at org.apache.flink.table.types.logical.RowType.of(RowType.java:290)
> at
> org.apache.flink.table.planner.calcite.FlinkTypeFactory$.toLogicalRowType(FlinkTypeFactory.scala:663)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalOverAggregate.translateToExecNode(StreamPhysicalOverAggregate.scala:57)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:74)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeGraphGenerator.generate(ExecNodeGraphGenerator.java:71)
> {code}
>
> I think we can add come logical in FlinkLogicalOverAggregate to avoid
> duplicate names of output rowType.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35858][State] Default namespace in async state [flink]
masteryhx commented on code in PR #25135:
URL: https://github.com/apache/flink/pull/25135#discussion_r1703812477
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStValueState.java:
##
@@ -87,8 +93,10 @@ public byte[] serializeKey(ContextKey contextKey)
throws IOException {
ctxKey -> {
SerializedCompositeKeyBuilder builder =
serializedKeyBuilder.get();
builder.setKeyAndKeyGroup(ctxKey.getRawKey(),
ctxKey.getKeyGroup());
+N namespace = contextKey.getNamespace(this);
Review Comment:
Coud we just mark ContextKey#getNamespace is Nonnull, and it will return
VoidNamespace if null ?
Then we could avoid introducing a new parameter in the
`AsyncKeyedStateBackend`.
Or Will we support other default namespace ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [mysql] Fix the restoration fail when checkpoint happened in a mysql transaction [flink-cdc]
kendn1993 commented on PR #421: URL: https://github.com/apache/flink-cdc/pull/421#issuecomment-2268596720 flink1.17.1+cdc 2.4.1 还是会出现这个问题 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33875][runtime] Support slots wait mechanism at DeclarativeSlotPoolBridge side for Default Scheduler [flink]
1996fanrui commented on code in PR #25134:
URL: https://github.com/apache/flink/pull/25134#discussion_r1703679953
##
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DeclarativeSlotPoolBridgeResourceDeclarationTest.java:
##
@@ -205,12 +224,36 @@ void testRequirementsDecreasedOnSlotAllocationFailure()
throws Exception {
.isZero();
}
-private static final class RequirementListener {
+/** Requirement listener for testing. */
+public static final class RequirementListener {
Review Comment:
```suggestion
static final class RequirementListener {
```
Default is enough.
##
flink-runtime/src/test/java/org/apache/flink/runtime/instance/SimplePhysicalSlot.java:
##
@@ -21,12 +21,12 @@
import org.apache.flink.runtime.clusterframework.types.AllocationID;
import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
import org.apache.flink.runtime.jobmanager.slots.TaskManagerGateway;
-import org.apache.flink.runtime.jobmaster.SlotContext;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
import org.apache.flink.util.Preconditions;
-/** Simple implementation of the {@link SlotContext} interface for the legacy
code. */
-public class SimpleSlotContext implements SlotContext {
+/** Simple implementation of the {@link PhysicalSlot} interface for the legacy
code. */
+public class SimplePhysicalSlot implements PhysicalSlot {
Review Comment:
After this PR, `SimplePhysicalSlot` and `TestingPhysicalSlot` are similar
(TestingPhysicalSlot has Payload payload related logic than
`SimplePhysicalSlot`.)
I'm curious is it possible to remove `SimplePhysicalSlot`, and all callers
use `TestingPhysicalSlot`?
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DeclarativeSlotPoolBridge.java:
##
@@ -548,4 +610,13 @@ void increaseResourceRequirementsBy(ResourceCounter
increment) {
boolean isBatchSlotRequestTimeoutCheckEnabled() {
return !isBatchSlotRequestTimeoutCheckDisabled;
}
+
+@VisibleForTesting
+public void tryWaitSlotRequestIsDone() {
Review Comment:
```suggestion
void tryWaitSlotRequestIsDone() {
```
public isn't needed.
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DeclarativeSlotPoolBridge.java:
##
@@ -548,4 +610,13 @@ void increaseResourceRequirementsBy(ResourceCounter
increment) {
boolean isBatchSlotRequestTimeoutCheckEnabled() {
return !isBatchSlotRequestTimeoutCheckDisabled;
}
+
+@VisibleForTesting
+public void tryWaitSlotRequestIsDone() {
+if (getDeclarativeSlotPool() instanceof DefaultDeclarativeSlotPool) {
+final DefaultDeclarativeSlotPool slotPool =
+(DefaultDeclarativeSlotPool) getDeclarativeSlotPool();
+slotPool.tryWaitSlotRequestIsDone();
+}
+}
Review Comment:
If last comment makes sense, could we move ` void
tryWaitSlotRequestIsDone()` method to `DeclarativeSlotPoolBridgeTest`?
Only `DeclarativeSlotPoolBridgeTest` calls `tryWaitSlotRequestIsDone()`.
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DeclarativeSlotPoolBridge.java:
##
@@ -234,6 +290,11 @@ void newSlotsAreAvailable(Collection newSlots) {
}
}
+@VisibleForTesting
+Collection getFreePhysicalSlots() {
Review Comment:
```suggestion
Collection getFreeSlotsInformation() {
```
nit: how about keeping it's same with interface?
##
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/TestingFreeSlotTracker.java:
##
@@ -151,23 +151,23 @@ public Builder
setReserveSlotConsumer(Consumer reserveSlotConsumer
return this;
}
-public Builder
setCreateNewFreeSlotInfoTrackerWithoutBlockedSlotsFunction(
-Function, FreeSlotInfoTracker>
-
createNewFreeSlotInfoTrackerWithoutBlockedSlotsFunction) {
-this.createNewFreeSlotInfoTrackerWithoutBlockedSlotsFunction =
-createNewFreeSlotInfoTrackerWithoutBlockedSlotsFunction;
+public Builder setCreateNewFreeSlotTrackerWithoutBlockedSlotsFunction(
Review Comment:
It seems this method is not called.
##
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DeclarativeSlotPoolBridgeTest.java:
##
@@ -55,19 +58,32 @@ void testSlotOffer() throws Exception {
final PhysicalSlot allocatedSlot =
createAllocatedSlot(expectedAllocationId);
final TestingDeclarativeSlotPoolFactory declarativeSlotPoolFactory =
-new
TestingDeclarativeSlotPoolFactory(TestingDeclarativeSlotPool.builder());
+new TestingDeclarativeSlotPoolFactory(
Review Comment:
As I understand,
Re: [PR] [FLINK-35815][Connector/Kinesis] Fix detection of recoverable exceptions for EFO operations [flink-connector-aws]
hlteoh37 merged PR #153: URL: https://github.com/apache/flink-connector-aws/pull/153 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-35815) KinesisProxySyncV2 doesn't always retry throttling exceptions.
[
https://issues.apache.org/jira/browse/FLINK-35815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh resolved FLINK-35815.
-
Resolution: Fixed
> KinesisProxySyncV2 doesn't always retry throttling exceptions.
> ---
>
> Key: FLINK-35815
> URL: https://issues.apache.org/jira/browse/FLINK-35815
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0, aws-connector-4.3.0, 1.19.1
>Reporter: Krzysztof Dziolak
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
> Fix For: aws-connector-4.4.0
>
>
> *Problem:*
> We have observed missing retrys on throttling for DescribeStreamSummary calls
> from Kinesis.
> {code:java}
> org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute
> application.
> ...
> Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkRuntimeException: Could not execute application.
> ...
> Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute
> application.
> ...
> Caused by: org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error: Error registering stream:
> ...
> Caused by:
> org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil$FlinkKinesisStreamConsumerRegistrarException:
> Error registering stream:
> ...
> Caused by:
> org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.model.LimitExceededException:
> Rate exceeded for stream . (Service: Kinesis, Status Code: 400,
> Request ID: efe4c2a9-3c3b-9c1c-b0ed-d9b05db93be2, Extended Request ID:
> pSG6kwQXgPWD2S7YoPT4RKf+g8QbRBaxc0grhNz6juEoti/uGUQTzyqsfmFCLSHoM+u1ydHvqxzsv/0ICUid6aTAQdndy2EO)
> ...
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.lambda$describeStreamSummary$0(KinesisProxySyncV2.java:91)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.invokeWithRetryAndBackoff(KinesisProxySyncV2.java:175)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.describeStreamSummary(KinesisProxySyncV2.java:90)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.StreamConsumerRegistrar.registerStreamConsumer(StreamConsumerRegistrar.java:92)
> at
> org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil.registerStreamConsumers(StreamConsumerRegistrarUtil.java:121)
> ... {code}
> The same problem occurs both with LAZY and EAGER registration strategies.
>
> *Why does it get stuck?*
> *[https://github.com/apache/flink-connector-aws/blob/c716ca439b2c8e6d4b5905a03c867c418e031688/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/AwsV2Util.java#L77]*
> The `isRecoverableException` check validates the cause of the exception only,
> but it doesn't inspect the actual exception being evaluated for retriability.
> In this particular case, LimitExceededException is thrown without wrappers
> and it appears that this case is not handled correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35815) KinesisProxySyncV2 doesn't always retry throttling exceptions.
[
https://issues.apache.org/jira/browse/FLINK-35815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871015#comment-17871015
]
Hong Liang Teoh commented on FLINK-35815:
-
merged commit
[{{6cfa13d}}|https://github.com/apache/flink-connector-aws/commit/6cfa13d64e090887552ad2a3d4f6a59b8d599b28]
into apache:main
> KinesisProxySyncV2 doesn't always retry throttling exceptions.
> ---
>
> Key: FLINK-35815
> URL: https://issues.apache.org/jira/browse/FLINK-35815
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0, aws-connector-4.3.0, 1.19.1
>Reporter: Krzysztof Dziolak
>Priority: Major
> Labels: pull-request-available
> Fix For: aws-connector-4.4.0
>
>
> *Problem:*
> We have observed missing retrys on throttling for DescribeStreamSummary calls
> from Kinesis.
> {code:java}
> org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute
> application.
> ...
> Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkRuntimeException: Could not execute application.
> ...
> Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute
> application.
> ...
> Caused by: org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error: Error registering stream:
> ...
> Caused by:
> org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil$FlinkKinesisStreamConsumerRegistrarException:
> Error registering stream:
> ...
> Caused by:
> org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.model.LimitExceededException:
> Rate exceeded for stream . (Service: Kinesis, Status Code: 400,
> Request ID: efe4c2a9-3c3b-9c1c-b0ed-d9b05db93be2, Extended Request ID:
> pSG6kwQXgPWD2S7YoPT4RKf+g8QbRBaxc0grhNz6juEoti/uGUQTzyqsfmFCLSHoM+u1ydHvqxzsv/0ICUid6aTAQdndy2EO)
> ...
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.lambda$describeStreamSummary$0(KinesisProxySyncV2.java:91)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.invokeWithRetryAndBackoff(KinesisProxySyncV2.java:175)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.describeStreamSummary(KinesisProxySyncV2.java:90)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.StreamConsumerRegistrar.registerStreamConsumer(StreamConsumerRegistrar.java:92)
> at
> org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil.registerStreamConsumers(StreamConsumerRegistrarUtil.java:121)
> ... {code}
> The same problem occurs both with LAZY and EAGER registration strategies.
>
> *Why does it get stuck?*
> *[https://github.com/apache/flink-connector-aws/blob/c716ca439b2c8e6d4b5905a03c867c418e031688/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/AwsV2Util.java#L77]*
> The `isRecoverableException` check validates the cause of the exception only,
> but it doesn't inspect the actual exception being evaluated for retriability.
> In this particular case, LimitExceededException is thrown without wrappers
> and it appears that this case is not handled correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35815) KinesisProxySyncV2 doesn't always retry throttling exceptions.
[
https://issues.apache.org/jira/browse/FLINK-35815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh reassigned FLINK-35815:
---
Assignee: Aleksandr Pilipenko
> KinesisProxySyncV2 doesn't always retry throttling exceptions.
> ---
>
> Key: FLINK-35815
> URL: https://issues.apache.org/jira/browse/FLINK-35815
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0, aws-connector-4.3.0, 1.19.1
>Reporter: Krzysztof Dziolak
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
> Fix For: aws-connector-4.4.0
>
>
> *Problem:*
> We have observed missing retrys on throttling for DescribeStreamSummary calls
> from Kinesis.
> {code:java}
> org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute
> application.
> ...
> Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkRuntimeException: Could not execute application.
> ...
> Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute
> application.
> ...
> Caused by: org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error: Error registering stream:
> ...
> Caused by:
> org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil$FlinkKinesisStreamConsumerRegistrarException:
> Error registering stream:
> ...
> Caused by:
> org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.model.LimitExceededException:
> Rate exceeded for stream . (Service: Kinesis, Status Code: 400,
> Request ID: efe4c2a9-3c3b-9c1c-b0ed-d9b05db93be2, Extended Request ID:
> pSG6kwQXgPWD2S7YoPT4RKf+g8QbRBaxc0grhNz6juEoti/uGUQTzyqsfmFCLSHoM+u1ydHvqxzsv/0ICUid6aTAQdndy2EO)
> ...
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.lambda$describeStreamSummary$0(KinesisProxySyncV2.java:91)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.invokeWithRetryAndBackoff(KinesisProxySyncV2.java:175)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxySyncV2.describeStreamSummary(KinesisProxySyncV2.java:90)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.StreamConsumerRegistrar.registerStreamConsumer(StreamConsumerRegistrar.java:92)
> at
> org.apache.flink.streaming.connectors.kinesis.util.StreamConsumerRegistrarUtil.registerStreamConsumers(StreamConsumerRegistrarUtil.java:121)
> ... {code}
> The same problem occurs both with LAZY and EAGER registration strategies.
>
> *Why does it get stuck?*
> *[https://github.com/apache/flink-connector-aws/blob/c716ca439b2c8e6d4b5905a03c867c418e031688/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/AwsV2Util.java#L77]*
> The `isRecoverableException` check validates the cause of the exception only,
> but it doesn't inspect the actual exception being evaluated for retriability.
> In this particular case, LimitExceededException is thrown without wrappers
> and it appears that this case is not handled correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-35955) Remove outdated versions
[
https://issues.apache.org/jira/browse/FLINK-35955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo resolved FLINK-35955.
Resolution: Resolved
> Remove outdated versions
>
>
> Key: FLINK-35955
> URL: https://issues.apache.org/jira/browse/FLINK-35955
> Project: Flink
> Issue Type: Sub-task
>Reporter: Weijie Guo
>Assignee: Weijie Guo
>Priority: Major
>
> h4. dist.apache.org
> For a new major release remove all release files older than 2 versions, e.g.,
> when releasing 1.7, remove all releases <= 1.5.
> For a new bugfix version remove all release files for previous bugfix
> releases in the same series, e.g., when releasing 1.7.1, remove the 1.7.0
> release.
> # If you have not already, check out the Flink section of the {{release}}
> repository on {{[dist.apache.org|http://dist.apache.org/]}} via Subversion.
> In a fresh directory:
> {code}
> svn checkout https://dist.apache.org/repos/dist/release/flink
> --depth=immediates
> cd flink
> {code}
> # Remove files for outdated releases and commit the changes.
> {code}
> svn remove flink-
> svn commit
> {code}
> # Verify that files are
> [removed|https://dist.apache.org/repos/dist/release/flink]
> (!) Remember to remove the corresponding download links from the website.
> h4. CI
> Disable the cron job for the now-unsupported version from
> (tools/azure-pipelines/[build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
> in the respective branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35910][table] Add the built-in function BTRIM [flink]
dylanhz commented on PR #25127: URL: https://github.com/apache/flink/pull/25127#issuecomment-2268840820 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35964) Add STARTSWITH function
[
https://issues.apache.org/jira/browse/FLINK-35964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dylan He updated FLINK-35964:
-
Description:
Add STARTSWITH function.
Returns if {{expr}} begins with {{startExpr}}.
Example:
{code:sql}
> SELECT STARTSWITH('SparkSQL', 'Spark');
true
> SELECT STARTSWITH('SparkSQL', 'spark');
false
{code}
Syntax:
{code:sql}
STARTSWITH(expr, startExpr)
{code}
Arguments:
* {{expr}}: A STRING or BINARY expression.
* {{startExpr}}: A STRING or BINARY expression.
Returns:
A BOOLEAN.
{{expr}} and {{startExpr}} should have same type.
If {{expr}} or {{startExpr}} is NULL, the result is NULL.
If {{startExpr}} is empty, the result is true.
See also:
*
[Spark|https://spark.apache.org/docs/3.5.1/sql-ref-functions-builtin.html#string-functions]
*
[Databricks|https://docs.databricks.com/en/sql/language-manual/functions/startswith.html]
was:
Add STARTSWITH function.
Returns if {{expr}} begins with {{startExpr}}.
Example:
{code:sql}
> SELECT STARTSWITH('SparkSQL', 'Spark');
true
> SELECT STARTSWITH('SparkSQL', 'spark');
false
{code}
Syntax:
{code:sql}
STARTSWITH(expr, startExpr)
{code}
Arguments:
* {{expr}}: A STRING or BINARY expression.
* {{startExpr}}: A STRING or BINARY expression.
Returns:
A BOOLEAN.
{{expr}} and {{startExpr}} should have same type.
If {{expr}} or {{startExpr}} is NULL, the result is NULL.
If {{startExpr}} is the empty, the result is true.
See also:
*
[Spark|https://spark.apache.org/docs/3.5.1/sql-ref-functions-builtin.html#string-functions]
*
[Databricks|https://docs.databricks.com/en/sql/language-manual/functions/startswith.html]
> Add STARTSWITH function
> ---
>
> Key: FLINK-35964
> URL: https://issues.apache.org/jira/browse/FLINK-35964
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Dylan He
>Priority: Major
>
> Add STARTSWITH function.
>
> Returns if {{expr}} begins with {{startExpr}}.
> Example:
> {code:sql}
> > SELECT STARTSWITH('SparkSQL', 'Spark');
> true
> > SELECT STARTSWITH('SparkSQL', 'spark');
> false
> {code}
> Syntax:
> {code:sql}
> STARTSWITH(expr, startExpr)
> {code}
> Arguments:
> * {{expr}}: A STRING or BINARY expression.
> * {{startExpr}}: A STRING or BINARY expression.
> Returns:
> A BOOLEAN.
> {{expr}} and {{startExpr}} should have same type.
> If {{expr}} or {{startExpr}} is NULL, the result is NULL.
> If {{startExpr}} is empty, the result is true.
> See also:
> *
> [Spark|https://spark.apache.org/docs/3.5.1/sql-ref-functions-builtin.html#string-functions]
> *
> [Databricks|https://docs.databricks.com/en/sql/language-manual/functions/startswith.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35977) Missing an import in datastream.md
guluo created FLINK-35977: - Summary: Missing an import in datastream.md Key: FLINK-35977 URL: https://issues.apache.org/jira/browse/FLINK-35977 Project: Flink Issue Type: Bug Affects Versions: 1.20.0 Reporter: guluo In document datastream.md, we missing an import about OpenContext. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35977][doc] Missing an import in datastream.md [flink]
guluo2016 opened a new pull request, #25153: URL: https://github.com/apache/flink/pull/25153 ## What is the purpose of the change This PR fix [FLINK-35977](https://issues.apache.org/jira/browse/FLINK-35977) that misssing an import about OpenContext. ## Brief change log - Adding an import about `OpenContext` in doc datastream.md - Removing an import about `Configuration` in doc datastream.md ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? ( no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35977) Missing an import in datastream.md
[ https://issues.apache.org/jira/browse/FLINK-35977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35977: --- Labels: pull-request-available (was: ) > Missing an import in datastream.md > -- > > Key: FLINK-35977 > URL: https://issues.apache.org/jira/browse/FLINK-35977 > Project: Flink > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: guluo >Priority: Minor > Labels: pull-request-available > > In document datastream.md, we missing an import about OpenContext. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35977][doc] Missing an import in datastream.md [flink]
flinkbot commented on PR #25153: URL: https://github.com/apache/flink/pull/25153#issuecomment-2268927425 ## CI report: * e2421af596c1e888c6416a815fcee48a1d9682bf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33761][Connector/JDBC] Add Snowflake JDBC Dialect [flink-connector-jdbc]
borislitvak commented on PR #118: URL: https://github.com/apache/flink-connector-jdbc/pull/118#issuecomment-2268938459 @davidradl WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
leonardBang commented on PR #3449: URL: https://github.com/apache/flink-cdc/pull/3449#issuecomment-2269086514 @aiwenmo @yuxiqian I append a commit to make sure our temporal function semantics align with Flink SQL, please help review my commit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35978) adapt labelling of boring cyborg
João Boto created FLINK-35978: - Summary: adapt labelling of boring cyborg Key: FLINK-35978 URL: https://issues.apache.org/jira/browse/FLINK-35978 Project: Flink Issue Type: Sub-task Reporter: João Boto -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35978] Adapt labelling of boring cyborg to new modules [flink-connector-jdbc]
eskabetxe opened a new pull request, #135: URL: https://github.com/apache/flink-connector-jdbc/pull/135 Adapt the labelling to new structure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35978) adapt labelling of boring cyborg
[ https://issues.apache.org/jira/browse/FLINK-35978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35978: --- Labels: pull-request-available (was: ) > adapt labelling of boring cyborg > > > Key: FLINK-35978 > URL: https://issues.apache.org/jira/browse/FLINK-35978 > Project: Flink > Issue Type: Sub-task >Reporter: João Boto >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
aiwenmo commented on PR #3449: URL: https://github.com/apache/flink-cdc/pull/3449#issuecomment-2269264056 thx. I'm reviewing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35979) Add flink version 2.0-SNAPSHOT to ci
João Boto created FLINK-35979: - Summary: Add flink version 2.0-SNAPSHOT to ci Key: FLINK-35979 URL: https://issues.apache.org/jira/browse/FLINK-35979 Project: Flink Issue Type: Sub-task Reporter: João Boto -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
aiwenmo commented on code in PR #3449:
URL: https://github.com/apache/flink-cdc/pull/3449#discussion_r170478
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##
@@ -359,6 +407,15 @@ private static Java.Rvalue generateTypeConvertMethod(
case "VARCHAR":
case "STRING":
return new Java.MethodInvocation(Location.NOWHERE, null,
"castToString", atoms);
+case "TIMESTAMP":
+List timestampFunctionParam = new
ArrayList<>(Arrays.asList(atoms));
Review Comment:
Do we need to add tests for `cast( ... as timestamp)` ?
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/functions/SystemFunctionUtils.java:
##
@@ -38,65 +41,62 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import static
org.apache.flink.cdc.common.utils.DateTimeUtils.timestampMillisToDate;
+import static
org.apache.flink.cdc.common.utils.DateTimeUtils.timestampMillisToTime;
+
/** System function utils to support the call of flink cdc pipeline transform.
*/
public class SystemFunctionUtils {
-private static final Logger LOG =
LoggerFactory.getLogger(SystemFunctionUtils.class);
-public static int localtime(long epochTime, String timezone) {
-return DateTimeUtils.timestampMillisToTime(epochTime);
-}
+private static final Logger LOG =
LoggerFactory.getLogger(SystemFunctionUtils.class);
-public static TimestampData localtimestamp(long epochTime, String
timezone) {
-return TimestampData.fromMillis(epochTime);
+public static LocalZonedTimestampData currentTimestamp(long epochTime) {
+return LocalZonedTimestampData.fromEpochMillis(epochTime);
}
-// synonym: localtime
-public static int currentTime(long epochTime, String timezone) {
-return localtime(epochTime, timezone);
+// synonym with currentTimestamp
+public static LocalZonedTimestampData now(long epochTime) {
+return LocalZonedTimestampData.fromEpochMillis(epochTime);
}
-public static int currentDate(long epochTime, String timezone) {
-return DateTimeUtils.timestampMillisToDate(epochTime);
+public static TimestampData localtimestamp(long epochTime, String
timezone) {
+return TimestampData.fromLocalDateTime(
+
Instant.ofEpochMilli(epochTime).atZone(ZoneId.of(timezone)).toLocalDateTime());
}
-public static TimestampData currentTimestamp(long epochTime, String
timezone) {
-return TimestampData.fromMillis(
-epochTime +
TimeZone.getTimeZone(timezone).getOffset(epochTime));
+public static int localtime(long epochTime, String timezone) {
+return timestampMillisToTime(localtimestamp(epochTime,
timezone).getMillisecond());
}
-public static LocalZonedTimestampData now(long epochTime, String timezone)
{
-return LocalZonedTimestampData.fromEpochMillis(epochTime);
+public static int currentTime(long epochTime, String timezone) {
+// the time value of currentTimestamp under given session time zone
+return timestampMillisToTime(localtimestamp(epochTime,
timezone).getMillisecond());
}
-public static String dateFormat(LocalZonedTimestampData timestamp, String
format) {
-SimpleDateFormat dateFormat = new SimpleDateFormat(format);
-return dateFormat.format(new Date(timestamp.getEpochMillisecond()));
+public static int currentDate(long epochTime, String timezone) {
+// the date value of currentTimestamp under given session time zone
+return timestampMillisToDate(localtimestamp(epochTime,
timezone).getMillisecond());
}
public static String dateFormat(TimestampData timestamp, String format) {
-SimpleDateFormat dateFormat = new SimpleDateFormat(format);
-return dateFormat.format(new Date(timestamp.getMillisecond()));
-}
-
-public static String dateFormat(ZonedTimestampData timestamp, String
format) {
-SimpleDateFormat dateFormat = new SimpleDateFormat(format);
-return dateFormat.format(new Date(timestamp.getMillisecond()));
+return DateTimeUtils.formatTimestampMillis(
+timestamp.getMillisecond(), format,
TimeZone.getTimeZone("UTC"));
Review Comment:
Does it not require timezone to participate in calculations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [hotfix][ci] Update ci to include 1.20.0 [flink-connector-aws]
z3d1k opened a new pull request, #154: URL: https://github.com/apache/flink-connector-aws/pull/154 ## Purpose of the change Update ci build configuration after 1.20.0 release ## Verifying this change This change is already covered by Github approval workflow. ## Significant changes *(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)* - [ ] Dependencies have been added or upgraded - [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`) - [ ] Serializers have been changed - [ ] New feature has been introduced - If yes, how is this documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix][ci] Update ci to include 1.20.0 [flink-connector-aws]
z3d1k commented on PR #154: URL: https://github.com/apache/flink-connector-aws/pull/154#issuecomment-2269514160 cc @hlteoh37 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34216][connectors/mongodb] FLIP-377: Support fine-grained configuration to control filter push down for MongoDB Connector [flink-connector-mongodb]
eskabetxe commented on PR #23: URL: https://github.com/apache/flink-connector-mongodb/pull/23#issuecomment-2269535265 hi @Jiabao-Sun I'm not an contributor to this connector but it LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35976][table-planner] Fix column name conflicts in StreamPhysicalOverAggregate [flink]
hackergin commented on code in PR #25152:
URL: https://github.com/apache/flink/pull/25152#discussion_r1704417002
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/OverAggregateUtil.scala:
##
@@ -219,4 +224,20 @@ object OverAggregateUtil {
}
}
}
+
+ def inferOutputRowType(
+ cluster: RelOptCluster,
+ inputType: RelDataType,
+ aggCalls: Seq[AggregateCall]): RelDataType = {
+
+val inputNameList = inputType.getFieldNames
+val inputTypeList = inputType.getFieldList.asScala.map(field =>
field.getType)
Review Comment:
nit: Maintain consistency with the code style below
```
val inputTypeList = inputType.getFieldList.asScala.map(_.getType)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [FLINK-35934] Add CompiledPlan annotations to BatchExecValues (#1) [flink]
jnh5y opened a new pull request, #25154: URL: https://github.com/apache/flink/pull/25154 ## What is the purpose of the change * In addition to the annotations, implement a BatchRestoreTest for this operator. * Moves the ValuesTestPrograms to a common package for re-use between streaming and batch. ## Verifying this change This change adds a BatchRestoreTest to cover the new annotations and show that the batch compiled plan can be restored and executed correctly. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (yes) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35934] Add CompiledPlan annotations to BatchExecValues (#1) [flink]
flinkbot commented on PR #25154: URL: https://github.com/apache/flink/pull/25154#issuecomment-2269574388 ## CI report: * eea2abcf55eb4e4c5e7b910d5e92449fd9888aba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [hotfix] [docs] Fix typo in KeyedStream [flink]
mattayes opened a new pull request, #25155: URL: https://github.com/apache/flink/pull/25155 ## What is the purpose of the change Fix typo in `KeyedStream.keySelector` and `KeyedStream.getKeySelector()` docs. ## Brief change log * Fix typo in `KeyedStream.keySelector` and `KeyedStream.getKeySelector()` docs. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] [docs] Fix typo in KeyedStream [flink]
flinkbot commented on PR #25155: URL: https://github.com/apache/flink/pull/25155#issuecomment-2269836853 ## CI report: * 116c31495887330b7c00ee3679e92696171926f8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35463]Fixed issue for route rule changed when restored from checkpoint. [flink-cdc]
github-actions[bot] commented on PR #3364: URL: https://github.com/apache/flink-cdc/pull/3364#issuecomment-2270123855 This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
yuxiqian commented on code in PR #3449:
URL: https://github.com/apache/flink-cdc/pull/3449#discussion_r1704771418
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##
@@ -300,7 +315,19 @@ private static Java.Rvalue generateOtherFunctionOperation(
}
}
-private static Java.Rvalue
generateNoOperandTimestampFunctionOperation(String operationName) {
+private static Java.Rvalue
generateTimezoneFreeTemporalFunctionOperation(String operationName) {
+List timestampFunctionParam = new ArrayList<>();
+timestampFunctionParam.add(
+new Java.AmbiguousName(Location.NOWHERE, new String[]
{DEFAULT_EPOCH_TIME}));
+return new Java.MethodInvocation(
+Location.NOWHERE,
+null,
+StringUtils.convertToCamelCase(operationName),
+timestampFunctionParam.toArray(new Java.Rvalue[0]));
Review Comment:
I wonder if is it necessary to initialize `timestampFunctionParam` here,
since it seems did the same thing:
```java
new Java.Rvalue[] {
new Java.AmbiguousName(Location.NOWHERE, new String[] {
DEFAULT_EPOCH_TIME
})
}
```
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##
@@ -313,6 +340,27 @@ private static Java.Rvalue
generateNoOperandTimestampFunctionOperation(String op
timestampFunctionParam.toArray(new Java.Rvalue[0]));
}
+private static Java.Rvalue
generateTimezoneFreeTemporalConversionFunctionOperation(
+String operationName) {
+return new Java.MethodInvocation(
+Location.NOWHERE,
+null,
+StringUtils.convertToCamelCase(operationName),
+new ArrayList<>().toArray(new Java.Rvalue[0]));
Review Comment:
```suggestion
new Java.Rvalue[0]);
```
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##
@@ -359,6 +407,15 @@ private static Java.Rvalue generateTypeConvertMethod(
case "VARCHAR":
case "STRING":
return new Java.MethodInvocation(Location.NOWHERE, null,
"castToString", atoms);
+case "TIMESTAMP":
+List timestampFunctionParam = new
ArrayList<>(Arrays.asList(atoms));
+timestampFunctionParam.add(
+new Java.AmbiguousName(Location.NOWHERE, new String[]
{DEFAULT_TIME_ZONE}));
+return new Java.MethodInvocation(
+Location.NOWHERE,
+null,
+"castToTimestamp",
+timestampFunctionParam.toArray(new Java.Rvalue[0]));
Review Comment:
I remember that there's a `ArrayUtils#add(T[], T)` method in Apache commons
that doesn't require initializing mutable lists and keeps code clearer. Is it
possible to use it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-35980) Add transform test coverage in Integrated / E2e tests
yux created FLINK-35980: --- Summary: Add transform test coverage in Integrated / E2e tests Key: FLINK-35980 URL: https://issues.apache.org/jira/browse/FLINK-35980 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: yux Currently, there are no enough UT/E2e test cases to cover transform features like built-in scalar functions, temporary functions, complex expressions, etc. Adding them should cover more use-cases and reduce the possibility of accidentally introduced mistakes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35980) Add transform test coverage in Integrated / E2e tests
[ https://issues.apache.org/jira/browse/FLINK-35980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun reassigned FLINK-35980: -- Assignee: yux > Add transform test coverage in Integrated / E2e tests > - > > Key: FLINK-35980 > URL: https://issues.apache.org/jira/browse/FLINK-35980 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Reporter: yux >Assignee: yux >Priority: Major > > Currently, there are no enough UT/E2e test cases to cover transform features > like built-in scalar functions, temporary functions, complex expressions, > etc. Adding them should cover more use-cases and reduce the possibility of > accidentally introduced mistakes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35978] Adapt labelling of boring cyborg to new modules [flink-connector-jdbc]
1996fanrui merged PR #135: URL: https://github.com/apache/flink-connector-jdbc/pull/135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35978) adapt labelling of boring cyborg
[ https://issues.apache.org/jira/browse/FLINK-35978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871196#comment-17871196 ] Rui Fan commented on FLINK-35978: - Merged to main(3.3.0) via: cc5e292ad1e1ef4bf0185dbf8185346b6b95d895 > adapt labelling of boring cyborg > > > Key: FLINK-35978 > URL: https://issues.apache.org/jira/browse/FLINK-35978 > Project: Flink > Issue Type: Sub-task >Reporter: João Boto >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-35978) adapt labelling of boring cyborg
[ https://issues.apache.org/jira/browse/FLINK-35978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan resolved FLINK-35978. - Fix Version/s: jdbc-3.3.0 Assignee: João Boto Resolution: Fixed > adapt labelling of boring cyborg > > > Key: FLINK-35978 > URL: https://issues.apache.org/jira/browse/FLINK-35978 > Project: Flink > Issue Type: Sub-task >Reporter: João Boto >Assignee: João Boto >Priority: Major > Labels: pull-request-available > Fix For: jdbc-3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34215] FLIP-377: Support fine-grained configuration to control filter push down for JDBC Connector [flink-connector-jdbc]
1996fanrui merged PR #95: URL: https://github.com/apache/flink-connector-jdbc/pull/95 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-34215) Support fine-grained configuration to control filter push down for JDBC Connector
[ https://issues.apache.org/jira/browse/FLINK-34215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan resolved FLINK-34215. - Fix Version/s: jdbc-3.3.0 (was: jdbc-3.1.3) Assignee: Jiabao Sun Resolution: Fixed Merged to main(3.3.0) via : b598054c9cb5cc0174993058a53990e5d7d1827b > Support fine-grained configuration to control filter push down for JDBC > Connector > - > > Key: FLINK-34215 > URL: https://issues.apache.org/jira/browse/FLINK-34215 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Affects Versions: jdbc-3.1.2 >Reporter: jiabao.sun >Assignee: Jiabao Sun >Priority: Major > Labels: pull-request-available > Fix For: jdbc-3.3.0 > > > Support fine-grained configuration to control filter push down for JDBC > Connector. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34214) FLIP-377: Support fine-grained configuration to control filter push down for Table/SQL Sources
[ https://issues.apache.org/jira/browse/FLINK-34214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-34214: Fix Version/s: jdbc-3.3.0 (was: jdbc-3.1.3) > FLIP-377: Support fine-grained configuration to control filter push down for > Table/SQL Sources > -- > > Key: FLINK-34214 > URL: https://issues.apache.org/jira/browse/FLINK-34214 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC, Connectors / MongoDB >Affects Versions: mongodb-1.0.2, jdbc-3.1.2 >Reporter: jiabao.sun >Assignee: jiabao.sun >Priority: Major > Fix For: mongodb-1.3.0, jdbc-3.3.0 > > > This improvement implements [FLIP-377 Support fine-grained configuration to > control filter push down for Table/SQL > Sources|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768] > This FLIP has 2 goals: > * Introduces a new configuration filter.handling.policy to the JDBC and > MongoDB connector. > * Suggests a convention option name if other connectors are going to add an > option for the same purpose. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34216][connectors/mongodb] FLIP-377: Support fine-grained configuration to control filter push down for MongoDB Connector [flink-connector-mongodb]
Jiabao-Sun commented on PR #23: URL: https://github.com/apache/flink-connector-mongodb/pull/23#issuecomment-2270202389 Thanks @eskabetxe and @RocMarshal for the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34216][connectors/mongodb] FLIP-377: Support fine-grained configuration to control filter push down for MongoDB Connector [flink-connector-mongodb]
Jiabao-Sun merged PR #23: URL: https://github.com/apache/flink-connector-mongodb/pull/23 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35910][table] Add the built-in function BTRIM [flink]
dylanhz commented on PR #25127: URL: https://github.com/apache/flink/pull/25127#issuecomment-2270203780 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-34216) Support fine-grained configuration to control filter push down for MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-34216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun resolved FLINK-34216. Assignee: Jiabao Sun Resolution: Implemented merged into main: 424d57a54c69d9d0e0fde4219e46bf5fcf457b59 > Support fine-grained configuration to control filter push down for MongoDB > Connector > > > Key: FLINK-34216 > URL: https://issues.apache.org/jira/browse/FLINK-34216 > Project: Flink > Issue Type: Sub-task > Components: Connectors / MongoDB >Affects Versions: mongodb-1.0.2 >Reporter: jiabao.sun >Assignee: Jiabao Sun >Priority: Major > Labels: pull-request-available > Fix For: mongodb-1.3.0 > > > Support fine-grained configuration to control filter push down for MongoDB > Connector. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-34214) FLIP-377: Support fine-grained configuration to control filter push down for Table/SQL Sources
[ https://issues.apache.org/jira/browse/FLINK-34214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun resolved FLINK-34214. Assignee: Jiabao Sun (was: jiabao.sun) Resolution: Implemented jdbc main(3.3.0) via : b598054c9cb5cc0174993058a53990e5d7d1827b mongodb main(1.3.0) via : 424d57a54c69d9d0e0fde4219e46bf5fcf457b59 > FLIP-377: Support fine-grained configuration to control filter push down for > Table/SQL Sources > -- > > Key: FLINK-34214 > URL: https://issues.apache.org/jira/browse/FLINK-34214 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC, Connectors / MongoDB >Affects Versions: mongodb-1.0.2, jdbc-3.1.2 >Reporter: jiabao.sun >Assignee: Jiabao Sun >Priority: Major > Fix For: mongodb-1.3.0, jdbc-3.3.0 > > > This improvement implements [FLIP-377 Support fine-grained configuration to > control filter push down for Table/SQL > Sources|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768] > This FLIP has 2 goals: > * Introduces a new configuration filter.handling.policy to the JDBC and > MongoDB connector. > * Suggests a convention option name if other connectors are going to add an > option for the same purpose. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] FLINK-35924: delay the SplitReader closure to until all the emitted records are processed. [flink]
becketqin commented on PR #25130: URL: https://github.com/apache/flink/pull/25130#issuecomment-2270206455 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35981) Transform rule doesn't support referencing one column more than once
yux created FLINK-35981: --- Summary: Transform rule doesn't support referencing one column more than once Key: FLINK-35981 URL: https://issues.apache.org/jira/browse/FLINK-35981 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: yux Currently, transform rule (projection / filtering) doesn't support referencing one column more than once, which means if we write such a projection rule: transform: - projection: \*, age * age AS age_square filter: age < 18 OR age > 60 Janino compiler will crash with the following exception stack. Seems duplicated columns were added into Janino arguments list: Caused by: org.apache.flink.shaded.guava31.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.flink.api.common.InvalidProgramException: Expression cannot be compiled. This is a bug. Please file an issue. Expression: import static org.apache.flink.cdc.runtime.functions.SystemFunctionUtils.*;age * age at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2055) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache.get(LocalCache.java:3966) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863) at org.apache.flink.cdc.runtime.operators.transform.TransformExpressionCompiler.compileExpression(TransformExpressionCompiler.java:46) ... 18 more Caused by: org.apache.flink.api.common.InvalidProgramException: Expression cannot be compiled. This is a bug. Please file an issue. Expression: import static org.apache.flink.cdc.runtime.functions.SystemFunctionUtils.*;age * age at org.apache.flink.cdc.runtime.operators.transform.TransformExpressionCompiler.lambda$compileExpression$0(TransformExpressionCompiler.java:62) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159) at org.apache.flink.shaded.guava31.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) ... 21 more Caused by: org.codehaus.commons.compiler.CompileException: Line 1, Column 82: Redefinition of parameter "age" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35964][table] Add the built-in function STARTSWITH [flink]
dylanhz opened a new pull request, #25156:
URL: https://github.com/apache/flink/pull/25156
## What is the purpose of the change
Add the built-in function STARTSWITH.
Examples:
```SQL
> SELECT STARTSWITH('SparkSQL', 'Spark');
true
> SELECT STARTSWITH('SparkSQL', 'spark');
false
```
## Brief change log
[FLINK-35964](https://issues.apache.org/jira/browse/FLINK-35964)
## Verifying this change
`StringFunctionsITCase#startswithTestCases`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35964) Add STARTSWITH function
[
https://issues.apache.org/jira/browse/FLINK-35964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35964:
---
Labels: pull-request-available (was: )
> Add STARTSWITH function
> ---
>
> Key: FLINK-35964
> URL: https://issues.apache.org/jira/browse/FLINK-35964
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Dylan He
>Priority: Major
> Labels: pull-request-available
>
> Add STARTSWITH function.
>
> Returns if {{expr}} begins with {{startExpr}}.
> Example:
> {code:sql}
> > SELECT STARTSWITH('SparkSQL', 'Spark');
> true
> > SELECT STARTSWITH('SparkSQL', 'spark');
> false
> {code}
> Syntax:
> {code:sql}
> STARTSWITH(expr, startExpr)
> {code}
> Arguments:
> * {{expr}}: A STRING or BINARY expression.
> * {{startExpr}}: A STRING or BINARY expression.
> Returns:
> A BOOLEAN.
> {{expr}} and {{startExpr}} should have same type.
> If {{expr}} or {{startExpr}} is NULL, the result is NULL.
> If {{startExpr}} is empty, the result is true.
> See also:
> *
> [Spark|https://spark.apache.org/docs/3.5.1/sql-ref-functions-builtin.html#string-functions]
> *
> [Databricks|https://docs.databricks.com/en/sql/language-manual/functions/startswith.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35925] remove hive connector from main repo [flink]
JingsongLi commented on PR #25129: URL: https://github.com/apache/flink/pull/25129#issuecomment-2270263413 CC @luoyuxia -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35964][table] Add the built-in function STARTSWITH [flink]
flinkbot commented on PR #25156: URL: https://github.com/apache/flink/pull/25156#issuecomment-2270266374 ## CI report: * 7c21f6231604599aa8e17bde284791ccef23625e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
leonardBang commented on code in PR #3449: URL: https://github.com/apache/flink-cdc/pull/3449#discussion_r1704843120 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java: ## @@ -359,6 +407,15 @@ private static Java.Rvalue generateTypeConvertMethod( case "VARCHAR": case "STRING": return new Java.MethodInvocation(Location.NOWHERE, null, "castToString", atoms); +case "TIMESTAMP": +List timestampFunctionParam = new ArrayList<>(Arrays.asList(atoms)); Review Comment: +1, I’ll add test to cover this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
leonardBang commented on code in PR #3449:
URL: https://github.com/apache/flink-cdc/pull/3449#discussion_r1704843782
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/functions/SystemFunctionUtils.java:
##
@@ -38,65 +41,62 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import static
org.apache.flink.cdc.common.utils.DateTimeUtils.timestampMillisToDate;
+import static
org.apache.flink.cdc.common.utils.DateTimeUtils.timestampMillisToTime;
+
/** System function utils to support the call of flink cdc pipeline transform.
*/
public class SystemFunctionUtils {
-private static final Logger LOG =
LoggerFactory.getLogger(SystemFunctionUtils.class);
-public static int localtime(long epochTime, String timezone) {
-return DateTimeUtils.timestampMillisToTime(epochTime);
-}
+private static final Logger LOG =
LoggerFactory.getLogger(SystemFunctionUtils.class);
-public static TimestampData localtimestamp(long epochTime, String
timezone) {
-return TimestampData.fromMillis(epochTime);
+public static LocalZonedTimestampData currentTimestamp(long epochTime) {
+return LocalZonedTimestampData.fromEpochMillis(epochTime);
}
-// synonym: localtime
-public static int currentTime(long epochTime, String timezone) {
-return localtime(epochTime, timezone);
+// synonym with currentTimestamp
+public static LocalZonedTimestampData now(long epochTime) {
+return LocalZonedTimestampData.fromEpochMillis(epochTime);
}
-public static int currentDate(long epochTime, String timezone) {
-return DateTimeUtils.timestampMillisToDate(epochTime);
+public static TimestampData localtimestamp(long epochTime, String
timezone) {
+return TimestampData.fromLocalDateTime(
+
Instant.ofEpochMilli(epochTime).atZone(ZoneId.of(timezone)).toLocalDateTime());
}
-public static TimestampData currentTimestamp(long epochTime, String
timezone) {
-return TimestampData.fromMillis(
-epochTime +
TimeZone.getTimeZone(timezone).getOffset(epochTime));
+public static int localtime(long epochTime, String timezone) {
+return timestampMillisToTime(localtimestamp(epochTime,
timezone).getMillisecond());
}
-public static LocalZonedTimestampData now(long epochTime, String timezone)
{
-return LocalZonedTimestampData.fromEpochMillis(epochTime);
+public static int currentTime(long epochTime, String timezone) {
+// the time value of currentTimestamp under given session time zone
+return timestampMillisToTime(localtimestamp(epochTime,
timezone).getMillisecond());
}
-public static String dateFormat(LocalZonedTimestampData timestamp, String
format) {
-SimpleDateFormat dateFormat = new SimpleDateFormat(format);
-return dateFormat.format(new Date(timestamp.getEpochMillisecond()));
+public static int currentDate(long epochTime, String timezone) {
+// the date value of currentTimestamp under given session time zone
+return timestampMillisToDate(localtimestamp(epochTime,
timezone).getMillisecond());
}
public static String dateFormat(TimestampData timestamp, String format) {
-SimpleDateFormat dateFormat = new SimpleDateFormat(format);
-return dateFormat.format(new Date(timestamp.getMillisecond()));
-}
-
-public static String dateFormat(ZonedTimestampData timestamp, String
format) {
-SimpleDateFormat dateFormat = new SimpleDateFormat(format);
-return dateFormat.format(new Date(timestamp.getMillisecond()));
+return DateTimeUtils.formatTimestampMillis(
+timestamp.getMillisecond(), format,
TimeZone.getTimeZone("UTC"));
Review Comment:
yes, TimestampData is timezone unrelated data structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-35982) Transform metadata config doesn't work if no projection block was provided
yux created FLINK-35982: --- Summary: Transform metadata config doesn't work if no projection block was provided Key: FLINK-35982 URL: https://issues.apache.org/jira/browse/FLINK-35982 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: yux One may readjust source table primary keys, partition keys, table options by specifying them in a Transform block like this: transform: - projection: '*' primary-keys: order_id, product_name partition-keys: order_id table-options: bucket=1 However, if projection field is omitted (which fallbacks to default behavior that does not change any source table columns), such configuration will not take effect: transform: - primary-keys: order_id, product_name partition-keys: order_id table-options: bucket=1 # These options will not apply -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35983) Job crashes when using wildcard (*) match with metadata column
yux created FLINK-35983: --- Summary: Job crashes when using wildcard (*) match with metadata column Key: FLINK-35983 URL: https://issues.apache.org/jira/browse/FLINK-35983 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: yux One may write such a projection rule: transform: - projection: '*, __namespace_name__, __schema_name__, __table_name__' to append some metadata columns at the end of existing columns. However this will crash the job since once a metadata column was declared, since wildcard matches metadata column, too: Caused by: java.lang.IllegalArgumentException: Field names must be unique. Found duplicates: [__namespace_name__, __schema_name__, __table_name__] at org.apache.flink.cdc.common.types.RowType.validateFields(RowType.java:158) at org.apache.flink.cdc.common.types.RowType.(RowType.java:54) at org.apache.flink.cdc.common.types.RowType.of(RowType.java:183) at org.apache.flink.cdc.common.types.RowType.of(RowType.java:175) at org.apache.flink.cdc.runtime.typeutils.DataTypeConverter.toRowType(DataTypeConverter.java:55) at org.apache.flink.cdc.runtime.operators.transform.TableChangeInfo.of(TableChangeInfo.java:100) at org.apache.flink.cdc.runtime.operators.transform.TransformSchemaOperator.cacheCreateTable(TransformSchemaOperator.java:183) at org.apache.flink.cdc.runtime.operators.transform.TransformSchemaOperator.processElement(TransformSchemaOperator.java:168) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:309) at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110) at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:101) at org.apache.flink.api.connector.source.lib.util.IteratorSourceReaderBase.pollNext(IteratorSourceReaderBase.java:111) at org.apache.flink.cdc.connectors.values.source.ValuesDataSource$EventIteratorReader.pollNext(ValuesDataSource.java:294) at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419) at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35984) Job crashes when metadata column names present in transform rules
yux created FLINK-35984: --- Summary: Job crashes when metadata column names present in transform rules Key: FLINK-35984 URL: https://issues.apache.org/jira/browse/FLINK-35984 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: yux Given such a transform rule: transform: projection: \*, '__namespace_name__schema_name__table_name__' AS string_literal Obviously this shouldn't insert any metadata columns. However since metadata column existence check was done by searching identifier string in statement, without considering any syntax info: ```java // TransformParser.java#L357 if (transformStatement.contains(DEFAULT_NAMESPACE_NAME) ... ``` Transform operator will mistakenly append metadata columns into Janino arguments list, and crash the job: Caused by: java.lang.IllegalArgumentException: wrong number of arguments at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.janino.ExpressionEvaluator.evaluate(ExpressionEvaluator.java:541) at org.codehaus.janino.ExpressionEvaluator.evaluate(ExpressionEvaluator.java:533) at org.apache.flink.cdc.runtime.operators.transform.ProjectionColumnProcessor.evaluate(ProjectionColumnProcessor.java:64) at org.apache.flink.cdc.runtime.operators.transform.TransformProjectionProcessor.processData(TransformProjectionProcessor.java:153) at org.apache.flink.cdc.runtime.operators.transform.TransformDataOperator.processProjection(TransformDataOperator.java:387) at org.apache.flink.cdc.runtime.operators.transform.TransformDataOperator.processDataChangeEvent(TransformDataOperator.java:328) at org.apache.flink.cdc.runtime.operators.transform.TransformDataOperator.processElement(TransformDataOperator.java:190) at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:237) at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146) at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
aiwenmo commented on PR #3449: URL: https://github.com/apache/flink-cdc/pull/3449#issuecomment-2270316674 @yuxiqian Hi. I'm busy with other things. Could you add an e2e test for the temporal function? For example: Mysql -> Transform -> Doris/Values `projection: *,LOCALTIME as time1,CURRENT_TIME as time2,CURRENT_TIMESTAMP as timestamp1,NOW() as timestamp2,LOCALTIMESTAMP as timestamp3,CURRENT_DATE as date1` I guess the data type generated by the temporal function modified by this PR may not match the Flink type in practical use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
yuxiqian commented on PR #3449: URL: https://github.com/apache/flink-cdc/pull/3449#issuecomment-2270319084 Sure, thanks for @aiwenmo's work so far, I'll create a patch for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-35965][table] Add the built-in function ENDSWITH [flink]
dylanhz opened a new pull request, #25157:
URL: https://github.com/apache/flink/pull/25157
## What is the purpose of the change
Add the built-in function ENDSWITH.
Examples:
```SQL
> SELECT ENDSWITH('SparkSQL', 'SQL');
true
> SELECT ENDSWITH('SparkSQL', 'sql');
false
```
## Brief change log
[FLINK-35965](https://issues.apache.org/jira/browse/FLINK-35965)
## Verifying this change
`StringFunctionsITCase#endswithTestCases`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35965) Add ENDSWITH function
[
https://issues.apache.org/jira/browse/FLINK-35965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35965:
---
Labels: pull-request-available (was: )
> Add ENDSWITH function
> -
>
> Key: FLINK-35965
> URL: https://issues.apache.org/jira/browse/FLINK-35965
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Dylan He
>Priority: Major
> Labels: pull-request-available
>
> Add ENDSWITH function.
>
> Returns if {{expr}} ends with {{endExpr}}.
> Example:
> {code:sql}
> > SELECT ENDSWITH('SparkSQL', 'SQL');
> true
> > SELECT ENDSWITH('SparkSQL', 'sql');
> false
> {code}
> Syntax:
> {code:sql}
> ENDSWITH(expr, endExpr)
> {code}
> Arguments:
> * {{expr}}: A STRING or BINARY expression.
> * {{endExpr}}: A STRING or BINARY expression.
> Returns:
> A BOOLEAN.
> {{expr}} and {{endExpr}} should have same type.
> If {{expr}} or {{endExpr}} is NULL, the result is NULL.
> If {{endExpr}} is the empty, the result is true.
> See also:
> *
> [Spark|https://spark.apache.org/docs/3.5.1/sql-ref-functions-builtin.html#string-functions]
> *
> [Databricks|https://docs.databricks.com/en/sql/language-manual/functions/endswith.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35965][table] Add the built-in function ENDSWITH [flink]
flinkbot commented on PR #25157: URL: https://github.com/apache/flink/pull/25157#issuecomment-2270322010 ## CI report: * 29ac26cbcfbcbcd8f6839743c33268e24bd1d2bd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35910][table] Add the built-in function BTRIM [flink]
dylanhz commented on PR #25127: URL: https://github.com/apache/flink/pull/25127#issuecomment-2270429171 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35985) SUBSTRING function not available in transform rules
yux created FLINK-35985: --- Summary: SUBSTRING function not available in transform rules Key: FLINK-35985 URL: https://issues.apache.org/jira/browse/FLINK-35985 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: yux Currently, one could not write `SUBSTRING(str FROM idx FOR len)` in the way that suggested in the doumentations[1], while the workable function, `SUBSTR(str, idx, len)`, isn't mentioned anywhere. Either the code or the docs needs update to avoid confusing users. [1] https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/core-concept/transform/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35986) NULL literal is not supported in Transform rules
yux created FLINK-35986: --- Summary: NULL literal is not supported in Transform rules Key: FLINK-35986 URL: https://issues.apache.org/jira/browse/FLINK-35986 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: yux Sometimes one may want to explicitly write down a NULL value, for example in some CASE - WHEN branches or when calling some UDF functions. However, it is not possible to write down such expressions, and an exception will be thrown: Caused by: java.lang.UnsupportedOperationException: Unsupported type: NULL at org.apache.flink.cdc.runtime.typeutils.DataTypeConverter.convertCalciteRelDataTypeToDataType(DataTypeConverter.java:181) at org.apache.flink.cdc.runtime.parser.TransformParser.generateProjectionColumns(TransformParser.java:189) ... -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35910][table] Add the built-in function BTRIM [flink]
dylanhz commented on PR #25127: URL: https://github.com/apache/flink/pull/25127#issuecomment-2270462361 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35980) Add transform test coverage in Integrated / E2e tests
[ https://issues.apache.org/jira/browse/FLINK-35980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35980: --- Labels: pull-request-available (was: ) > Add transform test coverage in Integrated / E2e tests > - > > Key: FLINK-35980 > URL: https://issues.apache.org/jira/browse/FLINK-35980 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Reporter: yux >Assignee: yux >Priority: Major > Labels: pull-request-available > > Currently, there are no enough UT/E2e test cases to cover transform features > like built-in scalar functions, temporary functions, complex expressions, > etc. Adding them should cover more use-cases and reduce the possibility of > accidentally introduced mistakes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35987) Add ELT function
Dylan He created FLINK-35987:
Summary: Add ELT function
Key: FLINK-35987
URL: https://issues.apache.org/jira/browse/FLINK-35987
Project: Flink
Issue Type: Sub-task
Components: Table SQL / API
Reporter: Dylan He
Add ELT function.
Returns the n-th expression.
Example:
{code:sql}
> SELECT ELT(1, 'scala', 'java');
scala
{code}
Syntax:
{code:sql}
ELT(index, expr...)
{code}
Arguments:
* {{index}}: An INTEGER expression.
* {{expr}}: Any expression that shares a least common type with all {{expr}}..
Returns:
The result has the type of the least common type of all {{expr}}.
{{index}} must be between 1 and the number of {{expr}}. Otherwise, the function
returns an error.
See also:
*
[Spark|https://spark.apache.org/docs/3.5.1/sql-ref-functions-builtin.html#string-functions]
*
[Databricks|https://docs.databricks.com/en/sql/language-manual/functions/elt.html]
*
[MySQL|https://dev.mysql.com/doc/refman/8.4/en/string-functions.html#function_elt]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35743][cdc-runtime] Fix the time zone configuration for temporal functions is not effective [flink-cdc]
leonardBang commented on code in PR #3449:
URL: https://github.com/apache/flink-cdc/pull/3449#discussion_r1705002901
##
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##
@@ -359,6 +407,15 @@ private static Java.Rvalue generateTypeConvertMethod(
case "VARCHAR":
case "STRING":
return new Java.MethodInvocation(Location.NOWHERE, null,
"castToString", atoms);
+case "TIMESTAMP":
+List timestampFunctionParam = new
ArrayList<>(Arrays.asList(atoms));
+timestampFunctionParam.add(
+new Java.AmbiguousName(Location.NOWHERE, new String[]
{DEFAULT_TIME_ZONE}));
+return new Java.MethodInvocation(
+Location.NOWHERE,
+null,
+"castToTimestamp",
+timestampFunctionParam.toArray(new Java.Rvalue[0]));
Review Comment:
I tried to use it but I'm hesitated as many lib used it and we may need to
deal dependency shade issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
