[jira] [Updated] (FLINK-31059) Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by native implementation
[ https://issues.apache.org/jira/browse/FLINK-31059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] miamiaoxyz updated FLINK-31059: --- Attachment: image-2023-02-21-15-59-44-470.png > Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by > native implementation > - > > Key: FLINK-31059 > URL: https://issues.apache.org/jira/browse/FLINK-31059 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Assignee: miamiaoxyz >Priority: Blocker > Fix For: 1.17.0 > > Attachments: image-2023-02-21-15-45-48-226.png, > image-2023-02-21-15-46-13-966.png, image-2023-02-21-15-47-54-043.png, > image-2023-02-21-15-49-58-854.png, image-2023-02-21-15-59-44-470.png > > > This task aims to verify > [FLINK-29717|https://issues.apache.org/jira/browse/FLINK-29717] which > improves the hive udaf performance. > As the document [PR|https://github.com/apache/flink/pull/21789] description, > please veriy: > 1. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions separately in the query to verify if the > hash-agg strategy is chosen via plan, and verify if the data results are the > same as when the option `table.exec.hive.native-agg-function.enabled` is > disabled. > 2. Enabling the option `table.exec.hive.native-agg-function.enabled`, combine > sum/count/avg/min/max functions in query, verify if the hash-agg strategy is > chosen via plan, and verify if the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. > 3. Enabling the option `table.exec.hive.native-agg-function.enabled`, count > or max array&struct and other complex types in query, verify whether the > sort-agg strategy is chosen via plan, verify whether the data result is the > same as when option `table.exec.hive.native-agg-function.enabled` is disabled. > 4. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count and first_value/last_value functions in the query simultaneously, > verify that the sort-agg strategy is chosen via plan, verify that the data is > the same as when option `table.exec.hive.native-agg-function.enabled` is > disabled. > 5. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions in the query and open sort-agg strategy > forcibly, verify that the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Samrat002 commented on pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable
Samrat002 commented on PR #21770: URL: https://github.com/apache/flink/pull/21770#issuecomment-1438005348 Test failure associated ticket https://issues.apache.org/jira/browse/FLINK-31120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-31120) ConcurrentModificationException occurred in StringFunctionsITCase.test
[
https://issues.apache.org/jira/browse/FLINK-31120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reassigned FLINK-31120:
--
Assignee: Shuiqiang Chen
> ConcurrentModificationException occurred in StringFunctionsITCase.test
> --
>
> Key: FLINK-31120
> URL: https://issues.apache.org/jira/browse/FLINK-31120
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Shuiqiang Chen
>Priority: Blocker
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46255&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4&l=12334
> {code}
> Feb 17 04:51:25 [ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 10.725 s <<< FAILURE! - in
> org.apache.flink.table.planner.functions.StringFunctionsITCase
> Feb 17 04:51:25 [ERROR]
> org.apache.flink.table.planner.functions.StringFunctionsITCase.test(TestCase)[4]
> Time elapsed: 4.367 s <<< ERROR!
> Feb 17 04:51:25 org.apache.flink.table.api.TableException: Failed to execute
> sql
> Feb 17 04:51:25 at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeQueryOperation(TableEnvironmentImpl.java:974)
> Feb 17 04:51:25 at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1422)
> Feb 17 04:51:25 at
> org.apache.flink.table.api.internal.TableImpl.execute(TableImpl.java:476)
> Feb 17 04:51:25 at
> org.apache.flink.table.planner.functions.BuiltInFunctionTestBase$ResultTestItem.test(BuiltInFunctionTestBase.java:354)
> Feb 17 04:51:25 at
> org.apache.flink.table.planner.functions.BuiltInFunctionTestBase$TestSetSpec.lambda$getTestCase$4(BuiltInFunctionTestBase.java:320)
> Feb 17 04:51:25 at
> org.apache.flink.table.planner.functions.BuiltInFunctionTestBase$TestCase.execute(BuiltInFunctionTestBase.java:113)
> Feb 17 04:51:25 at
> org.apache.flink.table.planner.functions.BuiltInFunctionTestBase.test(BuiltInFunctionTestBase.java:93)
> Feb 17 04:51:25 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31109) Fails with proxy user not supported even when security.kerberos.fetch.delegation-token is set to false
[
https://issues.apache.org/jira/browse/FLINK-31109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691447#comment-17691447
]
Martijn Visser commented on FLINK-31109:
[~vsowrirajan] Is there any eta on this? Since its a blocker for 1.17, we would
like to understand how long this is expecting to take before being resolved.
> Fails with proxy user not supported even when
> security.kerberos.fetch.delegation-token is set to false
> --
>
> Key: FLINK-31109
> URL: https://issues.apache.org/jira/browse/FLINK-31109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0
>Reporter: Venkata krishnan Sowrirajan
>Priority: Blocker
>
> With
> {code:java}
> security.kerberos.fetch.delegation-token: false
> {code}
> and delegation tokens obtained through our internal service which sets both
> HADOOP_TOKEN_FILE_LOCATION to pick up the DTs and also sets the
> HADOOP_PROXY_USER which fails with the below error
> {code:java}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/export/home/vsowrira/flink-1.18-SNAPSHOT/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/export/apps/hadoop/hadoop-bin_2100503/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> org.apache.flink.runtime.security.modules.SecurityModule$SecurityInstallException:
> Unable to set the Hadoop login user
> at
> org.apache.flink.runtime.security.modules.HadoopModule.install(HadoopModule.java:106)
> at
> org.apache.flink.runtime.security.SecurityUtils.installModules(SecurityUtils.java:76)
> at
> org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:57)
> at
> org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1188)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
> Caused by: java.lang.UnsupportedOperationException: Proxy user is not
> supported
> at
> org.apache.flink.runtime.security.token.hadoop.KerberosLoginProvider.throwProxyUserNotSupported(KerberosLoginProvider.java:137)
> at
> org.apache.flink.runtime.security.token.hadoop.KerberosLoginProvider.isLoginPossible(KerberosLoginProvider.java:81)
> at
> org.apache.flink.runtime.security.modules.HadoopModule.install(HadoopModule.java:73)
> ... 4 more
> {code}
> This seems to have gotten changed after
> [480e6edf|https://github.com/apache/flink/commit/480e6edf9732f8334ef7576080fdbfc98051cb28]
> ([FLINK-28330][runtime][security] Remove old delegation token framework code)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-31109) Fails with proxy user not supported even when security.kerberos.fetch.delegation-token is set to false
[
https://issues.apache.org/jira/browse/FLINK-31109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reassigned FLINK-31109:
--
Assignee: Venkata krishnan Sowrirajan
> Fails with proxy user not supported even when
> security.kerberos.fetch.delegation-token is set to false
> --
>
> Key: FLINK-31109
> URL: https://issues.apache.org/jira/browse/FLINK-31109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0
>Reporter: Venkata krishnan Sowrirajan
>Assignee: Venkata krishnan Sowrirajan
>Priority: Blocker
>
> With
> {code:java}
> security.kerberos.fetch.delegation-token: false
> {code}
> and delegation tokens obtained through our internal service which sets both
> HADOOP_TOKEN_FILE_LOCATION to pick up the DTs and also sets the
> HADOOP_PROXY_USER which fails with the below error
> {code:java}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/export/home/vsowrira/flink-1.18-SNAPSHOT/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/export/apps/hadoop/hadoop-bin_2100503/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> org.apache.flink.runtime.security.modules.SecurityModule$SecurityInstallException:
> Unable to set the Hadoop login user
> at
> org.apache.flink.runtime.security.modules.HadoopModule.install(HadoopModule.java:106)
> at
> org.apache.flink.runtime.security.SecurityUtils.installModules(SecurityUtils.java:76)
> at
> org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:57)
> at
> org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1188)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
> Caused by: java.lang.UnsupportedOperationException: Proxy user is not
> supported
> at
> org.apache.flink.runtime.security.token.hadoop.KerberosLoginProvider.throwProxyUserNotSupported(KerberosLoginProvider.java:137)
> at
> org.apache.flink.runtime.security.token.hadoop.KerberosLoginProvider.isLoginPossible(KerberosLoginProvider.java:81)
> at
> org.apache.flink.runtime.security.modules.HadoopModule.install(HadoopModule.java:73)
> ... 4 more
> {code}
> This seems to have gotten changed after
> [480e6edf|https://github.com/apache/flink/commit/480e6edf9732f8334ef7576080fdbfc98051cb28]
> ([FLINK-28330][runtime][security] Remove old delegation token framework code)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31092) Hive ITCases fail with OutOfMemoryError
[
https://issues.apache.org/jira/browse/FLINK-31092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691453#comment-17691453
]
Martijn Visser commented on FLINK-31092:
[~luoyuxia] Can you take a look?
> Hive ITCases fail with OutOfMemoryError
> ---
>
> Key: FLINK-31092
> URL: https://issues.apache.org/jira/browse/FLINK-31092
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
> Attachments: VisualVM-FLINK-31092.png
>
>
> We're experiencing a OutOfMemoryError where the heap space reaches the upper
> limit:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46161&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=23142
> {code}
> Feb 15 05:05:14 [INFO] Running
> org.apache.flink.table.catalog.hive.HiveCatalogITCase
> Feb 15 05:05:17 [INFO] java.lang.OutOfMemoryError: Java heap space
> Feb 15 05:05:17 [INFO] Dumping heap to java_pid9669.hprof ...
> Feb 15 05:05:28 [INFO] Heap dump file created [1957090051 bytes in 11.718
> secs]
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.maven.surefire.booter.ForkedBooter.cancelPingScheduler(ForkedBooter.java:209)
> at
> org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:419)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:186)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #213: [FLINK-31126] Move classes not depending on Flink runtime from flink-ml-core to flink-ml-servable-core
zhipeng93 commented on code in PR #213: URL: https://github.com/apache/flink-ml/pull/213#discussion_r1112704530 ## flink-ml-servable-core/pom.xml: ## @@ -0,0 +1,127 @@ + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + 4.0.0 + + +org.apache.flink +flink-ml-parent +2.2-SNAPSHOT + + + flink-ml-servable-core + Flink ML : Servable : Core + + + + + org.apache.flink Review Comment: It seems that `flink-ml-servable-core` still relies on `flink-core` and is not consistent with the goal of FLIP-289. Can you explain why do we move XXTypeInfo into `flink-ml-servable-core`? Moreover, there are some classes that do not depend on flink runtime, but not used in `flink-ml-servable-core` (e.g., `MLMetrics`). Can we move it back where it is used? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #213: [FLINK-31126] Move classes not depending on Flink runtime from flink-ml-core to flink-ml-servable-core
zhipeng93 commented on code in PR #213: URL: https://github.com/apache/flink-ml/pull/213#discussion_r1112704530 ## flink-ml-servable-core/pom.xml: ## @@ -0,0 +1,127 @@ + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + 4.0.0 + + +org.apache.flink +flink-ml-parent +2.2-SNAPSHOT + + + flink-ml-servable-core + Flink ML : Servable : Core + + + + + org.apache.flink Review Comment: It seems that `flink-ml-servable-core` still relies on `flink-core` and is not consistent with the goal of FLIP-289. Can you explain why do we move XXTypeInfo into `flink-ml-servable-core`? Moreover, there are some classes that do not depend on flink runtime, but not used in `flink-ml-servable-core` (e.g., `MLMetrics`). Can we move it back where it is used? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-31159) Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask level
[ https://issues.apache.org/jira/browse/FLINK-31159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-31159: -- Assignee: Hong Liang Teoh (was: Jane Chan) > Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask > level > > > Key: FLINK-31159 > URL: https://issues.apache.org/jira/browse/FLINK-31159 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST, Runtime / Web Frontend >Affects Versions: 1.17.0 >Reporter: Hong Liang Teoh >Assignee: Hong Liang Teoh >Priority: Major > Fix For: 1.17.0 > > Attachments: Map_record_distribution.png, all_subtasks.png, > subtask0.png, subtask3.png > > > The issue aims to verify FLINK-30583. > Please verify: > # When below conditions are met: > ** Job has more than 1 parallel subtask > ** Some subtasks are busy, whilst others are not (e.g. some receive records, > some don't) > ** The FlameGraph accurately reflects the busy subtask -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-31159) Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask level
[ https://issues.apache.org/jira/browse/FLINK-31159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh resolved FLINK-31159. - Resolution: Fixed This issue has been completed! Sorry for the churn [~qingyue] + [~renqs] > Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask > level > > > Key: FLINK-31159 > URL: https://issues.apache.org/jira/browse/FLINK-31159 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST, Runtime / Web Frontend >Affects Versions: 1.17.0 >Reporter: Hong Liang Teoh >Assignee: Hong Liang Teoh >Priority: Major > Fix For: 1.17.0 > > Attachments: Map_record_distribution.png, all_subtasks.png, > subtask0.png, subtask3.png > > > The issue aims to verify FLINK-30583. > Please verify: > # When below conditions are met: > ** Job has more than 1 parallel subtask > ** Some subtasks are busy, whilst others are not (e.g. some receive records, > some don't) > ** The FlameGraph accurately reflects the busy subtask -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31159) Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask level
[ https://issues.apache.org/jira/browse/FLINK-31159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh closed FLINK-31159. --- > Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask > level > > > Key: FLINK-31159 > URL: https://issues.apache.org/jira/browse/FLINK-31159 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST, Runtime / Web Frontend >Affects Versions: 1.17.0 >Reporter: Hong Liang Teoh >Assignee: Hong Liang Teoh >Priority: Major > Fix For: 1.17.0 > > Attachments: Map_record_distribution.png, all_subtasks.png, > subtask0.png, subtask3.png > > > The issue aims to verify FLINK-30583. > Please verify: > # When below conditions are met: > ** Job has more than 1 parallel subtask > ** Some subtasks are busy, whilst others are not (e.g. some receive records, > some don't) > ** The FlameGraph accurately reflects the busy subtask -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31059) Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by native implementation
[ https://issues.apache.org/jira/browse/FLINK-31059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] miamiaoxyz updated FLINK-31059: --- Attachment: image-2023-02-21-16-28-22-038.png > Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by > native implementation > - > > Key: FLINK-31059 > URL: https://issues.apache.org/jira/browse/FLINK-31059 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Assignee: miamiaoxyz >Priority: Blocker > Fix For: 1.17.0 > > Attachments: image-2023-02-21-15-45-48-226.png, > image-2023-02-21-15-46-13-966.png, image-2023-02-21-15-47-54-043.png, > image-2023-02-21-15-49-58-854.png, image-2023-02-21-15-59-44-470.png, > image-2023-02-21-16-28-22-038.png > > > This task aims to verify > [FLINK-29717|https://issues.apache.org/jira/browse/FLINK-29717] which > improves the hive udaf performance. > As the document [PR|https://github.com/apache/flink/pull/21789] description, > please veriy: > 1. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions separately in the query to verify if the > hash-agg strategy is chosen via plan, and verify if the data results are the > same as when the option `table.exec.hive.native-agg-function.enabled` is > disabled. > 2. Enabling the option `table.exec.hive.native-agg-function.enabled`, combine > sum/count/avg/min/max functions in query, verify if the hash-agg strategy is > chosen via plan, and verify if the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. > 3. Enabling the option `table.exec.hive.native-agg-function.enabled`, count > or max array&struct and other complex types in query, verify whether the > sort-agg strategy is chosen via plan, verify whether the data result is the > same as when option `table.exec.hive.native-agg-function.enabled` is disabled. > 4. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count and first_value/last_value functions in the query simultaneously, > verify that the sort-agg strategy is chosen via plan, verify that the data is > the same as when option `table.exec.hive.native-agg-function.enabled` is > disabled. > 5. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions in the query and open sort-agg strategy > forcibly, verify that the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31059) Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by native implementation
[ https://issues.apache.org/jira/browse/FLINK-31059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] miamiaoxyz updated FLINK-31059: --- Attachment: image-2023-02-21-16-29-42-983.png > Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by > native implementation > - > > Key: FLINK-31059 > URL: https://issues.apache.org/jira/browse/FLINK-31059 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Assignee: miamiaoxyz >Priority: Blocker > Fix For: 1.17.0 > > Attachments: image-2023-02-21-15-45-48-226.png, > image-2023-02-21-15-46-13-966.png, image-2023-02-21-15-47-54-043.png, > image-2023-02-21-15-49-58-854.png, image-2023-02-21-15-59-44-470.png, > image-2023-02-21-16-28-22-038.png, image-2023-02-21-16-29-42-983.png > > > This task aims to verify > [FLINK-29717|https://issues.apache.org/jira/browse/FLINK-29717] which > improves the hive udaf performance. > As the document [PR|https://github.com/apache/flink/pull/21789] description, > please veriy: > 1. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions separately in the query to verify if the > hash-agg strategy is chosen via plan, and verify if the data results are the > same as when the option `table.exec.hive.native-agg-function.enabled` is > disabled. > 2. Enabling the option `table.exec.hive.native-agg-function.enabled`, combine > sum/count/avg/min/max functions in query, verify if the hash-agg strategy is > chosen via plan, and verify if the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. > 3. Enabling the option `table.exec.hive.native-agg-function.enabled`, count > or max array&struct and other complex types in query, verify whether the > sort-agg strategy is chosen via plan, verify whether the data result is the > same as when option `table.exec.hive.native-agg-function.enabled` is disabled. > 4. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count and first_value/last_value functions in the query simultaneously, > verify that the sort-agg strategy is chosen via plan, verify that the data is > the same as when option `table.exec.hive.native-agg-function.enabled` is > disabled. > 5. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions in the query and open sort-agg strategy > forcibly, verify that the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] snuyanzin commented on pull request #21958: [FLINK-31118][table] Add ARRAY_UNION function.
snuyanzin commented on PR #21958: URL: https://github.com/apache/flink/pull/21958#issuecomment-1438042696 > so i ask you do you have a good implements unfortunately right now i do not have where i can just pick it up. You probably could do some research in this direction -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31059) Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by native implementation
[ https://issues.apache.org/jira/browse/FLINK-31059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] miamiaoxyz updated FLINK-31059: --- Attachment: image-2023-02-21-16-31-58-361.png > Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by > native implementation > - > > Key: FLINK-31059 > URL: https://issues.apache.org/jira/browse/FLINK-31059 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Assignee: miamiaoxyz >Priority: Blocker > Fix For: 1.17.0 > > Attachments: image-2023-02-21-15-45-48-226.png, > image-2023-02-21-15-46-13-966.png, image-2023-02-21-15-47-54-043.png, > image-2023-02-21-15-49-58-854.png, image-2023-02-21-15-59-44-470.png, > image-2023-02-21-16-28-22-038.png, image-2023-02-21-16-29-42-983.png, > image-2023-02-21-16-31-58-361.png > > > This task aims to verify > [FLINK-29717|https://issues.apache.org/jira/browse/FLINK-29717] which > improves the hive udaf performance. > As the document [PR|https://github.com/apache/flink/pull/21789] description, > please veriy: > 1. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions separately in the query to verify if the > hash-agg strategy is chosen via plan, and verify if the data results are the > same as when the option `table.exec.hive.native-agg-function.enabled` is > disabled. > 2. Enabling the option `table.exec.hive.native-agg-function.enabled`, combine > sum/count/avg/min/max functions in query, verify if the hash-agg strategy is > chosen via plan, and verify if the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. > 3. Enabling the option `table.exec.hive.native-agg-function.enabled`, count > or max array&struct and other complex types in query, verify whether the > sort-agg strategy is chosen via plan, verify whether the data result is the > same as when option `table.exec.hive.native-agg-function.enabled` is disabled. > 4. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count and first_value/last_value functions in the query simultaneously, > verify that the sort-agg strategy is chosen via plan, verify that the data is > the same as when option `table.exec.hive.native-agg-function.enabled` is > disabled. > 5. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions in the query and open sort-agg strategy > forcibly, verify that the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31164) Unexpected correlate variable $cor0 in the plan error in where clause
[
https://issues.apache.org/jira/browse/FLINK-31164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-31164.
--
Resolution: Duplicate
> Unexpected correlate variable $cor0 in the plan error in where clause
> -
>
> Key: FLINK-31164
> URL: https://issues.apache.org/jira/browse/FLINK-31164
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.16.0
>Reporter: P Rohan Kumar
>Priority: Major
>
> {code:java}
> val env: StreamExecutionEnvironment =
> StreamExecutionEnvironment.getExecutionEnvironment
> val tableEnv = StreamTableEnvironment.create(env)
> val accountsTd =
> TableDescriptor.forConnector("datagen").option("rows-per-second", "10")
> .option("number-of-rows", "10")
> .schema(Schema
> .newBuilder()
> .column("account_num", DataTypes.VARCHAR(2147483647))
> .column("acc_name", DataTypes.VARCHAR(2147483647))
> .column("acc_phone_num", DataTypes.VARCHAR(2147483647))
> .build())
> .build()
> val accountsTable = tableEnv.from(accountsTd)
> tableEnv.createTemporaryView("accounts", accountsTable)
> val transactionsTd =
> TableDescriptor.forConnector("datagen").option("rows-per-second", "10")
> .option("number-of-rows", "10")
> .schema(Schema
> .newBuilder()
> .column("account_num", DataTypes.VARCHAR(2147483647))
> .column("transaction_place", DataTypes.VARCHAR(2147483647))
> .column("transaction_time", DataTypes.BIGINT())
> .column("amount", DataTypes.INT())
> .build())
> .build()
> val transactionsTable = tableEnv.from(transactionsTd)
> tableEnv.createTemporaryView("transaction_data", transactionsTable)
> val newTable = tableEnv.sqlQuery("select acc.account_num, (select count(*)
> from transaction_data where transaction_place = trans.transaction_place and
> account_num = acc.account_num) from accounts acc,transaction_data trans")
> tableEnv.toChangelogStream(newTable).print()
> env.execute() {code}
> I get the following error if I run the above code.
>
> {code:java}
> Exception in thread "main" org.apache.flink.table.api.TableException:
> unexpected correlate variable $cor0 in the plan
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkDecorrelateProgram.checkCorrelVariableExists(FlinkDecorrelateProgram.scala:59)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkDecorrelateProgram.optimize(FlinkDecorrelateProgram.scala:42)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram.$anonfun$optimize$2(FlinkGroupProgram.scala:59)
> at
> scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:187)
> at
> scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:185)
> at scala.collection.Iterator.foreach(Iterator.scala:943)
> at scala.collection.Iterator.foreach$(Iterator.scala:943)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:189)
> at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:184)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram.$anonfun$optimize$1(FlinkGroupProgram.scala:56)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram.$anonfun$optimize$1$adapted(FlinkGroupProgram.scala:51)
> at
> scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:187)
> at
> scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:185)
> at scala.collection.immutable.Range.foreach(Range.scala:158)
> at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:189)
> at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:184)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram.optimize(FlinkGroupProgram.scala:51)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:59)
> at
> scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:187)
> at
> scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:185)
> at scala.collection.Iterator.foreach(Iterator.scala:943)
> at scala.collection.Iterator.foreach$(Iterator.scala:943)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
> at scala.colle
[jira] [Updated] (FLINK-31059) Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by native implementation
[ https://issues.apache.org/jira/browse/FLINK-31059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] miamiaoxyz updated FLINK-31059: --- Attachment: image-2023-02-21-16-35-46-294.png > Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by > native implementation > - > > Key: FLINK-31059 > URL: https://issues.apache.org/jira/browse/FLINK-31059 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Assignee: miamiaoxyz >Priority: Blocker > Fix For: 1.17.0 > > Attachments: image-2023-02-21-15-45-48-226.png, > image-2023-02-21-15-46-13-966.png, image-2023-02-21-15-47-54-043.png, > image-2023-02-21-15-49-58-854.png, image-2023-02-21-15-59-44-470.png, > image-2023-02-21-16-28-22-038.png, image-2023-02-21-16-29-42-983.png, > image-2023-02-21-16-31-58-361.png, image-2023-02-21-16-35-46-294.png > > > This task aims to verify > [FLINK-29717|https://issues.apache.org/jira/browse/FLINK-29717] which > improves the hive udaf performance. > As the document [PR|https://github.com/apache/flink/pull/21789] description, > please veriy: > 1. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions separately in the query to verify if the > hash-agg strategy is chosen via plan, and verify if the data results are the > same as when the option `table.exec.hive.native-agg-function.enabled` is > disabled. > 2. Enabling the option `table.exec.hive.native-agg-function.enabled`, combine > sum/count/avg/min/max functions in query, verify if the hash-agg strategy is > chosen via plan, and verify if the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. > 3. Enabling the option `table.exec.hive.native-agg-function.enabled`, count > or max array&struct and other complex types in query, verify whether the > sort-agg strategy is chosen via plan, verify whether the data result is the > same as when option `table.exec.hive.native-agg-function.enabled` is disabled. > 4. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count and first_value/last_value functions in the query simultaneously, > verify that the sort-agg strategy is chosen via plan, verify that the data is > the same as when option `table.exec.hive.native-agg-function.enabled` is > disabled. > 5. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions in the query and open sort-agg strategy > forcibly, verify that the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31142) Some queries lead to abrupt sql client close
[
https://issues.apache.org/jira/browse/FLINK-31142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-31142:
Description:
Although the behavior has been changed in 1.17.0, I'm not sure whether it is a
blocker or not, since in both cases it is invalid query.
The difference in the behavior is that before 1.17.0
a query like
{code:sql}
select /* multiline comment;
{code}
fails to execute and sql client prompts to submit another query.
In 1.17.0 it fails with
{noformat}
Exception in thread "main" org.apache.flink.table.client.SqlClientException:
Could not read from command line.
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:205)
at
org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:168)
at
org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:113)
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
at
org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error at
line 1, column 29. Encountered: after : ""
at
org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:26752)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.scan(SqlCommandParserImpl.java:89)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.next(SqlCommandParserImpl.java:81)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.checkIncompleteStatement(SqlCommandParserImpl.java:141)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.getCommand(SqlCommandParserImpl.java:111)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement(SqlCommandParserImpl.java:52)
at
org.apache.flink.table.client.cli.parser.SqlMultiLineParser.parse(SqlMultiLineParser.java:82)
at
org.jline.reader.impl.LineReaderImpl.acceptLine(LineReaderImpl.java:2964)
at
org.jline.reader.impl.LineReaderImpl$1.apply(LineReaderImpl.java:3778)
at
org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:679)
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:183)
... 6 more
Shutting down the session...
done.
{noformat}
And it shuts down the sessoin
was:
Although the behavior has been changed in 1.17.0, I'm not sure whether it is a
blocker or not, since in both cases it is invalid query.
The difference in the behavior is that before 1.17.0
a query like
{code:sql}
select /* multiline comment;
{code}
query fails to execute and sql client prompts to submit another query.
In 1.17.0 it fails with
{noformat}
Exception in thread "main" org.apache.flink.table.client.SqlClientException:
Could not read from command line.
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:205)
at
org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:168)
at
org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:113)
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
at
org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error at
line 1, column 29. Encountered: after : ""
at
org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:26752)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.scan(SqlCommandParserImpl.java:89)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.next(SqlCommandParserImpl.java:81)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.checkIncompleteStatement(SqlCommandParserImpl.java:141)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.getCommand(SqlCommandParserImpl.java:111)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement(SqlCommandParserImpl.java:52)
at
org.apache.flink.table.client.cli.parser.SqlMultiLineParser.parse(SqlMultiLineParser.java:82)
at
org.jline.reader.impl.LineReaderImpl.acceptLine(LineReaderImpl.java:2964)
at
org.jline.reader.impl.LineReaderImpl$1.apply(LineReaderImpl.java:3778)
at
org.jline.reader.impl.LineReaderIm
[jira] [Updated] (FLINK-31142) Some queries lead to abrupt sql client close
[
https://issues.apache.org/jira/browse/FLINK-31142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-31142:
Description:
Although the behavior has been changed in 1.17.0, I'm not sure whether it is a
blocker or not, since in both cases it is invalid query.
The difference in the behavior is that before 1.17.0
a query like
{code:sql}
select /* multiline comment;
{code}
fails to execute and sql client prompts to submit another query.
In 1.17.0 it shuts down the session failing with
{noformat}
Exception in thread "main" org.apache.flink.table.client.SqlClientException:
Could not read from command line.
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:205)
at
org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:168)
at
org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:113)
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
at
org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error at
line 1, column 29. Encountered: after : ""
at
org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:26752)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.scan(SqlCommandParserImpl.java:89)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.next(SqlCommandParserImpl.java:81)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.checkIncompleteStatement(SqlCommandParserImpl.java:141)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.getCommand(SqlCommandParserImpl.java:111)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement(SqlCommandParserImpl.java:52)
at
org.apache.flink.table.client.cli.parser.SqlMultiLineParser.parse(SqlMultiLineParser.java:82)
at
org.jline.reader.impl.LineReaderImpl.acceptLine(LineReaderImpl.java:2964)
at
org.jline.reader.impl.LineReaderImpl$1.apply(LineReaderImpl.java:3778)
at
org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:679)
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:183)
... 6 more
Shutting down the session...
done.
{noformat}
was:
Although the behavior has been changed in 1.17.0, I'm not sure whether it is a
blocker or not, since in both cases it is invalid query.
The difference in the behavior is that before 1.17.0
a query like
{code:sql}
select /* multiline comment;
{code}
fails to execute and sql client prompts to submit another query.
In 1.17.0 it fails with
{noformat}
Exception in thread "main" org.apache.flink.table.client.SqlClientException:
Could not read from command line.
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:205)
at
org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:168)
at
org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:113)
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
at
org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error at
line 1, column 29. Encountered: after : ""
at
org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:26752)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.scan(SqlCommandParserImpl.java:89)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.next(SqlCommandParserImpl.java:81)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.checkIncompleteStatement(SqlCommandParserImpl.java:141)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.getCommand(SqlCommandParserImpl.java:111)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement(SqlCommandParserImpl.java:52)
at
org.apache.flink.table.client.cli.parser.SqlMultiLineParser.parse(SqlMultiLineParser.java:82)
at
org.jline.reader.impl.LineReaderImpl.acceptLine(LineReaderImpl.java:2964)
at
org.jline.reader.impl.LineReaderImpl$1.apply(LineReaderImpl.java:3778)
at
org.jline.reader.impl.LineReaderImpl.readLi
[jira] [Updated] (FLINK-31142) Some queries lead to abrupt sql client close
[
https://issues.apache.org/jira/browse/FLINK-31142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-31142:
Labels: (was: release-testing)
> Some queries lead to abrupt sql client close
>
>
> Key: FLINK-31142
> URL: https://issues.apache.org/jira/browse/FLINK-31142
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> Although the behavior has been changed in 1.17.0, I'm not sure whether it is
> a blocker or not, since in both cases it is invalid query.
> The difference in the behavior is that before 1.17.0
> a query like
> {code:sql}
> select /* multiline comment;
> {code}
> fails to execute and sql client prompts to submit another query.
> In 1.17.0 it shuts down the session failing with
> {noformat}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Could not read from command line.
> at
> org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:205)
> at
> org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:168)
> at
> org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:113)
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error at
> line 1, column 29. Encountered: after : ""
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:26752)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.scan(SqlCommandParserImpl.java:89)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.next(SqlCommandParserImpl.java:81)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.checkIncompleteStatement(SqlCommandParserImpl.java:141)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.getCommand(SqlCommandParserImpl.java:111)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement(SqlCommandParserImpl.java:52)
> at
> org.apache.flink.table.client.cli.parser.SqlMultiLineParser.parse(SqlMultiLineParser.java:82)
> at
> org.jline.reader.impl.LineReaderImpl.acceptLine(LineReaderImpl.java:2964)
> at
> org.jline.reader.impl.LineReaderImpl$1.apply(LineReaderImpl.java:3778)
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:679)
> at
> org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:183)
> ... 6 more
> Shutting down the session...
> done.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] rkhachatryan opened a new pull request, #21981: [FLINK-21450][runtime] Support LocalRecovery by AdaptiveScheduler
rkhachatryan opened a new pull request, #21981: URL: https://github.com/apache/flink/pull/21981 ## What is the purpose of the change Adjust slot assignment by Adaptive Scheduler to try to re-use previous allocations so that TMs can use Local Recovery. Contributed mostly by @dmvk. The main defferences from the original contribution: 1. Previous `ExecutionGraph` is passed from the previous state explicitly (currently, `WaitingForResources` stage, which triggers the computation, doesn't have the graph) 2. In `SlotAssigner`, the split into two methods is removed mostly for consistency (two methods mostly duplicated each other). That results in higher asymptotical complexity of `StateLocalitySlotAssigner` (`O(mnlog*mnlog)` vs `O(mnlog`) 3. DoP is computed according to FLINK-30895 ## Brief change log - Support LocalRecovery by AdaptiveScheduler - Add previous ExecutionGraph to WaitingForResources AdaptiveScheduler state - Make LocalRecoveryITCase fail when allocations don't match ## Verifying this change - Adjusted `LocalRecoveryITCase` - Added `SlotSharingSlotAllocatorTest.testStickyAllocation` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21450) Add local recovery support to adaptive scheduler
[
https://issues.apache.org/jira/browse/FLINK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-21450:
---
Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
pull-request-available (was: auto-deprioritized-major auto-deprioritized-minor
auto-unassigned)
> Add local recovery support to adaptive scheduler
>
>
> Key: FLINK-21450
> URL: https://issues.apache.org/jira/browse/FLINK-21450
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Robert Metzger
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned, pull-request-available
> Fix For: 1.18.0
>
>
> local recovery means that, on a failure, we are able to re-use the state in a
> taskmanager, instead of loading it again from distributed storage (which
> means the scheduler needs to know where which state is located, and schedule
> tasks accordingly).
> Adaptive Scheduler is currently not respecting the location of state, so
> failures require the re-loading of state from the distributed storage.
> Adding this feature will allow us to enable the {{Local recovery and sticky
> scheduling end-to-end test}} for adaptive scheduler again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new major release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Minor releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new major release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Cron job has been added on the release branch in
([apache-flink:./tools/azure-pipelines/build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
* Originating branch has the version information updated to the new version
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure {{flink-docker}} has {{dev-x.y}} branch and docker e2e tests run
against this branch
* docs/config.toml has been updated appropriately.
* The {{dev-x.y}} branch of ({{{}$CURRENT_SNAPSHOT_VERSION{}}}) have been
created in the Flink Benchmark repo.
* The {{flink.version}} property of Flink Benchmark repo has been updated to
the latest snapshot version.
was:
If you are doing a new major release, you need to update Flink version in two
repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Minor releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{v1_16("1.16")}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.gi
[GitHub] [flink] reswqa commented on a diff in pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
reswqa commented on code in PR #21977:
URL: https://github.com/apache/flink/pull/21977#discussion_r1112733682
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkTest.java:
##
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive;
+
+import org.apache.flink.api.common.RuntimeExecutionMode;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.table.FileSystemConnectorOptions;
+import org.apache.flink.connector.file.table.batch.BatchSink;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveTable;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.graph.StreamNode;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogTableImpl;
+import org.apache.flink.table.catalog.CatalogTest;
+import org.apache.flink.table.catalog.Column;
+import org.apache.flink.table.catalog.ObjectIdentifier;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.ResolvedCatalogTable;
+import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.catalog.hive.HiveCatalog;
+import org.apache.flink.table.catalog.hive.HiveTestUtils;
+import org.apache.flink.table.connector.sink.DataStreamSinkProvider;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.factories.DynamicTableSinkFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import
org.apache.flink.table.runtime.connector.sink.SinkRuntimeProviderContext;
+import org.apache.flink.util.TestLoggerExtension;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link HiveTableSink} enable auto-compaction. */
+@ExtendWith(TestLoggerExtension.class)
+class HiveTableCompactSinkTest {
+
+private static HiveCatalog catalog;
+
+private Configuration tableConf;
+
+@BeforeAll
+static void before() {
+catalog = HiveTestUtils.createHiveCatalog();
+catalog.open();
+}
+
+@AfterAll
+static void after() {
+catalog.close();
+}
+
+@BeforeEach
+void resetTableConf() {
+tableConf = new Configuration();
+tableConf.set(FactoryUtil.CONNECTOR, SqlCreateHiveTable.IDENTIFIER);
+tableConf.set(FileSystemConnectorOptions.AUTO_COMPACTION, true);
+}
+
+/** If only sink parallelism is set, compact operator should follow this
setting. */
+@Test
+void testOnlySetSinkParallelism() throws Exception {
+final int sinkParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.SINK_PARALLELISM,
sinkParallelism);
+
+assertSinkAndCompactOperatorParallelism(
+tableConf, true, true, sinkParallelism, sinkParallelism);
+}
+
+@Test
+void testOnlySetCompactParallelism() throws Exception {
+final int compactParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.COMPACTION_PARALLELISM,
compactParallelism);
+
+assertSinkAndCompactOperatorParallelism(tableConf, false, true, -1,
compactParallelism);
+}
+
+@Test
+void testSetBothSinkAndCompactParallelism() throws Exception {
+final int sinkParallelism = 8;
+final int compactParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.SINK_PARALLELISM,
sinkParallelism);
+tableConf.set(FileSystemConnectorOptions.COMPACTION_PARALLELISM,
compactParallelism);
+
+assertSinkAndCompactOperator
[GitHub] [flink] reswqa commented on a diff in pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
reswqa commented on code in PR #21977:
URL: https://github.com/apache/flink/pull/21977#discussion_r1112739085
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkTest.java:
##
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive;
+
+import org.apache.flink.api.common.RuntimeExecutionMode;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.table.FileSystemConnectorOptions;
+import org.apache.flink.connector.file.table.batch.BatchSink;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveTable;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.graph.StreamNode;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogTableImpl;
+import org.apache.flink.table.catalog.CatalogTest;
+import org.apache.flink.table.catalog.Column;
+import org.apache.flink.table.catalog.ObjectIdentifier;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.ResolvedCatalogTable;
+import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.catalog.hive.HiveCatalog;
+import org.apache.flink.table.catalog.hive.HiveTestUtils;
+import org.apache.flink.table.connector.sink.DataStreamSinkProvider;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.factories.DynamicTableSinkFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import
org.apache.flink.table.runtime.connector.sink.SinkRuntimeProviderContext;
+import org.apache.flink.util.TestLoggerExtension;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link HiveTableSink} enable auto-compaction. */
+@ExtendWith(TestLoggerExtension.class)
+class HiveTableCompactSinkTest {
+
+private static HiveCatalog catalog;
+
+private Configuration tableConf;
+
+@BeforeAll
+static void before() {
+catalog = HiveTestUtils.createHiveCatalog();
+catalog.open();
+}
+
+@AfterAll
+static void after() {
+catalog.close();
+}
+
+@BeforeEach
+void resetTableConf() {
+tableConf = new Configuration();
+tableConf.set(FactoryUtil.CONNECTOR, SqlCreateHiveTable.IDENTIFIER);
+tableConf.set(FileSystemConnectorOptions.AUTO_COMPACTION, true);
+}
+
+/** If only sink parallelism is set, compact operator should follow this
setting. */
+@Test
+void testOnlySetSinkParallelism() throws Exception {
+final int sinkParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.SINK_PARALLELISM,
sinkParallelism);
+
+assertSinkAndCompactOperatorParallelism(
+tableConf, true, true, sinkParallelism, sinkParallelism);
+}
+
+@Test
+void testOnlySetCompactParallelism() throws Exception {
+final int compactParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.COMPACTION_PARALLELISM,
compactParallelism);
+
+assertSinkAndCompactOperatorParallelism(tableConf, false, true, -1,
compactParallelism);
+}
+
+@Test
+void testSetBothSinkAndCompactParallelism() throws Exception {
+final int sinkParallelism = 8;
+final int compactParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.SINK_PARALLELISM,
sinkParallelism);
+tableConf.set(FileSystemConnectorOptions.COMPACTION_PARALLELISM,
compactParallelism);
+
+assertSinkAndCompactOperator
[GitHub] [flink] reswqa commented on a diff in pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
reswqa commented on code in PR #21977:
URL: https://github.com/apache/flink/pull/21977#discussion_r1112739085
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkTest.java:
##
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive;
+
+import org.apache.flink.api.common.RuntimeExecutionMode;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.table.FileSystemConnectorOptions;
+import org.apache.flink.connector.file.table.batch.BatchSink;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveTable;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.graph.StreamNode;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogTableImpl;
+import org.apache.flink.table.catalog.CatalogTest;
+import org.apache.flink.table.catalog.Column;
+import org.apache.flink.table.catalog.ObjectIdentifier;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.ResolvedCatalogTable;
+import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.catalog.hive.HiveCatalog;
+import org.apache.flink.table.catalog.hive.HiveTestUtils;
+import org.apache.flink.table.connector.sink.DataStreamSinkProvider;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.factories.DynamicTableSinkFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import
org.apache.flink.table.runtime.connector.sink.SinkRuntimeProviderContext;
+import org.apache.flink.util.TestLoggerExtension;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link HiveTableSink} enable auto-compaction. */
+@ExtendWith(TestLoggerExtension.class)
+class HiveTableCompactSinkTest {
+
+private static HiveCatalog catalog;
+
+private Configuration tableConf;
+
+@BeforeAll
+static void before() {
+catalog = HiveTestUtils.createHiveCatalog();
+catalog.open();
+}
+
+@AfterAll
+static void after() {
+catalog.close();
+}
+
+@BeforeEach
+void resetTableConf() {
+tableConf = new Configuration();
+tableConf.set(FactoryUtil.CONNECTOR, SqlCreateHiveTable.IDENTIFIER);
+tableConf.set(FileSystemConnectorOptions.AUTO_COMPACTION, true);
+}
+
+/** If only sink parallelism is set, compact operator should follow this
setting. */
+@Test
+void testOnlySetSinkParallelism() throws Exception {
+final int sinkParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.SINK_PARALLELISM,
sinkParallelism);
+
+assertSinkAndCompactOperatorParallelism(
+tableConf, true, true, sinkParallelism, sinkParallelism);
+}
+
+@Test
+void testOnlySetCompactParallelism() throws Exception {
+final int compactParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.COMPACTION_PARALLELISM,
compactParallelism);
+
+assertSinkAndCompactOperatorParallelism(tableConf, false, true, -1,
compactParallelism);
+}
+
+@Test
+void testSetBothSinkAndCompactParallelism() throws Exception {
+final int sinkParallelism = 8;
+final int compactParallelism = 4;
+
+tableConf.set(FileSystemConnectorOptions.SINK_PARALLELISM,
sinkParallelism);
+tableConf.set(FileSystemConnectorOptions.COMPACTION_PARALLELISM,
compactParallelism);
+
+assertSinkAndCompactOperator
[GitHub] [flink] reswqa commented on pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
reswqa commented on PR #21977: URL: https://github.com/apache/flink/pull/21977#issuecomment-1438069932 Thanks @luoyuxia and @JunRuiLee for the review, I have updated this pr in two fixup commits, PTAL agian~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691462#comment-17691462
]
Matthias Pohl commented on FLINK-31153:
---
{quote}
BTW, you mentioned in the description: you need to update Flink version in
"two" repositories followed with three repos. Did you mean "three"?
{quote}
Thanks for the hint. The content in the description is coming from the [Flink
docs on how to create a
release|https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management].
I copied it and tried to align it but must have missed that bit. (y)
{quote}
It seems those tasks contain more or different information. Are you planning to
build a template for the future release?
{quote}
We're experimenting a bit with improving the documentation. One idea is to move
the documentation into Jira to enable release managers to document their
efforts (see [the release sync protocol from February 14, 2023
week|https://cwiki.apache.org/confluence/display/FLINK/1.17+Release#id-1.17Release-2023-02-14]).
For now, this is meant as a test. If it works well, we will create an improved
Jira issue collection which can serve as templates. AFAIU, the Apache Jira
doesn't provide templates, though. But having a dedicated set of FLINK Jira
issues that can be cloned might be good enough.
> Create a release branch
> ---
>
> Key: FLINK-31153
> URL: https://issues.apache.org/jira/browse/FLINK-31153
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>
> If you are doing a new major release, you need to update Flink version in the
> following repositories:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Minor releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ cd ./tools
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Flink repository
> Create a branch for the new version that we want to release before updating
> the master branch to the next development version:
> {code:bash}
> $ cd ./tools
> $ releasing/create_snapshot_branch.sh
> $ git checkout master
> $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
> releasing/update_branch_version.sh
> {code}
> In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> as the last entry:
> {code:java}
> // ...
> v1_12("1.12"),
> v1_13("1.13"),
> v1_14("1.14"),
> v1_15("1.15"),
> v1_16("1.16");
> {code}
> The newly created branch and updated {{master}} branch need to be pushed to
> the official repository.
> h4. Flink Docker Repository
> Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
> [apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
> sure that
> [apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
> points to the correct snapshot version; for {{dev-x.y}} it should point to
> {{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
> recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
> After pushing the new major release branch, as the last step you should also
> update the documentation workflow to also build the documentation for the new
> release branch. Check [Managing
> Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
> on details on how to do that. You may also want to manually trigger a build
> to make the changes visible as soon as possible.
>
>
> h3. Expectations (Minor Version only)
> * Release branch has been created and pushed
> * Cron job has been added on the release branch in
> ([apache-flink:./tools/azure-pipelines/build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
> * Originating branch has the version information updated to the new version
> * New version is added to the
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> enum.
> * Make sure {{flink-docker}} has {{dev-x.y}} branch and docker e2e tests run
> against this branch
> * docs/config.toml has been updated appropriately.
> * The {{dev-x.y}} branch of ({{{}$CURRENT_SNAPSHOT_VERSION{}}}) have been
> created in the Flink Benchmark repo.
> * The {{flink.version}} property of Fl
[GitHub] [flink] flinkbot commented on pull request #21981: [FLINK-21450][runtime] Support LocalRecovery by AdaptiveScheduler
flinkbot commented on PR #21981: URL: https://github.com/apache/flink/pull/21981#issuecomment-1438071101 ## CI report: * 90f669d3099333992edcd961682b0cde4fa06dc8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a diff in pull request #21981: [FLINK-21450][runtime] Support LocalRecovery by AdaptiveScheduler
rkhachatryan commented on code in PR #21981:
URL: https://github.com/apache/flink/pull/21981#discussion_r1112741455
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/allocator/SlotSharingSlotAllocator.java:
##
@@ -121,16 +133,13 @@ public Optional
determineParallelism(
slotSharingGroupParallelism.get(
slotSharingGroup.getSlotSharingGroupId()));
-final Iterable
sharedSlotToVertexAssignment =
+final List sharedSlotToVertexAssignment
=
createExecutionSlotSharingGroups(vertexParallelism);
-for (ExecutionSlotSharingGroup executionSlotSharingGroup :
-sharedSlotToVertexAssignment) {
-final SlotInfo slotInfo = slotIterator.next();
-
-assignments.add(
-new
ExecutionSlotSharingGroupAndSlot(executionSlotSharingGroup, slotInfo));
-}
+SlotAssigner.AssignmentResult result =
+slotAssigner.assignSlots(freeSlots,
sharedSlotToVertexAssignment);
+assignments.addAll(result.assignments);
+freeSlots = result.remainingSlots;
Review Comment:
Because `assignSlots` is called in a loop, and internally iterates over the
`freeSlots`, the complexity here is suboptimal. Lower-priority slots are added
over and over into the queue inside `assignSlots`, resulting in quadratic
complexity.
I was thinking about making free slots part of `SlotAssigner` state (passed
into constructor).
WDYT @dmvk ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] pnowojski commented on a diff in pull request #21923: FLINK-13871: Consolidate volatile status fields in StreamTask
pnowojski commented on code in PR #21923:
URL: https://github.com/apache/flink/pull/21923#discussion_r1112741636
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java:
##
@@ -949,7 +952,9 @@ public final void cleanUp(Throwable throwable) throws
Exception {
// disabled the interruptions or not.
getCompletionFuture().exceptionally(unused -> null).join();
// clean up everything we initialized
-isRunning = false;
+if (!isCanceled() && !isFailing()) {
Review Comment:
> On a side note, those volatile accesses aren't on the hot path so
performance-wise it shouldn't change much
Yes, this doesn't affect performance. This ticket is about cleaning up the
code and improving readability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691462#comment-17691462
]
Matthias Pohl edited comment on FLINK-31153 at 2/21/23 8:48 AM:
{quote}
BTW, you mentioned in the description: you need to update Flink version in
"two" repositories followed with three repos. Did you mean "three"?
{quote}
Thanks for the hint. I updated the description accordingly. The content in the
description is coming from the [Flink docs on how to create a
release|https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management].
I copied it and tried to align it but must have missed that bit. (y)
{quote}
It seems those tasks contain more or different information. Are you planning to
build a template for the future release?
{quote}
We're experimenting a bit with improving the documentation. One idea is to move
the documentation into Jira to enable release managers to document their
efforts (see [the release sync protocol from February 14, 2023
week|https://cwiki.apache.org/confluence/display/FLINK/1.17+Release#id-1.17Release-2023-02-14]).
For now, this is meant as a test. If it works well, we will create an improved
Jira issue collection which can serve as templates. AFAIU, the Apache Jira
doesn't provide templates, though. But having a dedicated set of FLINK Jira
issues that can be cloned might be good enough.
was (Author: mapohl):
{quote}
BTW, you mentioned in the description: you need to update Flink version in
"two" repositories followed with three repos. Did you mean "three"?
{quote}
Thanks for the hint. The content in the description is coming from the [Flink
docs on how to create a
release|https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management].
I copied it and tried to align it but must have missed that bit. (y)
{quote}
It seems those tasks contain more or different information. Are you planning to
build a template for the future release?
{quote}
We're experimenting a bit with improving the documentation. One idea is to move
the documentation into Jira to enable release managers to document their
efforts (see [the release sync protocol from February 14, 2023
week|https://cwiki.apache.org/confluence/display/FLINK/1.17+Release#id-1.17Release-2023-02-14]).
For now, this is meant as a test. If it works well, we will create an improved
Jira issue collection which can serve as templates. AFAIU, the Apache Jira
doesn't provide templates, though. But having a dedicated set of FLINK Jira
issues that can be cloned might be good enough.
> Create a release branch
> ---
>
> Key: FLINK-31153
> URL: https://issues.apache.org/jira/browse/FLINK-31153
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>
> If you are doing a new major release, you need to update Flink version in the
> following repositories:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Minor releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ cd ./tools
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Flink repository
> Create a branch for the new version that we want to release before updating
> the master branch to the next development version:
> {code:bash}
> $ cd ./tools
> $ releasing/create_snapshot_branch.sh
> $ git checkout master
> $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
> releasing/update_branch_version.sh
> {code}
> In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> as the last entry:
> {code:java}
> // ...
> v1_12("1.12"),
> v1_13("1.13"),
> v1_14("1.14"),
> v1_15("1.15"),
> v1_16("1.16");
> {code}
> The newly created branch and updated {{master}} branch need to be pushed to
> the official repository.
> h4. Flink Docker Repository
> Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
> [apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
> sure that
> [apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
> points to the correct snapshot version; for {{dev-x.y}} it should point to
> {{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
> recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
> After pushing the new major release branch, as the last step you should also
>
[GitHub] [flink] MartijnVisser merged pull request #21964: [FLINK-30948][Formats/AWS] Remove GlueSchemaRegistry Avro and JSON fo…
MartijnVisser merged PR #21964: URL: https://github.com/apache/flink/pull/21964 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-30948) Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry from Flink main repo
[ https://issues.apache.org/jira/browse/FLINK-30948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-30948: --- Fix Version/s: 1.17.0 > Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry > from Flink main repo > --- > > Key: FLINK-30948 > URL: https://issues.apache.org/jira/browse/FLINK-30948 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) > Environment: Remove flink-avro-glue-schema-registry and > flink-json-glue-schema-registry from Flink main repo, along with associated > end-to-end tests >Reporter: Hong Liang Teoh >Assignee: Hong Liang Teoh >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new minor release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Cron job has been added on the release branch in
([apache-flink:./tools/azure-pipelines/build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
* Originating branch has the version information updated to the new version
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure {{flink-docker}} has {{dev-x.y}} branch and docker e2e tests run
against this branch
* docs/config.toml has been updated appropriately.
* The {{dev-x.y}} branch of ({{{}$CURRENT_SNAPSHOT_VERSION{}}}) have been
created in the Flink Benchmark repo.
* The {{flink.version}} property of Flink Benchmark repo has been updated to
the latest snapshot version.
was:
If you are doing a new major release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Minor releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/fl
[jira] [Commented] (FLINK-31059) Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by native implementation
[
https://issues.apache.org/jira/browse/FLINK-31059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691464#comment-17691464
]
miamiaoxyz commented on FLINK-31059:
I first use sql client to test the feature, but the
`exec.hive.native-agg-function.enabled` do not work.
I then use ITCase to verify.
# I use IT case to turn on and off `exec.hive.native-agg-function.enabled`
to verify the two results are the same, testSql function test whether the same
sql get same result with on and off `exec.hive.native-agg-function.enabled`.
I verfied that the plan use Hashagg when turn on the
`exec.hive.native-agg-function.enabled`, and the plan use SortAgg when turn off
by IT case.
!image-2023-02-21-15-45-48-226.png|width=549,height=234!
It pass all the IT Case below.
!image-2023-02-21-15-46-13-966.png|width=501,height=371!
2. I verified that data results are the same when combine sum/count/avg/min/max
functions in query using `exec.hive.native-agg-function.enabled` on and off
using the IT case below.
I verfied that the plan use Hashagg when turn on the
`exec.hive.native-agg-function.enabled`, and the plan use SortAgg when turn off
by IT case.
!image-2023-02-21-15-49-58-854.png|width=536,height=219!
3. For `array` and `struct` do not support the max function. For count
function, it does not store `array` or `struct` in agg, so they use bigint
instead, and hash-agg is chosen .
!image-2023-02-21-15-59-44-470.png|width=1016,height=189!
4. For `first_value` and `last_value` are not implemented in hive,
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-Built-inAggregateFunctions(UDAF)]
I use `collect_set` to test instead. All the plan use SortAgg, and get same
result, which meet the expectations.
```
--- turn on `table.exec.hive.native-agg-function.enabled`
== Abstract Syntax Tree ==
LogicalProject(x=[$0], _o__c1=[$1])
+- LogicalAggregate(group=[\{0}], agg#0=[collect_set($1)])
+- LogicalProject($f0=[$0], $f1=[$1])
+- LogicalTableScan(table=[[test-catalog, default, foo]])
== Optimized Physical Plan ==
SortAggregate(isMerge=[true], groupBy=[x], select=[x, Final_collect_set($f1) AS
$f1])
+- Sort(orderBy=[x ASC])
+- Exchange(distribution=[hash[x]])
+- LocalSortAggregate(groupBy=[x], select=[x, Partial_collect_set(y) AS
$f1])
+- Sort(orderBy=[x ASC])
+- TableSourceScan(table=[[test-catalog, default, foo]], fields=[x,
y])
== Optimized Execution Plan ==
SortAggregate(isMerge=[true], groupBy=[x], select=[x, Final_collect_set($f1) AS
$f1])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[x ASC])
+- Exchange(distribution=[hash[x]])
+- LocalSortAggregate(groupBy=[x], select=[x, Partial_collect_set(y)
AS $f1])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[x ASC])
+- TableSourceScan(table=[[test-catalog, default, foo]],
fields=[x, y])
--- turn off `table.exec.hive.native-agg-function.enabled`
== Abstract Syntax Tree ==
LogicalProject(x=[$0], _o__c1=[$1])
+- LogicalAggregate(group=[\{0}], agg#0=[collect_set($1)])
+- LogicalProject($f0=[$0], $f1=[$1])
+- LogicalTableScan(table=[[test-catalog, default, foo]])
== Optimized Physical Plan ==
SortAggregate(isMerge=[true], groupBy=[x], select=[x, Final_collect_set($f1) AS
$f1])
+- Sort(orderBy=[x ASC])
+- Exchange(distribution=[hash[x]])
+- LocalSortAggregate(groupBy=[x], select=[x, Partial_collect_set(y) AS
$f1])
+- Sort(orderBy=[x ASC])
+- TableSourceScan(table=[[test-catalog, default, foo]], fields=[x,
y])
== Optimized Execution Plan ==
SortAggregate(isMerge=[true], groupBy=[x], select=[x, Final_collect_set($f1) AS
$f1])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[x ASC])
+- Exchange(distribution=[hash[x]])
+- LocalSortAggregate(groupBy=[x], select=[x, Partial_collect_set(y)
AS $f1])
+- Exchange(distribution=[forward])
+- Sort(orderBy=[x ASC])
+- TableSourceScan(table=[[test-catalog, default, foo]],
fields=[x, y])
```
!image-2023-02-21-16-31-58-361.png|width=620,height=261!
5. I disable the hashagg to force use sortagg to process all of the test above,
which can see that the result of forcing to close hashagg is the same as the
result of turn on and off`exec.hive.native-agg-function.enabled`, which meets
the expectations
!image-2023-02-21-16-35-46-294.png|width=632,height=392!
Problems:
a. The `exec.hive.native-agg-function.enabled` do not work on sql client. the
hashagg is not chosen on sql client.
!https://intranetproxy.alipay.com/skylark/lark/0/2023/png/83756403/1676952029939-182fa078-3a07-4e45-bdbb-832f7f74c838.png|width=703,height=383,id=u4fc84338!
b. Enable and disable `table.exec.hive.native-agg-function.enabled` get
different result.
!image-2023-02-21-16-28-22-
[GitHub] [flink] FangYongs commented on a diff in pull request #21901: [FLINK-30968][sql-client] Sql client supports dynamic config to open session
FangYongs commented on code in PR #21901:
URL: https://github.com/apache/flink/pull/21901#discussion_r1112748189
##
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/DefaultContextUtils.java:
##
@@ -51,12 +52,17 @@ public static DefaultContext
buildDefaultContext(CliOptions.EmbeddedCliOptions o
} else {
libDirs = Collections.emptyList();
}
-return DefaultContext.load(
-options.getPythonConfiguration(), discoverDependencies(jars,
libDirs), true);
+Configuration pythonConfiguration = options.getPythonConfiguration();
+pythonConfiguration.addAll(
+
ConfigurationUtils.createConfiguration(options.getSessionConfig()));
+return DefaultContext.load(pythonConfiguration,
discoverDependencies(jars, libDirs), true);
Review Comment:
LGTM, DONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] MartijnVisser merged pull request #21969: [FLINK-30948][Formats/AWS] Remove GlueSchemaRegistry Avro and JSON fo…
MartijnVisser merged PR #21969: URL: https://github.com/apache/flink/pull/21969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-31150) Cross team testing
[ https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl reassigned FLINK-31150: - Assignee: Qingsheng Ren > Cross team testing > -- > > Key: FLINK-31150 > URL: https://issues.apache.org/jira/browse/FLINK-31150 > Project: Flink > Issue Type: Sub-task >Reporter: Matthias Pohl >Assignee: Qingsheng Ren >Priority: Major > > For user facing features that go into the release we'd like to ensure they > can actually _be used_ by Flink users. To achieve this the release managers > ensure that an issue for cross team testing is created in the Apache Flink > Jira. This can and should be picked up by other community members to verify > the functionality and usability of the feature. > The issue should contain some entry points which enables other community > members to test it. It should not contain documentation on how to use the > feature as this should be part of the actual documentation. The cross team > tests are performed after the feature freeze. Documentation should be in > place before that. Those tests are manual tests, so do not confuse them with > automated tests. > To sum that up: > * User facing features should be tested by other contributors > * The scope is usability and sanity of the feature > * The feature needs to be already documented > * The contributor creates an issue containing some pointers on how to get > started (e.g. link to the documentation, suggested targets of verification) > * Other community members pick those issues up and provide feedback > * Cross team testing happens right after the feature freeze -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31165) Over Agg: The window rank function without order by error in top N query
P Rohan Kumar created FLINK-31165:
-
Summary: Over Agg: The window rank function without order by error
in top N query
Key: FLINK-31165
URL: https://issues.apache.org/jira/browse/FLINK-31165
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.16.0
Reporter: P Rohan Kumar
{code:java}
val env: StreamExecutionEnvironment =
StreamExecutionEnvironment.getExecutionEnvironment
val tableEnv = StreamTableEnvironment.create(env)
val td = TableDescriptor.forConnector("datagen").option("rows-per-second", "10")
.option("number-of-rows", "10")
.schema(Schema
.newBuilder()
.column("NAME", DataTypes.VARCHAR(2147483647))
.column("ROLLNO", DataTypes.DECIMAL(5, 0))
.column("DOB", DataTypes.DATE())
.column("CLASS", DataTypes.DECIMAL(2, 0))
.column("SUBJECT", DataTypes.VARCHAR(2147483647))
.build())
.build()
val table = tableEnv.from(td)
tableEnv.createTemporaryView("temp_table", table)
val newTable = tableEnv.sqlQuery("select temp_table.*,cast('2022-01-01' as
date) SRC_NO from temp_table")
tableEnv.createTemporaryView("temp_table2", newTable)
val newTable2 = tableEnv.sqlQuery("select * from (select
NAME,ROLLNO,row_number() over (partition by NAME ORDER BY SRC_NO) AS rownum
from temp_table2 a) where rownum <= 1")
tableEnv.toChangelogStream(newTable2).print()
env.execute()
{code}
I am getting the below error if I run the above code.
I have already provided an order by column.
If I change the order by column to some other column, such as "SUBJECT", then
the job runs fine.
{code:java}
Exception in thread "main" java.lang.RuntimeException: Error while applying
rule FlinkLogicalOverAggregateConverter(in:NONE,out:LOGICAL), args
[rel#245:LogicalWindow.NONE.any.None:
0.[NONE].[NONE](input=RelSubset#244,window#0=window(partition {0} rows between
UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()]))]
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:256)
at
org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:58)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:510)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:312)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:59)
at
scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:187)
at
scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:185)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:189)
at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:184)
at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:55)
at
org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:176)
at
org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.doOptimize(StreamCommonSubGraphBasedOptimizer.scala:83)
at
org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:87)
at
org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:315)
at
org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:195)
at
org.apache.flink.table.api.bridge.internal.AbstractStreamTableEnvironmentImpl.toStreamInternal(AbstractStreamTableEnvironmentImpl.java:224)
at
org.apache.flink.table.api.bridge.internal.AbstractStreamTableEnvironmentImpl.toStreamInternal(AbstractStreamTableEnvironmentImpl.java:219)
at
org.apache.flink.table.api.bridge.scala.internal.StreamTableEnvironmentImpl.toChangelogStream(StreamTableEnvironmentImpl.scala:160)
at org.example.OverAggregateBug$.main(OverAggregateBug.scala:39)
at org.example.OverAggregateBug.main(OverAggregateBug.scala)
Caused by: org.apache.flink.table.api.ValidationException: Over Agg: The window
rank function without order by. please re-check the over window statement.
at
org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalOverAggregateConverter.$anonfu
[jira] [Assigned] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-31153:
-
Assignee: Leonard Xu
> Create a release branch
> ---
>
> Key: FLINK-31153
> URL: https://issues.apache.org/jira/browse/FLINK-31153
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Leonard Xu
>Priority: Major
>
> If you are doing a new minor release, you need to update Flink version in the
> following repositories:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Patch releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ cd ./tools
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Flink repository
> Create a branch for the new version that we want to release before updating
> the master branch to the next development version:
> {code:bash}
> $ cd ./tools
> $ releasing/create_snapshot_branch.sh
> $ git checkout master
> $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
> releasing/update_branch_version.sh
> {code}
> In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> as the last entry:
> {code:java}
> // ...
> v1_12("1.12"),
> v1_13("1.13"),
> v1_14("1.14"),
> v1_15("1.15"),
> v1_16("1.16");
> {code}
> The newly created branch and updated {{master}} branch need to be pushed to
> the official repository.
> h4. Flink Docker Repository
> Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
> [apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
> sure that
> [apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
> points to the correct snapshot version; for {{dev-x.y}} it should point to
> {{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
> recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
> After pushing the new minor release branch, as the last step you should also
> update the documentation workflow to also build the documentation for the new
> release branch. Check [Managing
> Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
> on details on how to do that. You may also want to manually trigger a build
> to make the changes visible as soon as possible.
>
>
> h3. Expectations (Minor Version only)
> * Release branch has been created and pushed
> * Cron job has been added on the release branch in
> ([apache-flink:./tools/azure-pipelines/build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
> * Originating branch has the version information updated to the new version
> * New version is added to the
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> enum.
> * Make sure {{flink-docker}} has {{dev-x.y}} branch and docker e2e tests run
> against this branch
> * docs/config.toml has been updated appropriately.
> * The {{dev-x.y}} branch of ({{{}$CURRENT_SNAPSHOT_VERSION{}}}) have been
> created in the Flink Benchmark repo.
> * The {{flink.version}} property of Flink Benchmark repo has been updated to
> the latest snapshot version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31150) Cross team testing
[ https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691465#comment-17691465 ] Matthias Pohl commented on FLINK-31150: --- [~renqs] created FLINK-30926 as an umbrella ticket for the 1.17 release-testing efforts. > Cross team testing > -- > > Key: FLINK-31150 > URL: https://issues.apache.org/jira/browse/FLINK-31150 > Project: Flink > Issue Type: Sub-task >Reporter: Matthias Pohl >Assignee: Qingsheng Ren >Priority: Major > > For user facing features that go into the release we'd like to ensure they > can actually _be used_ by Flink users. To achieve this the release managers > ensure that an issue for cross team testing is created in the Apache Flink > Jira. This can and should be picked up by other community members to verify > the functionality and usability of the feature. > The issue should contain some entry points which enables other community > members to test it. It should not contain documentation on how to use the > feature as this should be part of the actual documentation. The cross team > tests are performed after the feature freeze. Documentation should be in > place before that. Those tests are manual tests, so do not confuse them with > automated tests. > To sum that up: > * User facing features should be tested by other contributors > * The scope is usability and sanity of the feature > * The feature needs to be already documented > * The contributor creates an issue containing some pointers on how to get > started (e.g. link to the documentation, suggested targets of verification) > * Other community members pick those issues up and provide feedback > * Cross team testing happens right after the feature freeze -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30948) Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry from Flink main repo
[ https://issues.apache.org/jira/browse/FLINK-30948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691466#comment-17691466 ] Martijn Visser commented on FLINK-30948: Fixed in: master: 29f009b7e8c714cd5af0626e9725eb8538a4bd0f release-1.17: d58829335557dac6ce428df5a80d4244fccf4491 > Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry > from Flink main repo > --- > > Key: FLINK-30948 > URL: https://issues.apache.org/jira/browse/FLINK-30948 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) > Environment: Remove flink-avro-glue-schema-registry and > flink-json-glue-schema-registry from Flink main repo, along with associated > end-to-end tests >Reporter: Hong Liang Teoh >Assignee: Hong Liang Teoh >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30948) Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry from Flink main repo
[ https://issues.apache.org/jira/browse/FLINK-30948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-30948. -- Resolution: Fixed > Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry > from Flink main repo > --- > > Key: FLINK-30948 > URL: https://issues.apache.org/jira/browse/FLINK-30948 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) > Environment: Remove flink-avro-glue-schema-registry and > flink-json-glue-schema-registry from Flink main repo, along with associated > end-to-end tests >Reporter: Hong Liang Teoh >Assignee: Hong Liang Teoh >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] dmvk commented on a diff in pull request #21981: [FLINK-21450][runtime] Support LocalRecovery by AdaptiveScheduler
dmvk commented on code in PR #21981:
URL: https://github.com/apache/flink/pull/21981#discussion_r1112754334
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java:
##
@@ -794,7 +802,23 @@ public void goToWaitingForResources() {
LOG,
desiredResources,
this.initialResourceAllocationTimeout,
-this.resourceStabilizationTimeout));
+this.resourceStabilizationTimeout,
+null));
+}
+
+@Override
+public void goToWaitingForResources(ExecutionGraph executionGraph) {
Review Comment:
I'm wondering, do we need an extra `goToWaitingForResources` method? 🤔
Can we do something along these lines instead?
```
final ExecutionGraph previousExecutionGraph =
state.as(StateWithExecutionGraph.class)
.map(StateWithExecutionGraph::getExecutionGraph)
.orElse(null);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new minor release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Cron job has been added on the release branch in
([apache-flink:./tools/azure-pipelines/build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
* Originating branch has the version information updated to the new version
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch and docker e2e tests run against this branch
* docs/config.toml has been updated appropriately.
* The {{dev-x.y}} branch of ({{{}$CURRENT_SNAPSHOT_VERSION{}}}) have been
created in the Flink Benchmark repo.
* The {{flink.version}} property of Flink Benchmark repo has been updated to
the latest snapshot version.
was:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] r
[GitHub] [flink-ml] Fanoid commented on a diff in pull request #213: [FLINK-31126] Move classes not depending on Flink runtime from flink-ml-core to flink-ml-servable-core
Fanoid commented on code in PR #213: URL: https://github.com/apache/flink-ml/pull/213#discussion_r1112756657 ## flink-ml-servable-core/pom.xml: ## @@ -0,0 +1,127 @@ + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + 4.0.0 + + +org.apache.flink +flink-ml-parent +2.2-SNAPSHOT + + + flink-ml-servable-core + Flink ML : Servable : Core + + + + + org.apache.flink Review Comment: @lindong28 and I had a discussion about `flink-core` in a previous draft PR [1]. We agreed this change can be put in a follow-up PR. [1] https://github.com/apache/flink-ml/pull/199#discussion_r1090122715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-31092) Hive ITCases fail with OutOfMemoryError
[
https://issues.apache.org/jira/browse/FLINK-31092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu reassigned FLINK-31092:
--
Assignee: luoyuxia
> Hive ITCases fail with OutOfMemoryError
> ---
>
> Key: FLINK-31092
> URL: https://issues.apache.org/jira/browse/FLINK-31092
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: luoyuxia
>Priority: Critical
> Labels: test-stability
> Attachments: VisualVM-FLINK-31092.png
>
>
> We're experiencing a OutOfMemoryError where the heap space reaches the upper
> limit:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46161&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=23142
> {code}
> Feb 15 05:05:14 [INFO] Running
> org.apache.flink.table.catalog.hive.HiveCatalogITCase
> Feb 15 05:05:17 [INFO] java.lang.OutOfMemoryError: Java heap space
> Feb 15 05:05:17 [INFO] Dumping heap to java_pid9669.hprof ...
> Feb 15 05:05:28 [INFO] Heap dump file created [1957090051 bytes in 11.718
> secs]
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.maven.surefire.booter.ForkedBooter.cancelPingScheduler(ForkedBooter.java:209)
> at
> org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:419)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:186)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30951) Release Testing: Verify FLINK-29635 Hive sink should support merge files in batch mode
[ https://issues.apache.org/jira/browse/FLINK-30951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang closed FLINK-30951. - Resolution: Fixed > Release Testing: Verify FLINK-29635 Hive sink should support merge files in > batch mode > -- > > Key: FLINK-30951 > URL: https://issues.apache.org/jira/browse/FLINK-30951 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: luoyuxia >Assignee: Shengkai Fang >Priority: Blocker > Fix For: 1.17.0 > > Attachments: screenshot-1.png > > > The issue aims to verfiy FLINK-29635. > Please verify in batch mode, the document is in > [https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/hive/hive_read_write/#file-compaction]: > > 1: enable auto-compaction, write some data to a Hive table which results in > the average size of files is less than compaction.small-files.avg-size(16MB > by default), verfiy these files should be merged. > 2: enable auto-compaction, set compaction.small-files.avg-size to a smaller > values, then write some data to a Hive table which results in the average > size of files is greater thant the compaction.small-files.avg-size, verfiy > these files shouldn't be merged. > 3. set sink.parallelism manually, check the parallelism of the compact > operator is equal to sink.parallelism. > 4. set compaction.parallelism manually, check the parallelism of the compact > operator is equal to compaction.parallelism. > 5. set compaction.file-size, check the size of the each target file merged is > about the `compaction.file-size`. > > We shoud verify it with writing non-partitioned table, static partition > table, dynamic partition table. > We can find the example sql for how to create & write hive table in the > codebase > [HiveTableCompactSinkITCase|[https://github.com/apache/flink/blob/0915c9850d861165e283acc0f60545cd836f0567/flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkITCase.java]]. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #213: [FLINK-31126] Move classes not depending on Flink runtime from flink-ml-core to flink-ml-servable-core
zhipeng93 commented on code in PR #213: URL: https://github.com/apache/flink-ml/pull/213#discussion_r1112772912 ## flink-ml-servable-core/pom.xml: ## @@ -0,0 +1,127 @@ + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + 4.0.0 + + +org.apache.flink +flink-ml-parent +2.2-SNAPSHOT + + + flink-ml-servable-core + Flink ML : Servable : Core + + + + + org.apache.flink Review Comment: Thanks for the explanation. I agree it is a reasonable solution in the long term. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31059) Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by native implementation
[ https://issues.apache.org/jira/browse/FLINK-31059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691472#comment-17691472 ] dalongliu commented on FLINK-31059: --- [~miamiaoxyz] Thanks for your verify, for the problem, I will see it. > Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by > native implementation > - > > Key: FLINK-31059 > URL: https://issues.apache.org/jira/browse/FLINK-31059 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Assignee: miamiaoxyz >Priority: Blocker > Fix For: 1.17.0 > > Attachments: image-2023-02-21-15-45-48-226.png, > image-2023-02-21-15-46-13-966.png, image-2023-02-21-15-47-54-043.png, > image-2023-02-21-15-49-58-854.png, image-2023-02-21-15-59-44-470.png, > image-2023-02-21-16-28-22-038.png, image-2023-02-21-16-29-42-983.png, > image-2023-02-21-16-31-58-361.png, image-2023-02-21-16-35-46-294.png > > > This task aims to verify > [FLINK-29717|https://issues.apache.org/jira/browse/FLINK-29717] which > improves the hive udaf performance. > As the document [PR|https://github.com/apache/flink/pull/21789] description, > please veriy: > 1. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions separately in the query to verify if the > hash-agg strategy is chosen via plan, and verify if the data results are the > same as when the option `table.exec.hive.native-agg-function.enabled` is > disabled. > 2. Enabling the option `table.exec.hive.native-agg-function.enabled`, combine > sum/count/avg/min/max functions in query, verify if the hash-agg strategy is > chosen via plan, and verify if the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. > 3. Enabling the option `table.exec.hive.native-agg-function.enabled`, count > or max array&struct and other complex types in query, verify whether the > sort-agg strategy is chosen via plan, verify whether the data result is the > same as when option `table.exec.hive.native-agg-function.enabled` is disabled. > 4. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count and first_value/last_value functions in the query simultaneously, > verify that the sort-agg strategy is chosen via plan, verify that the data is > the same as when option `table.exec.hive.native-agg-function.enabled` is > disabled. > 5. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions in the query and open sort-agg strategy > forcibly, verify that the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] zhipeng93 commented on pull request #213: [FLINK-31126] Move classes not depending on Flink runtime from flink-ml-core to flink-ml-servable-core
zhipeng93 commented on PR #213: URL: https://github.com/apache/flink-ml/pull/213#issuecomment-1438119732 Thanks for the PR. LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 merged pull request #213: [FLINK-31126] Move classes not depending on Flink runtime from flink-ml-core to flink-ml-servable-core
zhipeng93 merged PR #213: URL: https://github.com/apache/flink-ml/pull/213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-31059) Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by native implementation
[ https://issues.apache.org/jira/browse/FLINK-31059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu closed FLINK-31059. - Resolution: Fixed > Release Testing: Verify FLINK-29717 Supports hive udaf such as sum/count by > native implementation > - > > Key: FLINK-31059 > URL: https://issues.apache.org/jira/browse/FLINK-31059 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Assignee: miamiaoxyz >Priority: Blocker > Fix For: 1.17.0 > > Attachments: image-2023-02-21-15-45-48-226.png, > image-2023-02-21-15-46-13-966.png, image-2023-02-21-15-47-54-043.png, > image-2023-02-21-15-49-58-854.png, image-2023-02-21-15-59-44-470.png, > image-2023-02-21-16-28-22-038.png, image-2023-02-21-16-29-42-983.png, > image-2023-02-21-16-31-58-361.png, image-2023-02-21-16-35-46-294.png > > > This task aims to verify > [FLINK-29717|https://issues.apache.org/jira/browse/FLINK-29717] which > improves the hive udaf performance. > As the document [PR|https://github.com/apache/flink/pull/21789] description, > please veriy: > 1. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions separately in the query to verify if the > hash-agg strategy is chosen via plan, and verify if the data results are the > same as when the option `table.exec.hive.native-agg-function.enabled` is > disabled. > 2. Enabling the option `table.exec.hive.native-agg-function.enabled`, combine > sum/count/avg/min/max functions in query, verify if the hash-agg strategy is > chosen via plan, and verify if the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. > 3. Enabling the option `table.exec.hive.native-agg-function.enabled`, count > or max array&struct and other complex types in query, verify whether the > sort-agg strategy is chosen via plan, verify whether the data result is the > same as when option `table.exec.hive.native-agg-function.enabled` is disabled. > 4. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count and first_value/last_value functions in the query simultaneously, > verify that the sort-agg strategy is chosen via plan, verify that the data is > the same as when option `table.exec.hive.native-agg-function.enabled` is > disabled. > 5. Enabling the option `table.exec.hive.native-agg-function.enabled`, use the > sum/count/avg/min/max functions in the query and open sort-agg strategy > forcibly, verify that the data results are the same as when option > `table.exec.hive.native-agg-function.enabled` is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] lindong28 commented on pull request #213: [FLINK-31126] Move classes not depending on Flink runtime from flink-ml-core to flink-ml-servable-core
lindong28 commented on PR #213: URL: https://github.com/apache/flink-ml/pull/213#issuecomment-1438122258 Thank you both for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new minor release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Cron job has been added on the release branch in
([apache-flink:./tools/azure-pipelines/build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
* Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* Originating branch has the version information updated to the new version
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch and docker e2e tests run against this branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
*
[apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
* The {{dev-x.y}} branch of ({{{}$CURRENT_SNAPSHOT_VERSION{}}}) have been
created in the Flink Benchmark repo.
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
was:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new valu
[jira] [Updated] (FLINK-31147) Create a new version in JIRA
[
https://issues.apache.org/jira/browse/FLINK-31147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31147:
--
Description:
When contributors resolve an issue in JIRA, they are tagging it with a release
that will contain their changes. With the release currently underway, new
issues should be resolved against a subsequent future release. Therefore, you
should create a release item for this subsequent release, as follows:
# In JIRA, navigate to the [Flink > Administration >
Versions|https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions].
# Add a new release: choose the next minor version number compared to the one
currently underway, select today’s date as the Start Date, and choose Add.
(Note: Only PMC members have access to the project administration. If you do
not have access, ask on the mailing list for assistance.)
h3. Expectations
* The new version should be listed in the dropdown menu of {{fixVersion}} or
{{affectedVersion}} under "unreleased versions" when creating a new Jira issue.
was:
When contributors resolve an issue in JIRA, they are tagging it with a release
that will contain their changes. With the release currently underway, new
issues should be resolved against a subsequent future release. Therefore, you
should create a release item for this subsequent release, as follows:
# In JIRA, navigate to the [Flink > Administration >
Versions|https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions].
# Add a new release: choose the next minor version number compared to the one
currently underway, select today’s date as the Start Date, and choose Add.
(Note: Only PMC members have access to the project administration. If you do
not have access, ask on the mailing list for assistance.)
> Create a new version in JIRA
>
>
> Key: FLINK-31147
> URL: https://issues.apache.org/jira/browse/FLINK-31147
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>
> When contributors resolve an issue in JIRA, they are tagging it with a
> release that will contain their changes. With the release currently underway,
> new issues should be resolved against a subsequent future release. Therefore,
> you should create a release item for this subsequent release, as follows:
> # In JIRA, navigate to the [Flink > Administration >
> Versions|https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions].
> # Add a new release: choose the next minor version number compared to the
> one currently underway, select today’s date as the Start Date, and choose Add.
> (Note: Only PMC members have access to the project administration. If you do
> not have access, ask on the mailing list for assistance.)
>
>
> h3. Expectations
> * The new version should be listed in the dropdown menu of {{fixVersion}} or
> {{affectedVersion}} under "unreleased versions" when creating a new Jira
> issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31149) Review and update documentation
[
https://issues.apache.org/jira/browse/FLINK-31149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31149:
--
Description:
There are a few pages in the documentation that need to be reviewed and updated
for each release.
* Ensure that there exists a release notes page for each non-bugfix release
(e.g., 1.5.0) in {{{}./docs/release-notes/{}}}, that it is up-to-date, and
linked from the start page of the documentation.
* Upgrading Applications and Flink Versions:
[https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html]
* ...
h3. Expectations
* Update upgrade compatibility table
([apache-flink:./docs/content/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content/docs/ops/upgrading.md#compatibility-table]
and
[apache-flink:./docs/content.zh/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content.zh/docs/ops/upgrading.md#compatibility-table]).
* Update [Release Overview in
Confluence|https://cwiki.apache.org/confluence/display/FLINK/Release+Management+and+Feature+Plan]
* (minor only) The documentation for the new major release is visible under
[https://nightlies.apache.org/flink/flink-docs-release-$SHORT_RELEASE_VERSION]
(after at least one [doc
build|https://github.com/apache/flink/actions/workflows/docs.yml] succeeded).
* (minor only) The documentation for the new major release does not contain
"-SNAPSHOT" in its version title, and all links refer to the corresponding
version docs instead of {{{}master{}}}.
was:
There are a few pages in the documentation that need to be reviewed and updated
for each release.
* Ensure that there exists a release notes page for each non-bugfix release
(e.g., 1.5.0) in {{{}./docs/release-notes/{}}}, that it is up-to-date, and
linked from the start page of the documentation.
* Upgrading Applications and Flink Versions:
[https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html]
* ...
h3. Expectations
* Update upgrade compatibility table
([apache-flink:./docs/content/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content/docs/ops/upgrading.md#compatibility-table]
and
[apache-flink:./docs/content.zh/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content.zh/docs/ops/upgrading.md#compatibility-table]).
* Update [Release Overview in
Confluence|https://cwiki.apache.org/confluence/display/FLINK/Release+Management+and+Feature+Plan]
* (major only) The documentation for the new major release is visible under
https://nightlies.apache.org/flink/flink-docs-release-$SHORT_RELEASE_VERSION
(after at least one [doc
build|https://github.com/apache/flink/actions/workflows/docs.yml] succeeded).
* (major only) The documentation for the new major release does not contain
"-SNAPSHOT" in its version title, and all links refer to the corresponding
version docs instead of {{master}}.
> Review and update documentation
> ---
>
> Key: FLINK-31149
> URL: https://issues.apache.org/jira/browse/FLINK-31149
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Priority: Major
>
> There are a few pages in the documentation that need to be reviewed and
> updated for each release.
> * Ensure that there exists a release notes page for each non-bugfix release
> (e.g., 1.5.0) in {{{}./docs/release-notes/{}}}, that it is up-to-date, and
> linked from the start page of the documentation.
> * Upgrading Applications and Flink Versions:
> [https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html]
> * ...
>
>
> h3. Expectations
> * Update upgrade compatibility table
> ([apache-flink:./docs/content/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content/docs/ops/upgrading.md#compatibility-table]
> and
> [apache-flink:./docs/content.zh/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content.zh/docs/ops/upgrading.md#compatibility-table]).
> * Update [Release Overview in
> Confluence|https://cwiki.apache.org/confluence/display/FLINK/Release+Management+and+Feature+Plan]
> * (minor only) The documentation for the new major release is visible under
> [https://nightlies.apache.org/flink/flink-docs-release-$SHORT_RELEASE_VERSION]
> (after at least one [doc
> build|https://github.com/apache/flink/actions/workflows/docs.yml] succeeded).
> * (minor only) The documentation for the new major release does not contain
> "-SNAPSHOT" in its version title, and all links refer to the corresponding
> version docs instead of {{{}master{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] reswqa commented on a diff in pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
reswqa commented on code in PR #21977:
URL: https://github.com/apache/flink/pull/21977#discussion_r1112641083
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkITCase.java:
##
@@ -42,17 +44,18 @@
import static org.assertj.core.api.Assertions.assertThat;
/** IT case for Hive table compaction in batch mode. */
-public class HiveTableCompactSinkITCase {
+@ExtendWith(TestLoggerExtension.class)
+class HiveTableCompactSinkITCase {
@RegisterExtension
-private static final MiniClusterExtension MINI_CLUSTER = new
MiniClusterExtension();
+public static final MiniClusterExtension MINI_CLUSTER = new
MiniClusterExtension();
Review Comment:
TBH, The reason I changed it is only to follow the usage in the
`MiniClusterExtension` class
[javadoc](https://github.com/apache/flink/blob/bacdc326b58749924acbd8921d63eda06663a225/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/junit5/MiniClusterExtension.java#L70).
At the same time, in the `architecture check` of IT case, we can see similar
[requirement](https://github.com/apache/flink/blob/bacdc326b58749924acbd8921d63eda06663a225/flink-architecture-tests/flink-architecture-tests-test/src/main/java/org/apache/flink/architecture/rules/ITCaseRules.java#L67).
Unfortunately, I looked at the actual check logic and didn't seem to require
it to be public, which really confused me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] FangYongs commented on pull request #21956: [FLINK-31113] Support AND filter push down in orc
FangYongs commented on PR #21956: URL: https://github.com/apache/flink/pull/21956#issuecomment-1438133167 Thanks @libenchao , I have updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #214: [FLINK-31127] Add public API classes for FLIP-289
lindong28 commented on PR #214: URL: https://github.com/apache/flink-ml/pull/214#issuecomment-1438134181 @jiangxin369 Can you help review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31152) Setup for the executing release manager
[
https://issues.apache.org/jira/browse/FLINK-31152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31152:
--
Description:
h4. GPG Key
You need to have a GPG key to sign the release artifacts. Please be aware of
the ASF-wide [release signing
guidelines|https://www.apache.org/dev/release-signing.html]. If you don’t have
a GPG key associated with your Apache account, please create one according to
the guidelines.
Determine your Apache GPG Key and Key ID, as follows:
{code:java}
$ gpg --list-keys
{code}
This will list your GPG keys. One of these should reflect your Apache account,
for example:
{code:java}
--
pub 2048R/845E6689 2016-02-23
uid Nomen Nescio
sub 2048R/BA4D50BE 2016-02-23
{code}
In the example above, the key ID is the 8-digit hex string in the {{pub}} line:
{{{}845E6689{}}}.
Now, add your Apache GPG key to the Flink’s {{KEYS}} file in the [Apache Flink
release KEYS file|https://dist.apache.org/repos/dist/release/flink/KEYS]
repository at [dist.apache.org|http://dist.apache.org/]. Follow the
instructions listed at the top of these files. (Note: Only PMC members have
write access to the release repository. If you end up getting 403 errors ask on
the mailing list for assistance.)
Configure {{git}} to use this key when signing code by giving it your key ID,
as follows:
{code:java}
$ git config --global user.signingkey 845E6689
{code}
You may drop the {{--global}} option if you’d prefer to use this key for the
current repository only.
You may wish to start {{gpg-agent}} to unlock your GPG key only once using your
passphrase. Otherwise, you may need to enter this passphrase hundreds of times.
The setup for {{gpg-agent}} varies based on operating system, but may be
something like this:
{code:bash}
$ eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info)
$ export GPG_TTY=$(tty)
$ export GPG_AGENT_INFO
{code}
h4. Access to Apache Nexus repository
Configure access to the [Apache Nexus
repository|https://repository.apache.org/], which enables final deployment of
releases to the Maven Central Repository.
# You log in with your Apache account.
# Confirm you have appropriate access by finding {{org.apache.flink}} under
{{{}Staging Profiles{}}}.
# Navigate to your {{Profile}} (top right drop-down menu of the page).
# Choose {{User Token}} from the dropdown, then click {{{}Access User
Token{}}}. Copy a snippet of the Maven XML configuration block.
# Insert this snippet twice into your global Maven {{settings.xml}} file,
typically {{{}${HOME}/.m2/settings.xml{}}}. The end result should look like
this, where {{TOKEN_NAME}} and {{TOKEN_PASSWORD}} are your secret tokens:
{code:xml}
apache.releases.https
TOKEN_NAME
TOKEN_PASSWORD
apache.snapshots.https
TOKEN_NAME
TOKEN_PASSWORD
{code}
h4. Website development setup
Get ready for updating the Flink website by following the [website development
instructions|https://flink.apache.org/contributing/improve-website.html].
h4. GNU Tar Setup for Mac (Skip this step if you are not using a Mac)
The default tar application on Mac does not support GNU archive format and
defaults to Pax. This bloats the archive with unnecessary metadata that can
result in additional files when decompressing (see [1.15.2-RC2 vote
thread|https://lists.apache.org/thread/mzbgsb7y9vdp9bs00gsgscsjv2ygy58q]).
Install gnu-tar and create a symbolic link to use in preference of the default
tar program.
{code:bash}
$ brew install gnu-tar
$ ln -s /usr/local/bin/gtar /usr/local/bin/tar
$ which tar
{code}
h3. Expectations
* Release Manager’s GPG key is published to
[dist.apache.org|http://dist.apache.org/]
* Release Manager’s GPG key is configured in git configuration
* Release Manager's GPG key is configured as the default gpg key.
* Release Manager has {{org.apache.flink}} listed under Staging Profiles in
Nexus
* Release Manager’s Nexus User Token is configured in settings.xml
was:
h4. GPG Key
You need to have a GPG key to sign the release artifacts. Please be aware of
the ASF-wide [release signing
guidelines|https://www.apache.org/dev/release-signing.html]. If you don’t have
a GPG key associated with your Apache account, please create one according to
the guidelines.
Determine your Apache GPG Key and Key ID, as follows:
{code:java}
$ gpg --list-keys
{code}
This will list your GPG keys. One of these should reflect your Apache account,
for example:
{code:java}
--
pub 2048R/845E6689 2016-02-23
uid Nomen Nescio
sub 2048R/BA4D50BE 2016-02-23
{code}
In the example above, the key ID is the 8-digit hex string in the {{pub}} line:
{{{}845E6689{}}}.
Now, add your Apache GPG key to the Flink’s {{KEYS}} file in the [Apache Flink
[jira] [Updated] (FLINK-31144) Slow scheduling on large-scale batch jobs
[ https://issues.apache.org/jira/browse/FLINK-31144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Tournay updated FLINK-31144: --- Attachment: image-2023-02-21-10-29-49-388.png > Slow scheduling on large-scale batch jobs > -- > > Key: FLINK-31144 > URL: https://issues.apache.org/jira/browse/FLINK-31144 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Julien Tournay >Priority: Major > Attachments: flink-1.17-snapshot-1676473798013.nps, > image-2023-02-21-10-29-49-388.png > > > When executing a complex job graph at high parallelism > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` can > get slow and cause long pauses where the JobManager becomes unresponsive and > all the taskmanagers just wait. I've attached a VisualVM snapshot to > illustrate the problem.[^flink-1.17-snapshot-1676473798013.nps] > At Spotify we have complex jobs where this issue can cause batch "pause" of > 40+ minutes and make the overall execution 30% slower or more. > More importantly this prevent us from running said jobs on larger cluster as > adding resources to the cluster worsen the issue. > We have successfully tested a modified Flink version where > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` was > completely commented and simply returns an empty collection and confirmed it > solves the issue. > In the same spirit as a recent change > ([https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L98-L102)] > there could be a mechanism in place to detect when Flink run into this > specific issue and just skip the call to `getInputLocationFutures` > [https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L105-L108.] > I'm not familiar enough with the internals of Flink to propose a more > advanced fix, however it seems like a configurable threshold on the number of > consumer vertices above which the preferred location is not computed would > do. If this solution is good enough, I'd be happy to submit a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #212: [FLINK-31125] Flink ML benchmark framework should minimize the source operator overhead
zhipeng93 commented on code in PR #212:
URL: https://github.com/apache/flink-ml/pull/212#discussion_r1112795550
##
flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark/datagenerator/common/DenseVectorArrayGenerator.java:
##
@@ -42,7 +42,7 @@ protected RowGenerator[] getRowGenerators() {
return new RowGenerator[] {
new RowGenerator(getNumValues(), getSeed()) {
@Override
-protected Row nextRow() {
+protected Row getRow() {
Review Comment:
Reusing one row to simulate the input may cause the benchmark result not
stable, since processing two different rows may take different time.
The two optimizations introduced in this PR give us 12% speedup. Is there a
breakdown analysis of this two optimization?
If the data generation cost is indeed high, can we sample a fixed number of
rows and reuse it later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #212: [FLINK-31125] Flink ML benchmark framework should minimize the source operator overhead
zhipeng93 commented on code in PR #212:
URL: https://github.com/apache/flink-ml/pull/212#discussion_r1112795550
##
flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark/datagenerator/common/DenseVectorArrayGenerator.java:
##
@@ -42,7 +42,7 @@ protected RowGenerator[] getRowGenerators() {
return new RowGenerator[] {
new RowGenerator(getNumValues(), getSeed()) {
@Override
-protected Row nextRow() {
+protected Row getRow() {
Review Comment:
Reusing one row to simulate the input may cause the benchmark result not
stable, since processing two different rows may take different time.
The two optimizations introduced in this PR give us 12% speedup. Is there a
breakdown analysis of this two optimization?
If the data generation cost is indeed high, can we sample a fixed number of
rows and reuse it in the evalution?
##
flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark/datagenerator/common/DenseVectorArrayGenerator.java:
##
@@ -42,7 +42,7 @@ protected RowGenerator[] getRowGenerators() {
return new RowGenerator[] {
new RowGenerator(getNumValues(), getSeed()) {
@Override
-protected Row nextRow() {
+protected Row getRow() {
Review Comment:
Reusing one row to simulate the input may cause the benchmark result not
stable, since processing two different rows may take different time.
The two optimizations introduced in this PR give us 12% speedup. Is there a
breakdown analysis of this two optimization?
If the data generation cost is indeed high, can we sample a fixed number of
rows and reuse it in the evaluation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31144) Slow scheduling on large-scale batch jobs
[
https://issues.apache.org/jira/browse/FLINK-31144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691474#comment-17691474
]
Julien Tournay commented on FLINK-31144:
Hi [~huwh],
Thank you for the quick reply :)
{quote}when the topology is complex.
{quote}
Indeed. For the issue to be noticeable, the jobgraph has to be fairly complex,
feature all-to-all distributions and execute with a high parallelism.
{quote}1. Is the slow scheduling or the scheduled result of location preferred
make your job slow?
{quote}
Yes it very much does. We have a job that takes ~2h30 (after many many tweaks
to get the best possible perf.). It's impossible to get it to run in less time
because adding more taskmanagers make the scheduling slow and overall the
execution gets longer. Removing preferred location makes it possible to run it
in less that 2h (We're aiming at ~1h45min).
{quote}2. "we have complex jobs where this issue can cause batch "pause" of 40+
minutes" What does "pause" meaning? Is the getPreferredLocationsBasedOnInputs
take more than 40+ minutes?
{quote}
By "pause" I mean that at the beginning of the execution, the taskmanagers will
wait for the JobManager for ~40min and then will start processing. With Flink
1.17 and no preferred location, the "pause" is down to ~5min.
I should also mention the JM is very unresponsive and the web console struggles
the show anything.
{quote}Could you provide the topology of the complex job.
{quote}
I can but not sure what format to use. The graph is quite big and a simple
screenshot is unreadable: !image-2023-02-21-10-29-49-388.png!
I can maybe share the archived execution json file (~500Mb) if that's helpful ?
> Slow scheduling on large-scale batch jobs
> --
>
> Key: FLINK-31144
> URL: https://issues.apache.org/jira/browse/FLINK-31144
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Reporter: Julien Tournay
>Priority: Major
> Attachments: flink-1.17-snapshot-1676473798013.nps,
> image-2023-02-21-10-29-49-388.png
>
>
> When executing a complex job graph at high parallelism
> `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` can
> get slow and cause long pauses where the JobManager becomes unresponsive and
> all the taskmanagers just wait. I've attached a VisualVM snapshot to
> illustrate the problem.[^flink-1.17-snapshot-1676473798013.nps]
> At Spotify we have complex jobs where this issue can cause batch "pause" of
> 40+ minutes and make the overall execution 30% slower or more.
> More importantly this prevent us from running said jobs on larger cluster as
> adding resources to the cluster worsen the issue.
> We have successfully tested a modified Flink version where
> `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` was
> completely commented and simply returns an empty collection and confirmed it
> solves the issue.
> In the same spirit as a recent change
> ([https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L98-L102)]
> there could be a mechanism in place to detect when Flink run into this
> specific issue and just skip the call to `getInputLocationFutures`
> [https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L105-L108.]
> I'm not familiar enough with the internals of Flink to propose a more
> advanced fix, however it seems like a configurable threshold on the number of
> consumer vertices above which the preferred location is not computed would
> do. If this solution is good enough, I'd be happy to submit a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new minor release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Cron job has been added on the release branch in
([apache-flink:./tools/azure-pipelines/build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
* Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* {{master}} branch has the version information updated to the new version
(check pom.xml files and
*
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum)
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch and docker e2e tests run against this branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
*
[apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
was:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VE
[GitHub] [flink-ml] zhipeng93 closed pull request #176: Bump jackson-databind from 2.12.6.1 to 2.12.7.1
zhipeng93 closed pull request #176: Bump jackson-databind from 2.12.6.1 to 2.12.7.1 URL: https://github.com/apache/flink-ml/pull/176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] dependabot[bot] commented on pull request #176: Bump jackson-databind from 2.12.6.1 to 2.12.7.1
dependabot[bot] commented on PR #176: URL: https://github.com/apache/flink-ml/pull/176#issuecomment-1438157269 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. If you change your mind, just re-open this PR and I'll resolve any conflicts on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dmvk commented on a diff in pull request #21981: [FLINK-21450][runtime] Support LocalRecovery by AdaptiveScheduler
dmvk commented on code in PR #21981:
URL: https://github.com/apache/flink/pull/21981#discussion_r1112810683
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/allocator/StateLocalitySlotAssigner.java:
##
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.adaptive.allocator;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.executiongraph.ExecutionGraph;
+import org.apache.flink.runtime.executiongraph.ExecutionJobVertex;
+import org.apache.flink.runtime.executiongraph.ExecutionVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import
org.apache.flink.runtime.scheduler.adaptive.allocator.SlotSharingSlotAllocator.ExecutionSlotSharingGroup;
+import
org.apache.flink.runtime.scheduler.adaptive.allocator.SlotSharingSlotAllocator.ExecutionSlotSharingGroupAndSlot;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyGroupRangeAssignment;
+import org.apache.flink.util.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.stream.StreamSupport;
+
+import static java.util.function.Function.identity;
+import static java.util.stream.Collectors.toMap;
+
+/** A {@link SlotAssigner} that assigns slots based on the number of local key
groups. */
+@Internal
+public class StateLocalitySlotAssigner implements SlotAssigner {
+
+private static class AllocationScore implements
Comparable {
+
+private final String group;
+private final AllocationID allocationId;
+
+public AllocationScore(String group, AllocationID allocationId, int
score) {
+this.group = group;
+this.allocationId = allocationId;
+this.score = score;
+}
+
+private final int score;
+
+public String getGroup() {
+return group;
+}
+
+public AllocationID getAllocationId() {
+return allocationId;
+}
+
+public int getScore() {
+return score;
+}
+
+@Override
+public int compareTo(StateLocalitySlotAssigner.AllocationScore other) {
+int result = Integer.compare(score, other.score);
+if (result != 0) {
+return result;
+}
+result = other.allocationId.compareTo(allocationId);
+if (result != 0) {
+return result;
+}
+return other.group.compareTo(group);
+}
+}
+
+private final Map> locality;
+private final Map maxParallelism;
+
+public StateLocalitySlotAssigner(ExecutionGraph archivedExecutionGraph) {
+this(
+calculateLocalKeyGroups(archivedExecutionGraph),
+StreamSupport.stream(
+
archivedExecutionGraph.getVerticesTopologically().spliterator(),
+false)
+.collect(
+toMap(
+ExecutionJobVertex::getJobVertexId,
+
ExecutionJobVertex::getMaxParallelism)));
+}
+
+public StateLocalitySlotAssigner(
+Map> locality,
+Map maxParallelism) {
+this.locality = locality;
+this.maxParallelism = maxParallelism;
+}
+
+@Override
+public AssignmentResult assignSlots(
+Collection slots,
Collection groups) {
+
+final Map parallelism = new HashMap<>();
+groups.forEach(
+group ->
+group.getContainedExecutionVertices()
+.forEach(
+evi ->
+parallelism.merge(
+
[GitHub] [flink] dmvk commented on a diff in pull request #21981: [FLINK-21450][runtime] Support LocalRecovery by AdaptiveScheduler
dmvk commented on code in PR #21981:
URL: https://github.com/apache/flink/pull/21981#discussion_r1112815613
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/allocator/SlotSharingSlotAllocator.java:
##
@@ -121,16 +133,13 @@ public Optional
determineParallelism(
slotSharingGroupParallelism.get(
slotSharingGroup.getSlotSharingGroupId()));
-final Iterable
sharedSlotToVertexAssignment =
+final List sharedSlotToVertexAssignment
=
createExecutionSlotSharingGroups(vertexParallelism);
-for (ExecutionSlotSharingGroup executionSlotSharingGroup :
-sharedSlotToVertexAssignment) {
-final SlotInfo slotInfo = slotIterator.next();
-
-assignments.add(
-new
ExecutionSlotSharingGroupAndSlot(executionSlotSharingGroup, slotInfo));
-}
+SlotAssigner.AssignmentResult result =
+slotAssigner.assignSlots(freeSlots,
sharedSlotToVertexAssignment);
+assignments.addAll(result.assignments);
+freeSlots = result.remainingSlots;
Review Comment:
🤔 Is there a reason to run this within the loop at all? Can we simply move
this to after the loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-31142) Some queries lead to abrupt sql client close
[
https://issues.apache.org/jira/browse/FLINK-31142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-31142:
Priority: Blocker (was: Major)
> Some queries lead to abrupt sql client close
>
>
> Key: FLINK-31142
> URL: https://issues.apache.org/jira/browse/FLINK-31142
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: Sergey Nuyanzin
>Priority: Blocker
>
> Although the behavior has been changed in 1.17.0, I'm not sure whether it is
> a blocker or not, since in both cases it is invalid query.
> The difference in the behavior is that before 1.17.0
> a query like
> {code:sql}
> select /* multiline comment;
> {code}
> fails to execute and sql client prompts to submit another query.
> In 1.17.0 it shuts down the session failing with
> {noformat}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Could not read from command line.
> at
> org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:205)
> at
> org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:168)
> at
> org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:113)
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error at
> line 1, column 29. Encountered: after : ""
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:26752)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.scan(SqlCommandParserImpl.java:89)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.next(SqlCommandParserImpl.java:81)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.checkIncompleteStatement(SqlCommandParserImpl.java:141)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.getCommand(SqlCommandParserImpl.java:111)
> at
> org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement(SqlCommandParserImpl.java:52)
> at
> org.apache.flink.table.client.cli.parser.SqlMultiLineParser.parse(SqlMultiLineParser.java:82)
> at
> org.jline.reader.impl.LineReaderImpl.acceptLine(LineReaderImpl.java:2964)
> at
> org.jline.reader.impl.LineReaderImpl$1.apply(LineReaderImpl.java:3778)
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:679)
> at
> org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:183)
> ... 6 more
> Shutting down the session...
> done.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31142) Some queries lead to abrupt sql client close
[
https://issues.apache.org/jira/browse/FLINK-31142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-31142:
Description:
Although the behavior has been changed in 1.17.0, I'm not sure whether it is a
blocker or not, since in both cases it is invalid query.
I put it to blocker just because of regression.
The difference in the behavior is that before 1.17.0
a query like
{code:sql}
select /* multiline comment;
{code}
fails to execute and sql client prompts to submit another query.
In 1.17.0 it shuts down the session failing with
{noformat}
Exception in thread "main" org.apache.flink.table.client.SqlClientException:
Could not read from command line.
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:205)
at
org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:168)
at
org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:113)
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
at
org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error at
line 1, column 29. Encountered: after : ""
at
org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:26752)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.scan(SqlCommandParserImpl.java:89)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.next(SqlCommandParserImpl.java:81)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.checkIncompleteStatement(SqlCommandParserImpl.java:141)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.getCommand(SqlCommandParserImpl.java:111)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement(SqlCommandParserImpl.java:52)
at
org.apache.flink.table.client.cli.parser.SqlMultiLineParser.parse(SqlMultiLineParser.java:82)
at
org.jline.reader.impl.LineReaderImpl.acceptLine(LineReaderImpl.java:2964)
at
org.jline.reader.impl.LineReaderImpl$1.apply(LineReaderImpl.java:3778)
at
org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:679)
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:183)
... 6 more
Shutting down the session...
done.
{noformat}
was:
Although the behavior has been changed in 1.17.0, I'm not sure whether it is a
blocker or not, since in both cases it is invalid query.
The difference in the behavior is that before 1.17.0
a query like
{code:sql}
select /* multiline comment;
{code}
fails to execute and sql client prompts to submit another query.
In 1.17.0 it shuts down the session failing with
{noformat}
Exception in thread "main" org.apache.flink.table.client.SqlClientException:
Could not read from command line.
at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:205)
at
org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:168)
at
org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:113)
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
at
org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error at
line 1, column 29. Encountered: after : ""
at
org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:26752)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.scan(SqlCommandParserImpl.java:89)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl$TokenIterator.next(SqlCommandParserImpl.java:81)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.checkIncompleteStatement(SqlCommandParserImpl.java:141)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.getCommand(SqlCommandParserImpl.java:111)
at
org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement(SqlCommandParserImpl.java:52)
at
org.apache.flink.table.client.cli.parser.SqlMultiLineParser.parse(SqlMultiLineParser.java:82)
at
org.jline.reader.impl.LineReaderImpl.acceptLine(LineReaderImpl.java:2964)
at
org.jline.reader.impl.LineReaderImpl$1.apply(LineRea
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new minor release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* {{master}} branch has the version information updated to the new version
(check pom.xml files and
*
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum)
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch and docker e2e tests run against this branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
*
[apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
was:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/
[jira] [Commented] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691491#comment-17691491
]
Jing Ge commented on FLINK-31153:
-
Make sense. Thanks [~mapohl] for the clarification. Looking forward to the
structure Flink Release Task Collection Template - FRACT ? ;)
> Create a release branch
> ---
>
> Key: FLINK-31153
> URL: https://issues.apache.org/jira/browse/FLINK-31153
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Leonard Xu
>Priority: Major
>
> If you are doing a new minor release, you need to update Flink version in the
> following repositories:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Patch releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ cd ./tools
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Flink repository
> Create a branch for the new version that we want to release before updating
> the master branch to the next development version:
> {code:bash}
> $ cd ./tools
> $ releasing/create_snapshot_branch.sh
> $ git checkout master
> $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
> releasing/update_branch_version.sh
> {code}
> In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> as the last entry:
> {code:java}
> // ...
> v1_12("1.12"),
> v1_13("1.13"),
> v1_14("1.14"),
> v1_15("1.15"),
> v1_16("1.16");
> {code}
> The newly created branch and updated {{master}} branch need to be pushed to
> the official repository.
> h4. Flink Docker Repository
> Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
> [apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
> sure that
> [apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
> points to the correct snapshot version; for {{dev-x.y}} it should point to
> {{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
> recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
> After pushing the new minor release branch, as the last step you should also
> update the documentation workflow to also build the documentation for the new
> release branch. Check [Managing
> Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
> on details on how to do that. You may also want to manually trigger a build
> to make the changes visible as soon as possible.
>
>
> h3. Expectations (Minor Version only)
> * Release branch has been created and pushed
> * Changes on the new release branch are picked up by [Azure
> CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
> * {{master}} branch has the version information updated to the new version
> (check pom.xml files and
> *
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> enum)
> * New version is added to the
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> enum.
> * Make sure [flink-docker|https://github.com/apache/flink-docker/] has
> {{dev-x.y}} branch and docker e2e tests run against this branch in the
> corresponding Apache Flink release branch (see
> [apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
> *
> [apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
> has been updated appropriately in the new Apache Flink release branch.
> * The {{flink.version}} property (see
> [apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
> of Flink Benchmark repo has been updated to the latest snapshot version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new minor release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h4. Flink Benchmark Repository
First of all, checkout the {{master}} branch to {{dev-x.y}} branch in
[apache/flink-benchmarks|https://github.com/apache/flink-benchmarks], so that
we can have a branch named {{dev-x.y}} which could be built on top of
(${{CURRENT_SNAPSHOT_VERSION}}).
Then, inside the repository you need to manually update the {{flink.version
}}property inside the main *pom.xml* file. It should be pointing to the most
recent snapshot version ({{{}$NEXT_SNAPSHOT_VERSION{}}}). For example:
{code:xml}
1.18-SNAPSHOT
{code}
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* {{master}} branch has the version information updated to the new version
(check pom.xml files and
*
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum)
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch and docker e2e tests run against this branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
*
[apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
was:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd .
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new minor release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h4. Flink Benchmark Repository
First of all, checkout the {{master}} branch to {{dev-x.y}} branch in
[apache/flink-benchmarks|https://github.com/apache/flink-benchmarks], so that
we can have a branch named {{dev-x.y}} which could be built on top of
(${{CURRENT_SNAPSHOT_VERSION}}).
Then, inside the repository you need to manually update the {{flink.version}}
property inside the parent *pom.xml* file. It should be pointing to the most
recent snapshot version ($NEXT_SNAPSHOT_VERSION). For example:
{code:xml}
1.18-SNAPSHOT
{code}
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* {{master}} branch has the version information updated to the new version
(check pom.xml files and
*
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum)
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch and docker e2e tests run against this branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
*
[apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
was:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Description:
If you are doing a new minor release, you need to update Flink version in the
following repositories and the [AzureCI project
configuration|https://dev.azure.com/apache-flink/apache-flink/]:
* [apache/flink|https://github.com/apache/flink]
* [apache/flink-docker|https://github.com/apache/flink-docker]
* [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
Patch releases don't require the these repositories to be touched. Simply
checkout the already existing branch for that version:
{code:java}
$ cd ./tools
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository
Create a branch for the new version that we want to release before updating the
master branch to the next development version:
{code:bash}
$ cd ./tools
$ releasing/create_snapshot_branch.sh
$ git checkout master
$ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the
official repository.
h4. Flink Docker Repository
Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
sure that
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
points to the correct snapshot version; for {{dev-x.y}} it should point to
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
After pushing the new minor release branch, as the last step you should also
update the documentation workflow to also build the documentation for the new
release branch. Check [Managing
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
on details on how to do that. You may also want to manually trigger a build to
make the changes visible as soon as possible.
h4. Flink Benchmark Repository
First of all, checkout the {{master}} branch to {{dev-x.y}} branch in
[apache/flink-benchmarks|https://github.com/apache/flink-benchmarks], so that
we can have a branch named {{dev-x.y}} which could be built on top of
(${{CURRENT_SNAPSHOT_VERSION}}).
Then, inside the repository you need to manually update the {{flink.version}}
property inside the parent *pom.xml* file. It should be pointing to the most
recent snapshot version ($NEXT_SNAPSHOT_VERSION). For example:
{code:xml}
1.18-SNAPSHOT
{code}
h4. AzureCI Project Configuration
The new release branch needs to be configured within AzureCI to make azure
aware of the new release branch. This matter can only be handled by Ververica
employees since they are owning the AzureCI setup.
h3. Expectations (Minor Version only)
* Release branch has been created and pushed
* Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* {{master}} branch has the version information updated to the new version
(check pom.xml files and
*
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum)
* New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
* Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch and docker e2e tests run against this branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
*
[apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
was:
If you are doing a new minor release, you need to update Flink version in the
following repositories:
* [apache/
[jira] [Closed] (FLINK-31036) StateCheckpointedITCase timed out
[
https://issues.apache.org/jira/browse/FLINK-31036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan closed FLINK-31036.
---
Resolution: Fixed
> StateCheckpointedITCase timed out
> -
>
> Key: FLINK-31036
> URL: https://issues.apache.org/jira/browse/FLINK-31036
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Rui Fan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0, 1.18.0
>
> Attachments: image-2023-02-16-20-29-52-050.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46023&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=0c010d0c-3dec-5bf1-d408-7b18988b1b2b&l=10608
> {code}
> "Legacy Source Thread - Source: Custom Source -> Filter (6/12)#69980"
> #13718026 prio=5 os_prio=0 tid=0x7f05f44f0800 nid=0x128157 waiting on
> condition [0x7f059feef000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xf0a974e8> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:384)
> at
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:356)
> at
> org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.requestNewBufferBuilderFromPool(BufferWritingResultPartition.java:414)
> at
> org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.requestNewUnicastBufferBuilder(BufferWritingResultPartition.java:390)
> at
> org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.appendUnicastDataForRecordContinuation(BufferWritingResultPartition.java:328)
> at
> org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.emitRecord(BufferWritingResultPartition.java:161)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:107)
> at
> org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:55)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:105)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:91)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:59)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:31)
> at
> org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:39)
> at
> org.apache.flink.streaming.runtime.io.RecordProcessorUtils$$Lambda$1311/1256184070.accept(Unknown
> Source)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:418)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:513)
> - locked <0xd55035c0> (a java.lang.Object)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:103)
> at
> org.apache.flink.test.checkpointing.StateCheckpointedITCase$StringGeneratingSourceFunction.run(StateCheckpointedITCase.java:178)
> - locked <0xd55035c0> (a java.lang.Object)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(S
[GitHub] [flink] luoyuxia commented on a diff in pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
luoyuxia commented on code in PR #21977:
URL: https://github.com/apache/flink/pull/21977#discussion_r1112798182
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkTest.java:
##
@@ -18,178 +18,174 @@
package org.apache.flink.connectors.hive;
-import org.apache.flink.api.common.RuntimeExecutionMode;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.connector.file.table.FileSystemConnectorOptions;
+import org.apache.flink.api.dag.Transformation;
import org.apache.flink.connector.file.table.batch.BatchSink;
-import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveTable;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-import org.apache.flink.streaming.api.graph.StreamGraph;
-import org.apache.flink.streaming.api.graph.StreamNode;
-import org.apache.flink.table.api.DataTypes;
-import org.apache.flink.table.api.TableSchema;
-import org.apache.flink.table.catalog.CatalogDatabaseImpl;
-import org.apache.flink.table.catalog.CatalogTable;
-import org.apache.flink.table.catalog.CatalogTableImpl;
-import org.apache.flink.table.catalog.CatalogTest;
-import org.apache.flink.table.catalog.Column;
-import org.apache.flink.table.catalog.ObjectIdentifier;
-import org.apache.flink.table.catalog.ObjectPath;
-import org.apache.flink.table.catalog.ResolvedCatalogTable;
-import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.api.SqlDialect;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.internal.TableEnvironmentImpl;
import org.apache.flink.table.catalog.hive.HiveCatalog;
import org.apache.flink.table.catalog.hive.HiveTestUtils;
-import org.apache.flink.table.connector.sink.DataStreamSinkProvider;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.data.binary.BinaryRowData;
-import org.apache.flink.table.factories.DynamicTableSinkFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import
org.apache.flink.table.runtime.connector.sink.SinkRuntimeProviderContext;
+import org.apache.flink.table.operations.ModifyOperation;
+import org.apache.flink.table.operations.Operation;
+import org.apache.flink.table.planner.delegation.PlannerBase;
import org.apache.flink.util.TestLoggerExtension;
-import org.junit.jupiter.api.AfterAll;
-import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Optional;
+import java.util.List;
+import static
org.apache.flink.connector.file.table.FileSystemConnectorOptions.COMPACTION_PARALLELISM;
+import static
org.apache.flink.connector.file.table.FileSystemConnectorOptions.SINK_PARALLELISM;
import static org.assertj.core.api.Assertions.assertThat;
/** Tests for {@link HiveTableSink} enable auto-compaction. */
@ExtendWith(TestLoggerExtension.class)
class HiveTableCompactSinkTest {
Review Comment:
Since it's mainly for test parallelism, how about rename it to
`HiveTableCompactSinkParallelismTest`?
Also the java doc for this class should update.
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkTest.java:
##
@@ -18,178 +18,174 @@
package org.apache.flink.connectors.hive;
-import org.apache.flink.api.common.RuntimeExecutionMode;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.connector.file.table.FileSystemConnectorOptions;
+import org.apache.flink.api.dag.Transformation;
import org.apache.flink.connector.file.table.batch.BatchSink;
-import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveTable;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-import org.apache.flink.streaming.api.graph.StreamGraph;
-import org.apache.flink.streaming.api.graph.StreamNode;
-import org.apache.flink.table.api.DataTypes;
-import org.apache.flink.table.api.TableSchema;
-import org.apache.flink.table.catalog.CatalogDatabaseImpl;
-import org.apache.flink.table.catalog.CatalogTable;
-import org.apache.flink.table.catalog.CatalogTableImpl;
-import org.apache.flink.table.catalog.CatalogTest;
-import org.apache.flink.table.catalog.Column;
-import org.apache.flink.table.catalog.ObjectIdentifier;
-import org.apache.flink.table.catalog.ObjectPath;
-import org.apache.flink.table.catalog.ResolvedCatalogTable;
-import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.api.SqlDialect;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.internal.TableEnvironmentImpl;
import org.apache.flink.table.catalog.hive.HiveCatalog;
import org.apache.flink.table.catalog.hive.HiveTestUtils;
-import org.apache.flink.
[GitHub] [flink] luoyuxia commented on a diff in pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
luoyuxia commented on code in PR #21977:
URL: https://github.com/apache/flink/pull/21977#discussion_r1112838547
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkTest.java:
##
@@ -18,178 +18,174 @@
package org.apache.flink.connectors.hive;
-import org.apache.flink.api.common.RuntimeExecutionMode;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.connector.file.table.FileSystemConnectorOptions;
+import org.apache.flink.api.dag.Transformation;
import org.apache.flink.connector.file.table.batch.BatchSink;
-import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveTable;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-import org.apache.flink.streaming.api.graph.StreamGraph;
-import org.apache.flink.streaming.api.graph.StreamNode;
-import org.apache.flink.table.api.DataTypes;
-import org.apache.flink.table.api.TableSchema;
-import org.apache.flink.table.catalog.CatalogDatabaseImpl;
-import org.apache.flink.table.catalog.CatalogTable;
-import org.apache.flink.table.catalog.CatalogTableImpl;
-import org.apache.flink.table.catalog.CatalogTest;
-import org.apache.flink.table.catalog.Column;
-import org.apache.flink.table.catalog.ObjectIdentifier;
-import org.apache.flink.table.catalog.ObjectPath;
-import org.apache.flink.table.catalog.ResolvedCatalogTable;
-import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.api.SqlDialect;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.internal.TableEnvironmentImpl;
import org.apache.flink.table.catalog.hive.HiveCatalog;
import org.apache.flink.table.catalog.hive.HiveTestUtils;
-import org.apache.flink.table.connector.sink.DataStreamSinkProvider;
-import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.data.binary.BinaryRowData;
-import org.apache.flink.table.factories.DynamicTableSinkFactory;
-import org.apache.flink.table.factories.FactoryUtil;
-import
org.apache.flink.table.runtime.connector.sink.SinkRuntimeProviderContext;
+import org.apache.flink.table.operations.ModifyOperation;
+import org.apache.flink.table.operations.Operation;
+import org.apache.flink.table.planner.delegation.PlannerBase;
import org.apache.flink.util.TestLoggerExtension;
-import org.junit.jupiter.api.AfterAll;
-import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Optional;
+import java.util.List;
+import static
org.apache.flink.connector.file.table.FileSystemConnectorOptions.COMPACTION_PARALLELISM;
+import static
org.apache.flink.connector.file.table.FileSystemConnectorOptions.SINK_PARALLELISM;
import static org.assertj.core.api.Assertions.assertThat;
/** Tests for {@link HiveTableSink} enable auto-compaction. */
@ExtendWith(TestLoggerExtension.class)
class HiveTableCompactSinkTest {
+/**
+ * Represents the parallelism need not check, it should follow the setting
of planer or auto
Review Comment:
If my understand commented below is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31165) Over Agg: The window rank function without order by error in top N query
[
https://issues.apache.org/jira/browse/FLINK-31165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691493#comment-17691493
]
Martijn Visser commented on FLINK-31165:
[~godfrey] [~lincoln.86xy] Any thoughts on this one?
> Over Agg: The window rank function without order by error in top N query
>
>
> Key: FLINK-31165
> URL: https://issues.apache.org/jira/browse/FLINK-31165
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.16.0
>Reporter: P Rohan Kumar
>Priority: Major
>
>
> {code:java}
> val env: StreamExecutionEnvironment =
> StreamExecutionEnvironment.getExecutionEnvironment
> val tableEnv = StreamTableEnvironment.create(env)
> val td = TableDescriptor.forConnector("datagen").option("rows-per-second",
> "10")
> .option("number-of-rows", "10")
> .schema(Schema
> .newBuilder()
> .column("NAME", DataTypes.VARCHAR(2147483647))
> .column("ROLLNO", DataTypes.DECIMAL(5, 0))
> .column("DOB", DataTypes.DATE())
> .column("CLASS", DataTypes.DECIMAL(2, 0))
> .column("SUBJECT", DataTypes.VARCHAR(2147483647))
> .build())
> .build()
> val table = tableEnv.from(td)
> tableEnv.createTemporaryView("temp_table", table)
> val newTable = tableEnv.sqlQuery("select temp_table.*,cast('2022-01-01' as
> date) SRC_NO from temp_table")
> tableEnv.createTemporaryView("temp_table2", newTable)
> val newTable2 = tableEnv.sqlQuery("select * from (select
> NAME,ROLLNO,row_number() over (partition by NAME ORDER BY SRC_NO) AS rownum
> from temp_table2 a) where rownum <= 1")
> tableEnv.toChangelogStream(newTable2).print()
> env.execute()
> {code}
>
>
> I am getting the below error if I run the above code.
> I have already provided an order by column.
> If I change the order by column to some other column, such as "SUBJECT", then
> the job runs fine.
>
>
> {code:java}
> Exception in thread "main" java.lang.RuntimeException: Error while applying
> rule FlinkLogicalOverAggregateConverter(in:NONE,out:LOGICAL), args
> [rel#245:LogicalWindow.NONE.any.None:
> 0.[NONE].[NONE](input=RelSubset#244,window#0=window(partition {0} rows
> between UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()]))]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:256)
> at
> org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:58)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:510)
> at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:312)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:59)
> at
> scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:187)
> at
> scala.collection.TraversableOnce$folder$1$.apply(TraversableOnce.scala:185)
> at scala.collection.Iterator.foreach(Iterator.scala:943)
> at scala.collection.Iterator.foreach$(Iterator.scala:943)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:189)
> at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:184)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:55)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:176)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.doOptimize(StreamCommonSubGraphBasedOptimizer.scala:83)
> at
> org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:87)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:315)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:195)
> at
> org.apache.flink.table.api.bridge.internal.AbstractStreamTableEnvironmentImpl.toStreamInternal(AbstractStreamTableEnvironmentImpl.java:224)
> at
> org.apache.flink.table.api.bridge.internal.AbstractStreamTableEnvironmentImpl.toStreamInternal(AbstractStreamTableEnvironmentImpl.java:219)
> at
> org.apache.flink.table.api.bridge.scala
[GitHub] [flink] snuyanzin commented on a diff in pull request #21945: [FLINK-31091][Table SQL/Client] Add Ser/de for Interval types
snuyanzin commented on code in PR #21945: URL: https://github.com/apache/flink/pull/21945#discussion_r1112846597 ## flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/rest/serde/LogicalTypeJsonSerializer.java: ## @@ -87,6 +89,10 @@ public final class LogicalTypeJsonSerializer extends StdSerializer public static final String FIELD_NAME_CLASS = "class"; public static final String FIELD_NAME_SERIALIZER = "serializer"; +// INTERVAL Review Comment: updated usage of `precision` with usage of `yearPrecision` and `dayPrecision` based on offline discussion in slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] reswqa commented on a diff in pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
reswqa commented on code in PR #21977:
URL: https://github.com/apache/flink/pull/21977#discussion_r1112847462
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkTest.java:
##
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive;
+
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.connector.file.table.batch.BatchSink;
+import org.apache.flink.table.api.SqlDialect;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.internal.TableEnvironmentImpl;
+import org.apache.flink.table.catalog.hive.HiveCatalog;
+import org.apache.flink.table.catalog.hive.HiveTestUtils;
+import org.apache.flink.table.operations.ModifyOperation;
+import org.apache.flink.table.operations.Operation;
+import org.apache.flink.table.planner.delegation.PlannerBase;
+import org.apache.flink.util.TestLoggerExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+import java.util.Collections;
+import java.util.List;
+
+import static
org.apache.flink.connector.file.table.FileSystemConnectorOptions.COMPACTION_PARALLELISM;
+import static
org.apache.flink.connector.file.table.FileSystemConnectorOptions.SINK_PARALLELISM;
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link HiveTableSink} enable auto-compaction. */
+@ExtendWith(TestLoggerExtension.class)
+class HiveTableCompactSinkTest {
+/**
+ * Represents the parallelism need not check, it should follow the setting
of planer or auto
+ * inference.
+ */
+public static final int NEED_NOT_CHECK_PARALLELISM = -1;
+
+private HiveCatalog catalog;
+
+private TableEnvironment tableEnv;
+
+@BeforeEach
+void before() {
+catalog = HiveTestUtils.createHiveCatalog();
+catalog.open();
+tableEnv = HiveTestUtils.createTableEnvInBatchMode(SqlDialect.HIVE);
+tableEnv.registerCatalog(catalog.getName(), catalog);
+tableEnv.useCatalog(catalog.getName());
+}
+
+@AfterEach
+void after() {
+if (catalog != null) {
+catalog.close();
+}
+}
+
+/** If only sink parallelism is set, compact operator should follow this
setting. */
+@Test
+void testOnlySetSinkParallelism() {
+final int sinkParallelism = 4;
+
+tableEnv.executeSql(
+String.format(
+"CREATE TABLE src ("
++ " key string,"
++ " value string"
++ ") TBLPROPERTIES ("
++ " 'auto-compaction' = 'true', "
++ " '%s' = '%s' )",
+SINK_PARALLELISM.key(), sinkParallelism));
+
+assertSinkAndCompactOperatorParallelism(true, true, sinkParallelism,
sinkParallelism);
+}
+
+@Test
+void testOnlySetCompactParallelism() {
+final int compactParallelism = 4;
+
+tableEnv.executeSql(
+String.format(
+"CREATE TABLE src ("
++ " key string,"
++ " value string"
++ ") TBLPROPERTIES ("
++ " 'auto-compaction' = 'true', "
++ " '%s' = '%s' )",
+COMPACTION_PARALLELISM.key(), compactParallelism));
+
+assertSinkAndCompactOperatorParallelism(
+false, true, NEED_NOT_CHECK_PARALLELISM, compactParallelism);
+}
+
+@Test
+void testSetBothSinkAndCompactParallelism() {
+final int sinkParallelism = 8;
+final int compactParallelism = 4;
+
+tableEnv.executeSql(
+String.format(
+"CREATE TABLE src ("
++ " key string,"
++ " value string"
++ ") TBLPROPERTIES ("
++
[GitHub] [flink] MartijnVisser commented on pull request #21965: [FLINK-31121] Support discarding too large records in kafka sink
MartijnVisser commented on PR #21965: URL: https://github.com/apache/flink/pull/21965#issuecomment-1438209724 Can we hold off merging this PR since we're in the process of externalizing the Kafka connector, which means that this PR should then be moved to the external Kafka connector repository? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Fix Version/s: 1.17.0
> Create a release branch
> ---
>
> Key: FLINK-31153
> URL: https://issues.apache.org/jira/browse/FLINK-31153
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Leonard Xu
>Priority: Major
> Fix For: 1.17.0
>
>
> If you are doing a new minor release, you need to update Flink version in the
> following repositories and the [AzureCI project
> configuration|https://dev.azure.com/apache-flink/apache-flink/]:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Patch releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ cd ./tools
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Flink repository
> Create a branch for the new version that we want to release before updating
> the master branch to the next development version:
> {code:bash}
> $ cd ./tools
> $ releasing/create_snapshot_branch.sh
> $ git checkout master
> $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
> releasing/update_branch_version.sh
> {code}
> In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> as the last entry:
> {code:java}
> // ...
> v1_12("1.12"),
> v1_13("1.13"),
> v1_14("1.14"),
> v1_15("1.15"),
> v1_16("1.16");
> {code}
> The newly created branch and updated {{master}} branch need to be pushed to
> the official repository.
> h4. Flink Docker Repository
> Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
> [apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
> sure that
> [apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
> points to the correct snapshot version; for {{dev-x.y}} it should point to
> {{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
> recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
> After pushing the new minor release branch, as the last step you should also
> update the documentation workflow to also build the documentation for the new
> release branch. Check [Managing
> Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
> on details on how to do that. You may also want to manually trigger a build
> to make the changes visible as soon as possible.
> h4. Flink Benchmark Repository
> First of all, checkout the {{master}} branch to {{dev-x.y}} branch in
> [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks], so that
> we can have a branch named {{dev-x.y}} which could be built on top of
> (${{CURRENT_SNAPSHOT_VERSION}}).
> Then, inside the repository you need to manually update the {{flink.version}}
> property inside the parent *pom.xml* file. It should be pointing to the most
> recent snapshot version ($NEXT_SNAPSHOT_VERSION). For example:
> {code:xml}
> 1.18-SNAPSHOT
> {code}
> h4. AzureCI Project Configuration
> The new release branch needs to be configured within AzureCI to make azure
> aware of the new release branch. This matter can only be handled by Ververica
> employees since they are owning the AzureCI setup.
>
>
> h3. Expectations (Minor Version only)
> * Release branch has been created and pushed
> * Changes on the new release branch are picked up by [Azure
> CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
> * {{master}} branch has the version information updated to the new version
> (check pom.xml files and
> *
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> enum)
> * New version is added to the
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> enum.
> * Make sure [flink-docker|https://github.com/apache/flink-docker/] has
> {{dev-x.y}} branch and docker e2e tests run against this branch in the
> corresponding Apache Flink release branch (see
> [apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docke
[jira] [Updated] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31153:
--
Affects Version/s: 1.17.0
> Create a release branch
> ---
>
> Key: FLINK-31153
> URL: https://issues.apache.org/jira/browse/FLINK-31153
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Leonard Xu
>Priority: Major
> Fix For: 1.17.0
>
>
> If you are doing a new minor release, you need to update Flink version in the
> following repositories and the [AzureCI project
> configuration|https://dev.azure.com/apache-flink/apache-flink/]:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Patch releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ cd ./tools
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Flink repository
> Create a branch for the new version that we want to release before updating
> the master branch to the next development version:
> {code:bash}
> $ cd ./tools
> $ releasing/create_snapshot_branch.sh
> $ git checkout master
> $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
> releasing/update_branch_version.sh
> {code}
> In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> as the last entry:
> {code:java}
> // ...
> v1_12("1.12"),
> v1_13("1.13"),
> v1_14("1.14"),
> v1_15("1.15"),
> v1_16("1.16");
> {code}
> The newly created branch and updated {{master}} branch need to be pushed to
> the official repository.
> h4. Flink Docker Repository
> Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
> [apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
> sure that
> [apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
> points to the correct snapshot version; for {{dev-x.y}} it should point to
> {{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
> recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
> After pushing the new minor release branch, as the last step you should also
> update the documentation workflow to also build the documentation for the new
> release branch. Check [Managing
> Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
> on details on how to do that. You may also want to manually trigger a build
> to make the changes visible as soon as possible.
> h4. Flink Benchmark Repository
> First of all, checkout the {{master}} branch to {{dev-x.y}} branch in
> [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks], so that
> we can have a branch named {{dev-x.y}} which could be built on top of
> (${{CURRENT_SNAPSHOT_VERSION}}).
> Then, inside the repository you need to manually update the {{flink.version}}
> property inside the parent *pom.xml* file. It should be pointing to the most
> recent snapshot version ($NEXT_SNAPSHOT_VERSION). For example:
> {code:xml}
> 1.18-SNAPSHOT
> {code}
> h4. AzureCI Project Configuration
> The new release branch needs to be configured within AzureCI to make azure
> aware of the new release branch. This matter can only be handled by Ververica
> employees since they are owning the AzureCI setup.
>
>
> h3. Expectations (Minor Version only)
> * Release branch has been created and pushed
> * Changes on the new release branch are picked up by [Azure
> CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
> * {{master}} branch has the version information updated to the new version
> (check pom.xml files and
> *
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> enum)
> * New version is added to the
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> enum.
> * Make sure [flink-docker|https://github.com/apache/flink-docker/] has
> {{dev-x.y}} branch and docker e2e tests run against this branch in the
> corresponding Apache Flink release branch (see
> [apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-e
[jira] [Resolved] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl resolved FLINK-31153.
---
Resolution: Fixed
* (/) Release branch has been created and pushed
* (/) Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* {{(/) master}} branch has the version information updated to the new version
(check pom.xml files)
** Related commit is
[923361a3|https://github.com/apache/flink/commit/923361a3006c7c4ce8acbc74a9b756f93c93020a]
* (/) New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
** Related commit is [d76c49b4|https://github.com/apache/flink/commit/d76c49b4]
* (/) Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch
* (/) Docker e2e tests run against the new flink-docker branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
** Related commit is [df4d2ff7|https://github.com/apache/flink/commit/df4d2ff7]
* [(/)
apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
** Related commit is [df4d2ff7|https://github.com/apache/flink/commit/df4d2ff7]
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
** Related commit is
[79fdbaf6|https://github.com/apache/flink-benchmarks/commit/79fdbaf6f7ae2da63d08374b3052422afcec34ea]
> Create a release branch
> ---
>
> Key: FLINK-31153
> URL: https://issues.apache.org/jira/browse/FLINK-31153
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Leonard Xu
>Priority: Major
>
> If you are doing a new minor release, you need to update Flink version in the
> following repositories and the [AzureCI project
> configuration|https://dev.azure.com/apache-flink/apache-flink/]:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Patch releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ cd ./tools
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Flink repository
> Create a branch for the new version that we want to release before updating
> the master branch to the next development version:
> {code:bash}
> $ cd ./tools
> $ releasing/create_snapshot_branch.sh
> $ git checkout master
> $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$NEXT_SNAPSHOT_VERSION
> releasing/update_branch_version.sh
> {code}
> In the {{master}} branch, add a new value (e.g. {{{}v1_16("1.16"){}}}) to
> [apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
> as the last entry:
> {code:java}
> // ...
> v1_12("1.12"),
> v1_13("1.13"),
> v1_14("1.14"),
> v1_15("1.15"),
> v1_16("1.16");
> {code}
> The newly created branch and updated {{master}} branch need to be pushed to
> the official repository.
> h4. Flink Docker Repository
> Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the
> [apache/flink-docker|https://github.com/apache/flink-docker] repository. Make
> sure that
> [apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
> points to the correct snapshot version; for {{dev-x.y}} it should point to
> {{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most
> recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).
> After pushing the new minor release branch, as the last step you should also
> update the documentation workflow to also build the documentation for the new
> release branch. Check [Managing
> Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
> on details on how to do that. You may also want to manually trigger a build
> to make the changes visible as soon as possible.
> h4. Flink Benchmark Repository
> First of all, checkout the {{master}} branch to {{dev-x.y}} branch in
> [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks], so that
> we can hav
[jira] [Comment Edited] (FLINK-31153) Create a release branch
[
https://issues.apache.org/jira/browse/FLINK-31153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691495#comment-17691495
]
Matthias Pohl edited comment on FLINK-31153 at 2/21/23 10:14 AM:
-
Closing this issue. All the expected artifacts are created:
* (/) Release branch has been created and pushed
* (/) Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* {{(/) master}} branch has the version information updated to the new version
(check pom.xml files)
** Related commit is
[923361a3|https://github.com/apache/flink/commit/923361a3006c7c4ce8acbc74a9b756f93c93020a]
* (/) New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
** Related commit is [d76c49b4|https://github.com/apache/flink/commit/d76c49b4]
* (/) Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch
* (/) Docker e2e tests run against the new flink-docker branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
** Related commit is [df4d2ff7|https://github.com/apache/flink/commit/df4d2ff7]
* [(/)
apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
** Related commit is [df4d2ff7|https://github.com/apache/flink/commit/df4d2ff7]
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
** Related commit is
[79fdbaf6|https://github.com/apache/flink-benchmarks/commit/79fdbaf6f7ae2da63d08374b3052422afcec34ea]
was (Author: mapohl):
* (/) Release branch has been created and pushed
* (/) Changes on the new release branch are picked up by [Azure
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
* {{(/) master}} branch has the version information updated to the new version
(check pom.xml files)
** Related commit is
[923361a3|https://github.com/apache/flink/commit/923361a3006c7c4ce8acbc74a9b756f93c93020a]
* (/) New version is added to the
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
enum.
** Related commit is [d76c49b4|https://github.com/apache/flink/commit/d76c49b4]
* (/) Make sure [flink-docker|https://github.com/apache/flink-docker/] has
{{dev-x.y}} branch
* (/) Docker e2e tests run against the new flink-docker branch in the
corresponding Apache Flink release branch (see
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
** Related commit is [df4d2ff7|https://github.com/apache/flink/commit/df4d2ff7]
* [(/)
apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
has been updated appropriately in the new Apache Flink release branch.
** Related commit is [df4d2ff7|https://github.com/apache/flink/commit/df4d2ff7]
* The {{flink.version}} property (see
[apache/flink-benchmarks:pom.xml|https://github.com/apache/flink-benchmarks/blob/master/pom.xml#L48]
of Flink Benchmark repo has been updated to the latest snapshot version.
** Related commit is
[79fdbaf6|https://github.com/apache/flink-benchmarks/commit/79fdbaf6f7ae2da63d08374b3052422afcec34ea]
> Create a release branch
> ---
>
> Key: FLINK-31153
> URL: https://issues.apache.org/jira/browse/FLINK-31153
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Leonard Xu
>Priority: Major
> Fix For: 1.17.0
>
>
> If you are doing a new minor release, you need to update Flink version in the
> following repositories and the [AzureCI project
> configuration|https://dev.azure.com/apache-flink/apache-flink/]:
> * [apache/flink|https://github.com/apache/flink]
> * [apache/flink-docker|https://github.com/apache/flink-docker]
> * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]
> Patch releases don't require the these repositories to be touched. Simply
> checkout the already existing branch for that version:
> {code:java}
> $ cd ./tools
> $ git checkout release-$SHORT_RELEASE_VERSION
> {code}
> h4. Fli
[GitHub] [flink] reswqa commented on a diff in pull request #21977: [FLINK-31132] Compact without setting parallelism does not follow the configured sink parallelism for HiveTableSink
reswqa commented on code in PR #21977:
URL: https://github.com/apache/flink/pull/21977#discussion_r1112847462
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableCompactSinkTest.java:
##
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive;
+
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.connector.file.table.batch.BatchSink;
+import org.apache.flink.table.api.SqlDialect;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.internal.TableEnvironmentImpl;
+import org.apache.flink.table.catalog.hive.HiveCatalog;
+import org.apache.flink.table.catalog.hive.HiveTestUtils;
+import org.apache.flink.table.operations.ModifyOperation;
+import org.apache.flink.table.operations.Operation;
+import org.apache.flink.table.planner.delegation.PlannerBase;
+import org.apache.flink.util.TestLoggerExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+import java.util.Collections;
+import java.util.List;
+
+import static
org.apache.flink.connector.file.table.FileSystemConnectorOptions.COMPACTION_PARALLELISM;
+import static
org.apache.flink.connector.file.table.FileSystemConnectorOptions.SINK_PARALLELISM;
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link HiveTableSink} enable auto-compaction. */
+@ExtendWith(TestLoggerExtension.class)
+class HiveTableCompactSinkTest {
+/**
+ * Represents the parallelism need not check, it should follow the setting
of planer or auto
+ * inference.
+ */
+public static final int NEED_NOT_CHECK_PARALLELISM = -1;
+
+private HiveCatalog catalog;
+
+private TableEnvironment tableEnv;
+
+@BeforeEach
+void before() {
+catalog = HiveTestUtils.createHiveCatalog();
+catalog.open();
+tableEnv = HiveTestUtils.createTableEnvInBatchMode(SqlDialect.HIVE);
+tableEnv.registerCatalog(catalog.getName(), catalog);
+tableEnv.useCatalog(catalog.getName());
+}
+
+@AfterEach
+void after() {
+if (catalog != null) {
+catalog.close();
+}
+}
+
+/** If only sink parallelism is set, compact operator should follow this
setting. */
+@Test
+void testOnlySetSinkParallelism() {
+final int sinkParallelism = 4;
+
+tableEnv.executeSql(
+String.format(
+"CREATE TABLE src ("
++ " key string,"
++ " value string"
++ ") TBLPROPERTIES ("
++ " 'auto-compaction' = 'true', "
++ " '%s' = '%s' )",
+SINK_PARALLELISM.key(), sinkParallelism));
+
+assertSinkAndCompactOperatorParallelism(true, true, sinkParallelism,
sinkParallelism);
+}
+
+@Test
+void testOnlySetCompactParallelism() {
+final int compactParallelism = 4;
+
+tableEnv.executeSql(
+String.format(
+"CREATE TABLE src ("
++ " key string,"
++ " value string"
++ ") TBLPROPERTIES ("
++ " 'auto-compaction' = 'true', "
++ " '%s' = '%s' )",
+COMPACTION_PARALLELISM.key(), compactParallelism));
+
+assertSinkAndCompactOperatorParallelism(
+false, true, NEED_NOT_CHECK_PARALLELISM, compactParallelism);
+}
+
+@Test
+void testSetBothSinkAndCompactParallelism() {
+final int sinkParallelism = 8;
+final int compactParallelism = 4;
+
+tableEnv.executeSql(
+String.format(
+"CREATE TABLE src ("
++ " key string,"
++ " value string"
++ ") TBLPROPERTIES ("
++
[jira] [Updated] (FLINK-31150) Cross team testing
[
https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31150:
--
Description:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
was:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
--
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
> Cross team testing
> --
>
> Key: FLINK-31150
> URL: https://issues.apache.org/jira/browse/FLINK-31150
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> For user facing features that go into the release we'd like to ensure they
> can actually _be used_ by Flink users. To achieve this the release managers
> ensure that an issue for cross team testing is created in the Apache Flink
> Jira. This can and should be picked up by other community members to verify
> the functionality and usability of the feature.
> The issue should contain some entry points which enables other community
> members to test it. It should not contain documentation on how to use the
> feature as this should be part of the actual documentation. The cross team
> tests are performed after the feature freeze. Documentation should be in
> place before that. Those tests are manual tests, so do not confuse them with
> automated tests.
> To sum that up:
> * User facing features should be tested by other contributors
> * The scope is usability and sanity of the feature
> * The feature needs to be already documented
> * The contributor creates an issue containing some pointers on how to get
> started (e.g. link to the documentation, suggested targets of verification)
> * Other community members pick those issues up and provide feedback
> * Cross team testing happens right after the feature freeze
>
> h3. Expectations
> * Jira issues for each expected release task according to the release plan is
> created and labeled as {{release-testing}}.
> * All the created release-testing-related Jira issues are resolved and the
> corresponding blocker issues are
[jira] [Commented] (FLINK-30337) Stateful Functions application throws an exception when stopping a job gracefully creating a final savepoint
[
https://issues.apache.org/jira/browse/FLINK-30337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691499#comment-17691499
]
Alexey Novakov commented on FLINK-30337:
I have been able to reproduce this issue as well.
Flink CLI Stop command fails with above exception after timeout and there is
one more exception in *statefun-worker* pod:
{code:java}
2023-02-21 10:06:36,818 INFO
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] -
feedback-union -> functions (1/1)#0 - asynchronous part of checkpoint 2 could
not be completed.
java.util.concurrent.CancellationException: null
at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:645)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:57)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:177)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
[flink-dist_2.12-1.14.3.jar:1.14.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?] {code}
> Stateful Functions application throws an exception when stopping a job
> gracefully creating a final savepoint
>
>
> Key: FLINK-30337
> URL: https://issues.apache.org/jira/browse/FLINK-30337
> Project: Flink
> Issue Type: Bug
> Components: Stateful Functions
>Affects Versions: statefun-3.2.0
>Reporter: Ali Bahadir Zeybek
>Priority: Minor
>
> When running a Stateful Functions applications, if the stop[1] command is
> executed, the client throws a FlinkException with the following stack trace
> where *953498833da99ec437758b49b7d5befd* is the specific job id:
>
> {code:java}
> The program finished with the following
> exception:org.apache.flink.util.FlinkException: Could not stop with a
> savepoint job "953498833da99ec437758b49b7d5befd".
> at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581)
> at
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
> at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.util.concurrent.TimeoutException
> at java.base/java.util.concurrent.CompletableFuture.timedGet(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579)
> ... 6 more {code}
>
> How to reproduce:
> # Follow the README[2] of the k8s deployment example of the
> *flink-statefun-playground* project to have a running application
> # Open the Flink UI that is started to get the *JOB_ID*
> # Detect the *STATEFUN_MASTER_POD_NAME* by running: *kubectl get pods
> --namespace statefun*
> # Start a shell into the *statefun-master* pod by issuing the: *kubectl exec
> -it --namespace statefun $STATEFUN_MASTER_POD_NAME – /bin/bash*
> # Run the stop command: *./bin/flink stop --savepointPath
> /tmp/flink-savepoints $JOB_ID*
>
> [1]:
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/cli/#stopping-a-job-gracefully-creating-a-final-savepoint]
> [2]:
> [https://github.com/apache/flink-statefun-playground/blob/main/deployments/k8s/README.md]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31150) Cross team testing
[
https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31150:
--
Description:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
was:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
> Cross team testing
> --
>
> Key: FLINK-31150
> URL: https://issues.apache.org/jira/browse/FLINK-31150
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> For user facing features that go into the release we'd like to ensure they
> can actually _be used_ by Flink users. To achieve this the release managers
> ensure that an issue for cross team testing is created in the Apache Flink
> Jira. This can and should be picked up by other community members to verify
> the functionality and usability of the feature.
> The issue should contain some entry points which enables other community
> members to test it. It should not contain documentation on how to use the
> feature as this should be part of the actual documentation. The cross team
> tests are performed after the feature freeze. Documentation should be in
> place before that. Those tests are manual tests, so do not confuse them with
> automated tests.
> To sum that up:
> * User facing features should be tested by other contributors
> * The scope is usability and sanity of the feature
> * The feature needs to be already documented
> * The contributor creates an issue containing some pointers on how to get
> started (e.g. link to the documentation, suggested targets of verification)
> * Other community members pick those issues up and provide feedback
> * Cross team testing happens right after the feature freeze
>
> h3. Expectations
> * Jira issues for each expected release task according to the release plan is
> created and labeled as {{release-testing}}.
> * All the created release-testing-related Jira issues are resolved and the
> corresponding blocker issues are f
[jira] [Updated] (FLINK-31150) Cross team testing
[
https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31150:
--
Description:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
was:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
> Cross team testing
> --
>
> Key: FLINK-31150
> URL: https://issues.apache.org/jira/browse/FLINK-31150
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> For user facing features that go into the release we'd like to ensure they
> can actually _be used_ by Flink users. To achieve this the release managers
> ensure that an issue for cross team testing is created in the Apache Flink
> Jira. This can and should be picked up by other community members to verify
> the functionality and usability of the feature.
> The issue should contain some entry points which enables other community
> members to test it. It should not contain documentation on how to use the
> feature as this should be part of the actual documentation. The cross team
> tests are performed after the feature freeze. Documentation should be in
> place before that. Those tests are manual tests, so do not confuse them with
> automated tests.
> To sum that up:
> * User facing features should be tested by other contributors
> * The scope is usability and sanity of the feature
> * The feature needs to be already documented
> * The contributor creates an issue containing some pointers on how to get
> started (e.g. link to the documentation, suggested targets of verification)
> * Other community members pick those issues up and provide feedback
> * Cross team testing happens right after the feature freeze
>
> h3. Expectations
> * Jira issues for each expected release task according to the release plan is
> created and labeled as {{release-testing}}.
> * All the created release-testing-related Jira issues are resolved and the
> corresponding blocker issues are f
[jira] [Updated] (FLINK-31150) Cross team testing
[
https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31150:
--
Description:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
was:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
> Cross team testing
> --
>
> Key: FLINK-31150
> URL: https://issues.apache.org/jira/browse/FLINK-31150
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> For user facing features that go into the release we'd like to ensure they
> can actually _be used_ by Flink users. To achieve this the release managers
> ensure that an issue for cross team testing is created in the Apache Flink
> Jira. This can and should be picked up by other community members to verify
> the functionality and usability of the feature.
> The issue should contain some entry points which enables other community
> members to test it. It should not contain documentation on how to use the
> feature as this should be part of the actual documentation. The cross team
> tests are performed after the feature freeze. Documentation should be in
> place before that. Those tests are manual tests, so do not confuse them with
> automated tests.
> To sum that up:
> * User facing features should be tested by other contributors
> * The scope is usability and sanity of the feature
> * The feature needs to be already documented
> * The contributor creates an issue containing some pointers on how to get
> started (e.g. link to the documentation, suggested targets of verification)
> * Other community members pick those issues up and provide feedback
> * Cross team testing happens right after the feature freeze
>
> h3. Expectations
> * Jira issues for each expected release task according to the release plan is
> created and labeled as {{release-testing}}.
> * All the created release-testing-related Jira issues are resolved and the
> corresponding blocker issues are fix
[jira] [Comment Edited] (FLINK-30337) Stateful Functions application throws an exception when stopping a job gracefully creating a final savepoint
[
https://issues.apache.org/jira/browse/FLINK-30337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691499#comment-17691499
]
Alexey Novakov edited comment on FLINK-30337 at 2/21/23 10:16 AM:
--
I have been able to reproduce this issue as well.
Flink CLI Stop command fails with above exception after timeout and there is
one more exception in *statefun-worker* pod:
{code:java}
2023-02-21 10:06:36,818 INFO
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] -
feedback-union -> functions (1/1)#0 - asynchronous part of checkpoint 2 could
not be completed.
java.util.concurrent.CancellationException: null
at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:645)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:57)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:177)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
[flink-dist_2.12-1.14.3.jar:1.14.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?] {code}
was (Author: novakov.alex):
I have been able to reproduce this issue as well.
Flink CLI Stop command fails with above exception after timeout and there is
one more exception in *statefun-worker* pod:
{code:java}
2023-02-21 10:06:36,818 INFO
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] -
feedback-union -> functions (1/1)#0 - asynchronous part of checkpoint 2 could
not be completed.
java.util.concurrent.CancellationException: null
at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:645)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:57)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:177)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
[flink-dist_2.12-1.14.3.jar:1.14.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?] {code}
> Stateful Functions application throws an exception when stopping a job
> gracefully creating a final savepoint
>
>
> Key: FLINK-30337
> URL: https://issues.apache.org/jira/browse/FLINK-30337
> Project: Flink
> Issue Type: Bug
> Components: Stateful Functions
>Affects Versions: statefun-3.2.0
>Reporter: Ali Bahadir Zeybek
>Priority: Minor
>
> When running a Stateful Functions applications, if the stop[1] command is
> executed, the client throws a FlinkException with the following stack trace
> where *953498833da99ec437758b49b7d5befd* is the specific job id:
>
> {code:java}
> The program finished with the following
> exception:org.apache.flink.util.FlinkException: Could not stop with a
> savepoint job "953498833da99ec437758b49b7d5befd".
> at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581)
> at
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
> at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.util.concurrent.TimeoutException
> at java.base/java.util.concurrent.CompletableFuture.timedGet(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFron
[jira] [Updated] (FLINK-31150) Cross team testing
[
https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31150:
--
Description:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
was:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
> Cross team testing
> --
>
> Key: FLINK-31150
> URL: https://issues.apache.org/jira/browse/FLINK-31150
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> For user facing features that go into the release we'd like to ensure they
> can actually _be used_ by Flink users. To achieve this the release managers
> ensure that an issue for cross team testing is created in the Apache Flink
> Jira. This can and should be picked up by other community members to verify
> the functionality and usability of the feature.
> The issue should contain some entry points which enables other community
> members to test it. It should not contain documentation on how to use the
> feature as this should be part of the actual documentation. The cross team
> tests are performed after the feature freeze. Documentation should be in
> place before that. Those tests are manual tests, so do not confuse them with
> automated tests.
> To sum that up:
> * User facing features should be tested by other contributors
> * The scope is usability and sanity of the feature
> * The feature needs to be already documented
> * The contributor creates an issue containing some pointers on how to get
> started (e.g. link to the documentation, suggested targets of verification)
> * Other community members pick those issues up and provide feedback
> * Cross team testing happens right after the feature freeze
>
> h3. Expectations
> * Jira issues for each expected release task according to the release plan is
> created and labeled as {{release-testing}}.
> * All the created release-testing-related Jira issues are resolved and the
> corresponding blocker issues are fix
[jira] [Updated] (FLINK-31150) Cross team testing
[
https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31150:
--
Description:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
--
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
was:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
> Cross team testing
> --
>
> Key: FLINK-31150
> URL: https://issues.apache.org/jira/browse/FLINK-31150
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> For user facing features that go into the release we'd like to ensure they
> can actually _be used_ by Flink users. To achieve this the release managers
> ensure that an issue for cross team testing is created in the Apache Flink
> Jira. This can and should be picked up by other community members to verify
> the functionality and usability of the feature.
> The issue should contain some entry points which enables other community
> members to test it. It should not contain documentation on how to use the
> feature as this should be part of the actual documentation. The cross team
> tests are performed after the feature freeze. Documentation should be in
> place before that. Those tests are manual tests, so do not confuse them with
> automated tests.
> To sum that up:
> * User facing features should be tested by other contributors
> * The scope is usability and sanity of the feature
> * The feature needs to be already documented
> * The contributor creates an issue containing some pointers on how to get
> started (e.g. link to the documentation, suggested targets of verification)
> * Other community members pick those issues up and provide feedback
> * Cross team testing happens right after the feature freeze
> --
> h3. Expectations
> * Jira issues for each expected release task according to the release plan is
> created and labeled as {{release-testing}}.
> * All the created release-testing-related Jira issues are resolved and the
> corresponding blocker issues are fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31150) Cross team testing
[
https://issues.apache.org/jira/browse/FLINK-31150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31150:
--
Description:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{{}release-testing{}}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
was:
For user facing features that go into the release we'd like to ensure they can
actually _be used_ by Flink users. To achieve this the release managers ensure
that an issue for cross team testing is created in the Apache Flink Jira. This
can and should be picked up by other community members to verify the
functionality and usability of the feature.
The issue should contain some entry points which enables other community
members to test it. It should not contain documentation on how to use the
feature as this should be part of the actual documentation. The cross team
tests are performed after the feature freeze. Documentation should be in place
before that. Those tests are manual tests, so do not confuse them with
automated tests.
To sum that up:
* User facing features should be tested by other contributors
* The scope is usability and sanity of the feature
* The feature needs to be already documented
* The contributor creates an issue containing some pointers on how to get
started (e.g. link to the documentation, suggested targets of verification)
* Other community members pick those issues up and provide feedback
* Cross team testing happens right after the feature freeze
h3. Expectations
* Jira issues for each expected release task according to the release plan is
created and labeled as {{release-testing}}.
* All the created release-testing-related Jira issues are resolved and the
corresponding blocker issues are fixed.
> Cross team testing
> --
>
> Key: FLINK-31150
> URL: https://issues.apache.org/jira/browse/FLINK-31150
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Qingsheng Ren
>Priority: Major
>
> For user facing features that go into the release we'd like to ensure they
> can actually _be used_ by Flink users. To achieve this the release managers
> ensure that an issue for cross team testing is created in the Apache Flink
> Jira. This can and should be picked up by other community members to verify
> the functionality and usability of the feature.
> The issue should contain some entry points which enables other community
> members to test it. It should not contain documentation on how to use the
> feature as this should be part of the actual documentation. The cross team
> tests are performed after the feature freeze. Documentation should be in
> place before that. Those tests are manual tests, so do not confuse them with
> automated tests.
> To sum that up:
> * User facing features should be tested by other contributors
> * The scope is usability and sanity of the feature
> * The feature needs to be already documented
> * The contributor creates an issue containing some pointers on how to get
> started (e.g. link to the documentation, suggested targets of verification)
> * Other community members pick those issues up and provide feedback
> * Cross team testing happens right after the feature freeze
>
>
> h3. Expectations
> * Jira issues for each expected release task according to the release plan
> is created and labeled as {{{}release-testing{}}}.
> * All the created release-testing-related Jira issues are resolved and the
> corresponding blo
[jira] [Comment Edited] (FLINK-30337) Stateful Functions application throws an exception when stopping a job gracefully creating a final savepoint
[
https://issues.apache.org/jira/browse/FLINK-30337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691499#comment-17691499
]
Alexey Novakov edited comment on FLINK-30337 at 2/21/23 10:17 AM:
--
I have been able to reproduce this issue as well.
Flink CLI Stop command fails with above exception after timeout and there is
one more exception in *statefun-worker* pod:
{code:java}
2023-02-21 10:06:36,818 INFO
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] -
feedback-union -> functions (1/1)#0 - asynchronous part of checkpoint 2 could
not be completed.
java.util.concurrent.CancellationException: null
at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:645)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:57)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:177)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
[flink-dist_2.12-1.14.3.jar:1.14.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]{code}
was (Author: novakov.alex):
I have been able to reproduce this issue as well.
Flink CLI Stop command fails with above exception after timeout and there is
one more exception in *statefun-worker* pod:
{code:java}
2023-02-21 10:06:36,818 INFO
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] -
feedback-union -> functions (1/1)#0 - asynchronous part of checkpoint 2 could
not be completed.
java.util.concurrent.CancellationException: null
at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:645)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:57)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:177)
~[flink-dist_2.12-1.14.3.jar:1.14.3]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
[flink-dist_2.12-1.14.3.jar:1.14.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?] {code}
> Stateful Functions application throws an exception when stopping a job
> gracefully creating a final savepoint
>
>
> Key: FLINK-30337
> URL: https://issues.apache.org/jira/browse/FLINK-30337
> Project: Flink
> Issue Type: Bug
> Components: Stateful Functions
>Affects Versions: statefun-3.2.0
>Reporter: Ali Bahadir Zeybek
>Priority: Minor
>
> When running a Stateful Functions applications, if the stop[1] command is
> executed, the client throws a FlinkException with the following stack trace
> where *953498833da99ec437758b49b7d5befd* is the specific job id:
>
> {code:java}
> The program finished with the following
> exception:org.apache.flink.util.FlinkException: Could not stop with a
> savepoint job "953498833da99ec437758b49b7d5befd".
> at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581)
> at
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
> at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.util.concurrent.TimeoutException
> at java.base/java.util.concurrent.CompletableFuture.timedGet(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java
