Re: [PR] [FLINK-14102] Introduce DB2Dialect. [flink-connector-jdbc]
davidradl commented on PR #8: URL: https://github.com/apache/flink-connector-jdbc/pull/8#issuecomment-1750191357 @MartijnVisser This is something we also would really like to be merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18356) flink-table-planner Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772493#comment-17772493
]
Sergey Nuyanzin commented on FLINK-18356:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53575&view=logs&j=a9db68b9-a7e0-54b6-0f98-010e0aff39e2&t=cdd32e0b-6047-565b-c58f-14054472f1be&l=11720
> flink-table-planner Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0,
> 1.19.0
>Reporter: Piotr Nowojski
>Assignee: Yunhong Zheng
>Priority: Critical
> Labels: pull-request-available, test-stability
> Attachments: 1234.jpg, app-profiling_4.gif,
> image-2023-01-11-22-21-57-784.png, image-2023-01-11-22-22-32-124.png,
> image-2023-02-16-20-18-09-431.png, image-2023-07-11-19-28-52-851.png,
> image-2023-07-11-19-35-54-530.png, image-2023-07-11-19-41-18-626.png,
> image-2023-07-11-19-41-37-105.png
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-18356) flink-table-planner Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772496#comment-17772496
]
Sergey Nuyanzin commented on FLINK-18356:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53578&view=logs&j=a9db68b9-a7e0-54b6-0f98-010e0aff39e2&t=cdd32e0b-6047-565b-c58f-14054472f1be&l=1000
> flink-table-planner Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0,
> 1.19.0
>Reporter: Piotr Nowojski
>Assignee: Yunhong Zheng
>Priority: Critical
> Labels: pull-request-available, test-stability
> Attachments: 1234.jpg, app-profiling_4.gif,
> image-2023-01-11-22-21-57-784.png, image-2023-01-11-22-22-32-124.png,
> image-2023-02-16-20-18-09-431.png, image-2023-07-11-19-28-52-851.png,
> image-2023-07-11-19-35-54-530.png, image-2023-07-11-19-41-18-626.png,
> image-2023-07-11-19-41-37-105.png
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-27076) Setup runners
[ https://issues.apache.org/jira/browse/FLINK-27076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl updated FLINK-27076: -- Description: h2. GitHub Documentation * About self-hosted runners * Adding self-hosted runners * Using self-hosted runners in a GitHub Actions workflow * Security hardening for self-hosted runners * Autoscaling with self-hosted runners * Alternative: Approving workflow runs from public forks: Not suitable for projects with large amounts of contributors (see Slack thread) h2. Apache Infra * Infra ML discussion thread from Apache Pekko * GitHub self-hosted-runners (most-recently updated wiki article) * GitHub Actions status (last modified on Jun 20, 2021) .h2 GitHub Runner * actions/runner * Closed issue #494 * Feature request * Draft PR #783 with fix from @ashb .h2 GitHub Runner Fork from @ashb * ashb/runner (releases/pr-security-options branch) * Automated release of custom GitHub runner - Apparently, we need to deploy the most-recent version of the GitHub runner to avoid having the runner being rejected by GitHub. .h2 Apache Flink ML * GHA migration roadmap (December 16, 2021) * Notice thread on migration to GitHub Actions (April 4, 2022) .h2 Miscellaneous * Apache Pulsar restricts CI runs to reviewed PRs only. Contributors are asked to create a branch in their fork as well to use GitHub’s runners, instead. The project itself relies on Apache’s hosted runners. (see related PR) There is a discussion about it in the follow-ups of this Infra Slack post There are concerns shared about losing contributors due to the extra work that’s necessary to set this up. > Setup runners > - > > Key: FLINK-27076 > URL: https://issues.apache.org/jira/browse/FLINK-27076 > Project: Flink > Issue Type: Sub-task > Components: Build System / CI >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > > h2. GitHub Documentation > * About self-hosted runners > * Adding self-hosted runners > * Using self-hosted runners in a GitHub Actions workflow > * Security hardening for self-hosted runners > * Autoscaling with self-hosted runners > * Alternative: Approving workflow runs from public forks: Not suitable for > projects with large amounts of contributors (see Slack thread) > h2. Apache Infra > * Infra ML discussion thread from Apache Pekko > * GitHub self-hosted-runners (most-recently updated wiki article) > * GitHub Actions status (last modified on Jun 20, 2021) > .h2 GitHub Runner > * actions/runner > * Closed issue #494 > * Feature request > * Draft PR #783 with fix from @ashb > .h2 GitHub Runner Fork from @ashb > * ashb/runner (releases/pr-security-options branch) > * Automated release of custom GitHub runner - Apparently, we need to deploy > the most-recent version of the GitHub runner to avoid having the runner being > rejected by GitHub. > .h2 Apache Flink ML > * GHA migration roadmap (December 16, 2021) > * Notice thread on migration to GitHub Actions (April 4, 2022) > .h2 Miscellaneous > * Apache Pulsar restricts CI runs to reviewed PRs only. Contributors are > asked to create a branch in their fork as well to use GitHub’s runners, > instead. The project itself relies on Apache’s hosted runners. (see related > PR) > There is a discussion about it in the follow-ups of this Infra Slack post > There are concerns shared about losing contributors due to the extra work > that’s necessary to set this up. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27076) Setup runners
[ https://issues.apache.org/jira/browse/FLINK-27076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl updated FLINK-27076: -- Description: h2. GitHub Documentation * [About self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners] * [Adding self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners] * [Using self-hosted runners in a GitHub Actions workflow|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-self-hosted-runners-in-a-workflow] * [Security hardening for self-hosted runners|https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners] * [Autoscaling with self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners] * Alternative: [Approving workflow runs from public forks|https://docs.github.com/en/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks] (not suitable for projects with large amounts of contributors; see [Slack thread|https://the-asf.slack.com/archives/CBX4TSBQ8/p1640076905230600]) h2. Apache Infra * [Infra ML discussion thread from Apache Pekko|https://cwiki.apache.org/confluence/display/INFRA/GitHub+self-hosted+runners] * [GitHub self-hosted-runners|https://cwiki.apache.org/confluence/display/INFRA/GitHub+self-hosted+runners] (most-recently updated wiki article) * [GitHub Actions status|https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status] (last modified on Jun 20, 2021) h2. GitHub Runner * [actions/runner|https://github.com/actions/runner] * [Closed issue #494|https://github.com/actions/runner/issues/494] * [Feature request|https://github.com/orgs/community/discussions/53430] * [Draft PR #783|https://github.com/actions/runner/pull/783] with fix from [@ashb|https://github.com/ashb] h2. GitHub Runner Fork from [@ashb|https://github.com/ashb] * [ashb/runner|https://github.com/ashb/runner] ([releases/pr-security-options|https://github.com/actions/runner/compare/main...ashb:runner:releases/pr-security-options] branch) * [Automated release of custom GitHub runner|https://github.com/ashb/runner/blob/main/.github/workflows/sync-with-upstream.yml] - Apparently, we need to deploy the most-recent version of the GitHub runner to avoid having the runner being rejected by GitHub. h2. Apache Flink ML * [GHA migration roadmap|https://lists.apache.org/thread/vcyx2nx0mhklqwm827vgykv8pc54gg3k] (December 16, 2021) * [Notice thread on migration to GitHub Actions|https://lists.apache.org/thread/438krxbs7y9lg0f009f1hlojqm54y57s] (April 4, 2022) h2. Miscellaneous * Apache Pulsar restricts CI runs to reviewed PRs only. Contributors are asked to create a branch in their fork as well to use GitHub’s runners, instead. The project itself relies on Apache’s hosted runners. (see [related PR|https://github.com/apache/pulsar/pull/17693/files]) ** There is a discussion about it in the follow-ups of [this Infra Slack post|https://the-asf.slack.com/archives/CBX4TSBQ8/p1663346680496919?thread_ts=1661512133.913279&cid=CBX4TSBQ8] ** There are concerns shared about losing contributors due to the extra work that’s necessary to set this up. was: h2. GitHub Documentation * About self-hosted runners * Adding self-hosted runners * Using self-hosted runners in a GitHub Actions workflow * Security hardening for self-hosted runners * Autoscaling with self-hosted runners * Alternative: Approving workflow runs from public forks: Not suitable for projects with large amounts of contributors (see Slack thread) h2. Apache Infra * Infra ML discussion thread from Apache Pekko * GitHub self-hosted-runners (most-recently updated wiki article) * GitHub Actions status (last modified on Jun 20, 2021) .h2 GitHub Runner * actions/runner * Closed issue #494 * Feature request * Draft PR #783 with fix from @ashb .h2 GitHub Runner Fork from @ashb * ashb/runner (releases/pr-security-options branch) * Automated release of custom GitHub runner - Apparently, we need to deploy the most-recent version of the GitHub runner to avoid having the runner being rejected by GitHub. .h2 Apache Flink ML * GHA migration roadmap (December 16, 2021) * Notice thread on migration to GitHub Actions (April 4, 2022) .h2 Miscellaneous * Apache Pulsar restricts CI runs to reviewed PRs only. Contributors are asked to create a branch in their fork as well to use GitHub’s runners, instead. The project itself relies on Apache’s hosted runners. (see related PR) There is a discussion about it in the follow-ups of this Infra Slack post There are concerns shared about losing contributors due to the extra work that’s necessary to set this up. > Setup runners
[jira] [Updated] (FLINK-27076) Setup runners
[ https://issues.apache.org/jira/browse/FLINK-27076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl updated FLINK-27076: -- Description: Below is a collection of links around the topic of self-hosted runners and the current state of the GitHub Actions migration efforts in Apache Flink. h2. GitHub Documentation * [About self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners] * [Adding self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners] * [Using self-hosted runners in a GitHub Actions workflow|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-self-hosted-runners-in-a-workflow] * [Security hardening for self-hosted runners|https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners] * [Autoscaling with self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners] * Alternative: [Approving workflow runs from public forks|https://docs.github.com/en/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks] (not suitable for projects with large amounts of contributors; see [Slack thread|https://the-asf.slack.com/archives/CBX4TSBQ8/p1640076905230600]) h2. Apache Infra * [Infra ML discussion thread from Apache Pekko|https://cwiki.apache.org/confluence/display/INFRA/GitHub+self-hosted+runners] * [GitHub self-hosted-runners|https://cwiki.apache.org/confluence/display/INFRA/GitHub+self-hosted+runners] (most-recently updated wiki article) * [GitHub Actions status|https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status] (last modified on Jun 20, 2021) h2. GitHub Runner * [actions/runner|https://github.com/actions/runner] * [Closed issue #494|https://github.com/actions/runner/issues/494] * [Feature request|https://github.com/orgs/community/discussions/53430] * [Draft PR #783|https://github.com/actions/runner/pull/783] with fix from [@ashb|https://github.com/ashb] h2. GitHub Runner Fork from [@ashb|https://github.com/ashb] * [ashb/runner|https://github.com/ashb/runner] ([releases/pr-security-options|https://github.com/actions/runner/compare/main...ashb:runner:releases/pr-security-options] branch) * [Automated release of custom GitHub runner|https://github.com/ashb/runner/blob/main/.github/workflows/sync-with-upstream.yml] - Apparently, we need to deploy the most-recent version of the GitHub runner to avoid having the runner being rejected by GitHub. h2. Apache Flink ML * [GHA migration roadmap|https://lists.apache.org/thread/vcyx2nx0mhklqwm827vgykv8pc54gg3k] (December 16, 2021) * [Notice thread on migration to GitHub Actions|https://lists.apache.org/thread/438krxbs7y9lg0f009f1hlojqm54y57s] (April 4, 2022) h2. Miscellaneous * Apache Pulsar restricts CI runs to reviewed PRs only. Contributors are asked to create a branch in their fork as well to use GitHub’s runners, instead. The project itself relies on Apache’s hosted runners. (see [related PR|https://github.com/apache/pulsar/pull/17693/files]) ** There is a discussion about it in the follow-ups of [this Infra Slack post|https://the-asf.slack.com/archives/CBX4TSBQ8/p1663346680496919?thread_ts=1661512133.913279&cid=CBX4TSBQ8] ** There are concerns shared about losing contributors due to the extra work that’s necessary to set this up. was: h2. GitHub Documentation * [About self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners] * [Adding self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners] * [Using self-hosted runners in a GitHub Actions workflow|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-self-hosted-runners-in-a-workflow] * [Security hardening for self-hosted runners|https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners] * [Autoscaling with self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners] * Alternative: [Approving workflow runs from public forks|https://docs.github.com/en/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks] (not suitable for projects with large amounts of contributors; see [Slack thread|https://the-asf.slack.com/archives/CBX4TSBQ8/p1640076905230600]) h2. Apache Infra * [Infra ML discussion thread from Apache Pekko|https://cwiki.apache.org/confluence/display/INFRA/GitHub+self-hosted+runners] * [GitHub self-hosted-runners|ht
[jira] [Created] (FLINK-33197) PyFlink support for ByteArraySchema
Liu Chong created FLINK-33197: - Summary: PyFlink support for ByteArraySchema Key: FLINK-33197 URL: https://issues.apache.org/jira/browse/FLINK-33197 Project: Flink Issue Type: New Feature Components: API / Python Affects Versions: 1.17.0 Reporter: Liu Chong Currently in Python Flink API, when reading messages from a Kafka source, only SimpleStringSchema is available. If the data is in arbitary binary format(e.g. marshalled Protocol Buffer msg) it may not be decodable with the default 'utf-8' encoding. There's currently a workaround which is to manually set the encoding to 'ISO-8859-1' which supports all possible byte combinations. However this is not an elegant solution. We should support ByteArraySchema which outputs a raw byte array for subsequent unmarshalling. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27076) Setup runners
[ https://issues.apache.org/jira/browse/FLINK-27076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl updated FLINK-27076: -- Description: Below is a collection of links around the topic of self-hosted runners and the current state of the GitHub Actions migration efforts in Apache Flink. h2. GitHub Documentation * [About self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners] * [Adding self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners] * [Using self-hosted runners in a GitHub Actions workflow|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-self-hosted-runners-in-a-workflow] * [Security hardening for self-hosted runners|https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners] * [Autoscaling with self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners] * Alternative: [Approving workflow runs from public forks|https://docs.github.com/en/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks] (not suitable for projects with large amounts of contributors; see [Slack thread|https://the-asf.slack.com/archives/CBX4TSBQ8/p1640076905230600]) h2. Apache Infra * [Infra ML discussion thread from Apache Pekko|https://cwiki.apache.org/confluence/display/INFRA/GitHub+self-hosted+runners] * [GitHub self-hosted-runners|https://cwiki.apache.org/confluence/display/INFRA/GitHub+self-hosted+runners] (most-recently updated wiki article) * [GitHub Actions status|https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status] (last modified on Jun 20, 2021) h2. GitHub Runner * [actions/runner|https://github.com/actions/runner] * [Closed issue #494|https://github.com/actions/runner/issues/494] * [Feature request|https://github.com/orgs/community/discussions/53430] * [Draft PR #783|https://github.com/actions/runner/pull/783] with fix from [@ashb|https://github.com/ashb] h2. GitHub Runner Fork from [@ashb|https://github.com/ashb] * [ashb/runner|https://github.com/ashb/runner] ([releases/pr-security-options|https://github.com/actions/runner/compare/main...ashb:runner:releases/pr-security-options] branch) * [Automated release of custom GitHub runner|https://github.com/ashb/runner/blob/main/.github/workflows/sync-with-upstream.yml] - Apparently, we need to deploy the most-recent version of the GitHub runner to avoid having the runner being rejected by GitHub. h2. Apache Flink * ML: [GHA migration roadmap|https://lists.apache.org/thread/vcyx2nx0mhklqwm827vgykv8pc54gg3k] (December 16, 2021) * ML: [Notice thread on migration to GitHub Actions|https://lists.apache.org/thread/438krxbs7y9lg0f009f1hlojqm54y57s] (April 4, 2022) * [exp_github_actions branch|https://github.com/apache/flink/tree/exp_github_actions] in Apache Flink repo h2. Miscellaneous * Apache Pulsar restricts CI runs to reviewed PRs only. Contributors are asked to create a branch in their fork as well to use GitHub’s runners, instead. The project itself relies on Apache’s hosted runners. (see [related PR|https://github.com/apache/pulsar/pull/17693/files]) ** There is a discussion about it in the follow-ups of [this Infra Slack post|https://the-asf.slack.com/archives/CBX4TSBQ8/p1663346680496919?thread_ts=1661512133.913279&cid=CBX4TSBQ8] ** There are concerns shared about losing contributors due to the extra work that’s necessary to set this up. was: Below is a collection of links around the topic of self-hosted runners and the current state of the GitHub Actions migration efforts in Apache Flink. h2. GitHub Documentation * [About self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners] * [Adding self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners] * [Using self-hosted runners in a GitHub Actions workflow|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-self-hosted-runners-in-a-workflow] * [Security hardening for self-hosted runners|https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners] * [Autoscaling with self-hosted runners|https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners] * Alternative: [Approving workflow runs from public forks|https://docs.github.com/en/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks] (not suitable for projects with large amounts of contributors
[jira] [Comment Edited] (FLINK-33104) Nightly run for Flink Kafka connector fails due to architecture tests failing
[
https://issues.apache.org/jira/browse/FLINK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770338#comment-17770338
]
Etienne Chauchot edited comment on FLINK-33104 at 10/6/23 10:27 AM:
[~martijnvisser] actually it was never fixed in cassandra (see FLINK-32353):
the problem is that when archunit rules change we need to update the archunit
violation store. And there is only a single violation store. As the nightly
tests the connector against several versions of flink, there will be failures
if not all these versions have the same archunit rules. To fix this problem we
need to skip archunit tests on Flink versions that are not the main one (the
one the connector is built against) so that the single arunit violation store
contains the violations for the main flink version. I proposed some change in
the github action script
[here|https://issues.apache.org/jira/browse/FLINK-32563] for which I was
waiting for your feedback.
was (Author: echauchot):
[~martijnvisser] actually it was never fixed in cassandra (see FLINK-32353):
the problem is that when archunit rules change we need to update the archunit
violation store. And there is only a single violation store. As the nightly
tests the connector against several versions of flink, there will be failures
if not all these versions have the same archunit rules. To fix this problem we
need to skip archunit tests on Flink versions that are not the main one (the
one the connector is built against) so that the single arunit violation store
contains the violations for the main flink version. I proposed some change in
the github action script here for which I was waiting for your feedback.
> Nightly run for Flink Kafka connector fails due to architecture tests failing
> -
>
> Key: FLINK-33104
> URL: https://issues.apache.org/jira/browse/FLINK-33104
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: kafka-3.1.0
>Reporter: Martijn Visser
>Priority: Blocker
>
> {code:java}
> 2023-09-17T00:29:07.1675694Z [WARNING] Tests run: 18, Failures: 0, Errors: 0,
> Skipped: 9, Time elapsed: 308.532 s - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerMigrationTest
> 2023-09-17T00:29:07.5171608Z [INFO]
> 2023-09-17T00:29:07.5172360Z [INFO] Results:
> 2023-09-17T00:29:07.5172773Z [INFO]
> 2023-09-17T00:29:07.5173139Z [ERROR] Failures:
> 2023-09-17T00:29:07.5174181Z [ERROR] Architecture Violation [Priority:
> MEDIUM] - Rule 'ITCASE tests should use a MiniCluster resource or extension'
> was violated (13 times):
> 2023-09-17T00:29:07.5176050Z
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase does
> not satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5177452Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5179831Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5181277Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5182154Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5182951Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5183906Z
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase does not satisfy: only
> one of the following predicates match:
> 2023-09-17T00:29:07.5184769Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5185812Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5186880Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.518792
[jira] [Commented] (FLINK-33104) Nightly run for Flink Kafka connector fails due to architecture tests failing
[
https://issues.apache.org/jira/browse/FLINK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772516#comment-17772516
]
Etienne Chauchot commented on FLINK-33104:
--
[~tzulitai] yes this is exactly what I wanted to do for Cassandra but it
requires a change in the github actions scripts and/or on main build proposed
[here|https://issues.apache.org/jira/browse/FLINK-32563] but I received no
feedback. Among the 3 proposed changes in that ticket, I prefer the one that
links the dep convergence tests and the archunit tests into a single maven
profile (like architecture tests). I'll do this change in a PR.
> Nightly run for Flink Kafka connector fails due to architecture tests failing
> -
>
> Key: FLINK-33104
> URL: https://issues.apache.org/jira/browse/FLINK-33104
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: kafka-3.1.0
>Reporter: Martijn Visser
>Priority: Blocker
>
> {code:java}
> 2023-09-17T00:29:07.1675694Z [WARNING] Tests run: 18, Failures: 0, Errors: 0,
> Skipped: 9, Time elapsed: 308.532 s - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerMigrationTest
> 2023-09-17T00:29:07.5171608Z [INFO]
> 2023-09-17T00:29:07.5172360Z [INFO] Results:
> 2023-09-17T00:29:07.5172773Z [INFO]
> 2023-09-17T00:29:07.5173139Z [ERROR] Failures:
> 2023-09-17T00:29:07.5174181Z [ERROR] Architecture Violation [Priority:
> MEDIUM] - Rule 'ITCASE tests should use a MiniCluster resource or extension'
> was violated (13 times):
> 2023-09-17T00:29:07.5176050Z
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase does
> not satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5177452Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5179831Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5181277Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5182154Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5182951Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5183906Z
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase does not satisfy: only
> one of the following predicates match:
> 2023-09-17T00:29:07.5184769Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5185812Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5186880Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5187929Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5189073Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5190076Z
> org.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase does not
> satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5190946Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5191983Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated wit
Re: [PR] [FLINK-32353] [hotfix] build against last flink 1.16 and 1.17 and enable archunit only on main flink release [flink-connector-cassandra]
echauchot commented on PR #19: URL: https://github.com/apache/flink-connector-cassandra/pull/19#issuecomment-1750384738 Hi @tzulitai yes I fell back this PR's scope to just upgrading the versions hence the self merge. But, as I just wrote in [this ticket](https://issues.apache.org/jira/browse/FLINK-33104?focusedCommentId=17772516&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17772516), I'll submit a new PR to change the build system and tackle this problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33159] [build] use variables in pom file for Java and Maven versions [flink]
davidradl commented on code in PR #23469: URL: https://github.com/apache/flink/pull/23469#discussion_r1348594028 ## flink-walkthroughs/flink-walkthrough-datastream-java/src/main/resources/archetype-resources/pom.xml: ## @@ -31,7 +31,6 @@ under the License. UTF-8 @project.version@ - 1.8 Review Comment: no worries - sorry misunderstood - I will reinstate this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33159] [build] use variables in pom file for Java and Maven versions [flink]
davidradl commented on code in PR #23469: URL: https://github.com/apache/flink/pull/23469#discussion_r1348594028 ## flink-walkthroughs/flink-walkthrough-datastream-java/src/main/resources/archetype-resources/pom.xml: ## @@ -31,7 +31,6 @@ under the License. UTF-8 @project.version@ - 1.8 Review Comment: no worries - sorry I misunderstood - I will reinstate this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33046][FLIP-333] Redesign Apache Flink website [flink-web]
dannycranmer commented on PR #676: URL: https://github.com/apache/flink-web/pull/676#issuecomment-1750463733 > Not sure if it's intended, but when hovering over the Flink logo in top left, then "Apache Flink" becomes visible (in dark mode). I think we should always display it, because the sudden appearance after a hover feels wrong? This is not intentional. I cannot reproduce this, I am also using Chrome on Mac. Any other details you can provide to help me reproduce? Chrome version? > The menu isn't on the left side like in the FLIP, but on the right hand side We put it on the left initially and internal feedback was that users thought it was a nav, rather than page index. We moved to right as per this feedback, this also aligns with the previous website. What do you think? I can raise this on the discussion thread if you think it is worth it. > When a link has been visited, we're using a different color in the menu. Can we just keep the same color as it is initially? This looks messy Ack, I agree, will do. How about link within a page? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] add announcement blog post for Flink 1.18 [flink-web]
knaufk opened a new pull request, #680: URL: https://github.com/apache/flink-web/pull/680 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33159] [build] use variables in pom file for Java and Maven versions [flink]
davidradl commented on PR #23469: URL: https://github.com/apache/flink/pull/23469#issuecomment-1750644172 @ferenc-csaky thanks I have made the requested change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32380] Support Java Records with PojoTypeInfo/Serializer [flink]
gyfora commented on PR #23490: URL: https://github.com/apache/flink/pull/23490#issuecomment-1750668456 I just realised that the current implementation won't work with schema evolution as we have to "remap" the parameter indexes for the constructor afterwards, also defining new primitive fields can be problematic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] add announcement blog post for Flink 1.18 [flink-web]
davidradl commented on code in PR #680: URL: https://github.com/apache/flink-web/pull/680#discussion_r1348732092 ## docs/content/posts/2023-10-10-release-1.18.0.md: ## @@ -0,0 +1,542 @@ +--- +authors: +- JingGe: + name: "Jing Ge" + twitter: jingengineer +- KonstantinKnauf: + name: "Konstantin Knauf" + twitter: snntrable +- SergeyNuyanzin: + name: "Sergey Nuyanzin" + twitter: uckamello +- QingshengRen: + name: "Qingsheng Ren" + twitter: renqstuite +date: "2023-10-10T08:00:00Z" +subtitle: "" +title: Announcing the Release of Apache Flink 1.18 +aliases: +- /news/2023/10/10/release-1.18.0.html +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 1.18.0. As usual, we are looking at a packed +release with a wide variety of improvements and new features. Overall, 176 people contributed to this release completing +18 FLIPS and 700+ issues. Thank you! + +Let's dive into the highlights. + +# Towards a Streaming Lakehouse + +## Flink SQL Improvements + +### Introduce Flink JDBC Driver For Sql Gateway + +Flink 1.18 comes with a JDBC Driver for the Flink SQL Gateway. So, you can now use any SQL Client that supports JDBC to +interact with your tables via Flink SQL. Here is an example using [SQLLine](https://julianhyde.github.io/sqlline/manual.html). + +```shell +sqlline> !connect jdbc:flink://localhost:8083 +``` + +```shell +sqlline version 1.12.0 +sqlline> !connect jdbc:flink://localhost:8083 +Enter username for jdbc:flink://localhost:8083: +Enter password for jdbc:flink://localhost:8083: +0: jdbc:flink://localhost:8083> CREATE TABLE T( +. . . . . . . . . . . . . . .)> a INT, +. . . . . . . . . . . . . . .)> b VARCHAR(10) +. . . . . . . . . . . . . . .)> ) WITH ( +. . . . . . . . . . . . . . .)> 'connector' = 'filesystem', +. . . . . . . . . . . . . . .)> 'path' = 'file:///tmp/T.csv', +. . . . . . . . . . . . . . .)> 'format' = 'csv' +. . . . . . . . . . . . . . .)> ); +No rows affected (0.122 seconds) +0: jdbc:flink://localhost:8083> INSERT INTO T VALUES (1, 'Hi'), (2, 'Hello'); ++--+ +| job id | ++--+ +| fbade1ab4450fc57ebd5269fdf60dcfd | ++--+ +1 row selected (1.282 seconds) +0: jdbc:flink://localhost:8083> SELECT * FROM T; ++---+---+ +| a | b | ++---+---+ +| 1 | Hi| +| 2 | Hello | ++---+---+ +2 rows selected (1.955 seconds) +0: jdbc:flink://localhost:8083> +``` + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/) +* [FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway](https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway) + + +### Stored Procedures + +Stored Procedures provide a convenient way to encapsulate complex logic to perform data manipulation or administrative +tasks in Apache Flink itself. Therefore， Flink introduces the support for calling stored procedures. +Flink now allows catalog developers to develop their own built-in stored procedures and then enables users to call these +predefined stored procedures. + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/procedures/) +* [FLIP-311: Support Call Stored Procedure](https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure) + +### Extended DDL Support + +From this release onwards, Flink supports + +- `REPLACE TABLE AS SELECT` +- `CREATE OR REPLACE TABLE AS SELECT` + +and both these commands and previously supported `CREATE TABLE AS` can now support atomicity provided the underlying +connector supports this. + +Moreover, Apache Flink now supports TRUNCATE TABLE in batch execution mode. As before, the underlying connector needs +to implement and provide this capability + +And, finally, we have also added support for adding, dropping and listing partitions via + +- `ALTER TABLE ADD PARTITION` +- `ALTER TABLE DROP PARTITION` +- `SHOW PARTITIONS` + +**More Information** +- [Documentation on TRUNCATE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/truncate/) +- [Documentation on CREATE OR REPLACE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-or-replace-table) +- [Documentation on ALTER TABLE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#alter-table) +- [FLIP-302: Support TRUNCATE TABLE statement in batch mode](https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement+in+batch+mode) +- [FLIP-303: Support REPLACE TABLE AS SELECT statement](https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement) +- [FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement](https://cwiki.apache.org/confluence/display/FLINK
Re: [PR] add announcement blog post for Flink 1.18 [flink-web]
davidradl commented on code in PR #680: URL: https://github.com/apache/flink-web/pull/680#discussion_r1348735360 ## docs/content/posts/2023-10-10-release-1.18.0.md: ## @@ -0,0 +1,542 @@ +--- +authors: +- JingGe: + name: "Jing Ge" + twitter: jingengineer +- KonstantinKnauf: + name: "Konstantin Knauf" + twitter: snntrable +- SergeyNuyanzin: + name: "Sergey Nuyanzin" + twitter: uckamello +- QingshengRen: + name: "Qingsheng Ren" + twitter: renqstuite +date: "2023-10-10T08:00:00Z" +subtitle: "" +title: Announcing the Release of Apache Flink 1.18 +aliases: +- /news/2023/10/10/release-1.18.0.html +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 1.18.0. As usual, we are looking at a packed +release with a wide variety of improvements and new features. Overall, 176 people contributed to this release completing +18 FLIPS and 700+ issues. Thank you! + +Let's dive into the highlights. + +# Towards a Streaming Lakehouse + +## Flink SQL Improvements + +### Introduce Flink JDBC Driver For Sql Gateway + +Flink 1.18 comes with a JDBC Driver for the Flink SQL Gateway. So, you can now use any SQL Client that supports JDBC to +interact with your tables via Flink SQL. Here is an example using [SQLLine](https://julianhyde.github.io/sqlline/manual.html). + +```shell +sqlline> !connect jdbc:flink://localhost:8083 +``` + +```shell +sqlline version 1.12.0 +sqlline> !connect jdbc:flink://localhost:8083 +Enter username for jdbc:flink://localhost:8083: +Enter password for jdbc:flink://localhost:8083: +0: jdbc:flink://localhost:8083> CREATE TABLE T( +. . . . . . . . . . . . . . .)> a INT, +. . . . . . . . . . . . . . .)> b VARCHAR(10) +. . . . . . . . . . . . . . .)> ) WITH ( +. . . . . . . . . . . . . . .)> 'connector' = 'filesystem', +. . . . . . . . . . . . . . .)> 'path' = 'file:///tmp/T.csv', +. . . . . . . . . . . . . . .)> 'format' = 'csv' +. . . . . . . . . . . . . . .)> ); +No rows affected (0.122 seconds) +0: jdbc:flink://localhost:8083> INSERT INTO T VALUES (1, 'Hi'), (2, 'Hello'); ++--+ +| job id | ++--+ +| fbade1ab4450fc57ebd5269fdf60dcfd | ++--+ +1 row selected (1.282 seconds) +0: jdbc:flink://localhost:8083> SELECT * FROM T; ++---+---+ +| a | b | ++---+---+ +| 1 | Hi| +| 2 | Hello | ++---+---+ +2 rows selected (1.955 seconds) +0: jdbc:flink://localhost:8083> +``` + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/) +* [FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway](https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway) + + +### Stored Procedures + +Stored Procedures provide a convenient way to encapsulate complex logic to perform data manipulation or administrative +tasks in Apache Flink itself. Therefore， Flink introduces the support for calling stored procedures. +Flink now allows catalog developers to develop their own built-in stored procedures and then enables users to call these +predefined stored procedures. + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/procedures/) +* [FLIP-311: Support Call Stored Procedure](https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure) + +### Extended DDL Support + +From this release onwards, Flink supports + +- `REPLACE TABLE AS SELECT` +- `CREATE OR REPLACE TABLE AS SELECT` + +and both these commands and previously supported `CREATE TABLE AS` can now support atomicity provided the underlying +connector supports this. + +Moreover, Apache Flink now supports TRUNCATE TABLE in batch execution mode. As before, the underlying connector needs +to implement and provide this capability + +And, finally, we have also added support for adding, dropping and listing partitions via + +- `ALTER TABLE ADD PARTITION` +- `ALTER TABLE DROP PARTITION` +- `SHOW PARTITIONS` + +**More Information** +- [Documentation on TRUNCATE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/truncate/) +- [Documentation on CREATE OR REPLACE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-or-replace-table) +- [Documentation on ALTER TABLE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#alter-table) +- [FLIP-302: Support TRUNCATE TABLE statement in batch mode](https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement+in+batch+mode) +- [FLIP-303: Support REPLACE TABLE AS SELECT statement](https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement) +- [FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement](https://cwiki.apache.org/confluence/display/FLINK
Re: [PR] add announcement blog post for Flink 1.18 [flink-web]
snuyanzin commented on code in PR #680: URL: https://github.com/apache/flink-web/pull/680#discussion_r1348755495 ## docs/content/posts/2023-10-10-release-1.18.0.md: ## @@ -0,0 +1,542 @@ +--- +authors: +- JingGe: + name: "Jing Ge" + twitter: jingengineer +- KonstantinKnauf: + name: "Konstantin Knauf" + twitter: snntrable +- SergeyNuyanzin: + name: "Sergey Nuyanzin" + twitter: uckamello +- QingshengRen: + name: "Qingsheng Ren" + twitter: renqstuite +date: "2023-10-10T08:00:00Z" +subtitle: "" +title: Announcing the Release of Apache Flink 1.18 +aliases: +- /news/2023/10/10/release-1.18.0.html +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 1.18.0. As usual, we are looking at a packed +release with a wide variety of improvements and new features. Overall, 176 people contributed to this release completing +18 FLIPS and 700+ issues. Thank you! + +Let's dive into the highlights. + +# Towards a Streaming Lakehouse + +## Flink SQL Improvements + +### Introduce Flink JDBC Driver For Sql Gateway + +Flink 1.18 comes with a JDBC Driver for the Flink SQL Gateway. So, you can now use any SQL Client that supports JDBC to +interact with your tables via Flink SQL. Here is an example using [SQLLine](https://julianhyde.github.io/sqlline/manual.html). Review Comment: based on license it should be ok this tool is used by Apache Calcite, Apache Drill, APache Phoenix and other projects There is no new features and that's the reason it was not updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] add announcement blog post for Flink 1.18 [flink-web]
snuyanzin commented on code in PR #680: URL: https://github.com/apache/flink-web/pull/680#discussion_r1348759823 ## docs/content/posts/2023-10-10-release-1.18.0.md: ## @@ -0,0 +1,542 @@ +--- +authors: +- JingGe: + name: "Jing Ge" + twitter: jingengineer +- KonstantinKnauf: + name: "Konstantin Knauf" + twitter: snntrable +- SergeyNuyanzin: + name: "Sergey Nuyanzin" + twitter: uckamello +- QingshengRen: + name: "Qingsheng Ren" + twitter: renqstuite +date: "2023-10-10T08:00:00Z" +subtitle: "" +title: Announcing the Release of Apache Flink 1.18 +aliases: +- /news/2023/10/10/release-1.18.0.html +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 1.18.0. As usual, we are looking at a packed +release with a wide variety of improvements and new features. Overall, 176 people contributed to this release completing +18 FLIPS and 700+ issues. Thank you! + +Let's dive into the highlights. + +# Towards a Streaming Lakehouse + +## Flink SQL Improvements + +### Introduce Flink JDBC Driver For Sql Gateway + +Flink 1.18 comes with a JDBC Driver for the Flink SQL Gateway. So, you can now use any SQL Client that supports JDBC to +interact with your tables via Flink SQL. Here is an example using [SQLLine](https://julianhyde.github.io/sqlline/manual.html). + +```shell +sqlline> !connect jdbc:flink://localhost:8083 +``` + +```shell +sqlline version 1.12.0 +sqlline> !connect jdbc:flink://localhost:8083 +Enter username for jdbc:flink://localhost:8083: +Enter password for jdbc:flink://localhost:8083: +0: jdbc:flink://localhost:8083> CREATE TABLE T( +. . . . . . . . . . . . . . .)> a INT, +. . . . . . . . . . . . . . .)> b VARCHAR(10) +. . . . . . . . . . . . . . .)> ) WITH ( +. . . . . . . . . . . . . . .)> 'connector' = 'filesystem', +. . . . . . . . . . . . . . .)> 'path' = 'file:///tmp/T.csv', +. . . . . . . . . . . . . . .)> 'format' = 'csv' +. . . . . . . . . . . . . . .)> ); +No rows affected (0.122 seconds) +0: jdbc:flink://localhost:8083> INSERT INTO T VALUES (1, 'Hi'), (2, 'Hello'); ++--+ +| job id | ++--+ +| fbade1ab4450fc57ebd5269fdf60dcfd | ++--+ +1 row selected (1.282 seconds) +0: jdbc:flink://localhost:8083> SELECT * FROM T; ++---+---+ +| a | b | ++---+---+ +| 1 | Hi| +| 2 | Hello | ++---+---+ +2 rows selected (1.955 seconds) +0: jdbc:flink://localhost:8083> +``` + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/) +* [FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway](https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway) + + +### Stored Procedures + +Stored Procedures provide a convenient way to encapsulate complex logic to perform data manipulation or administrative +tasks in Apache Flink itself. Therefore， Flink introduces the support for calling stored procedures. +Flink now allows catalog developers to develop their own built-in stored procedures and then enables users to call these +predefined stored procedures. + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/procedures/) +* [FLIP-311: Support Call Stored Procedure](https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure) + +### Extended DDL Support + +From this release onwards, Flink supports + +- `REPLACE TABLE AS SELECT` +- `CREATE OR REPLACE TABLE AS SELECT` + +and both these commands and previously supported `CREATE TABLE AS` can now support atomicity provided the underlying +connector supports this. + +Moreover, Apache Flink now supports TRUNCATE TABLE in batch execution mode. As before, the underlying connector needs +to implement and provide this capability + +And, finally, we have also added support for adding, dropping and listing partitions via + +- `ALTER TABLE ADD PARTITION` +- `ALTER TABLE DROP PARTITION` +- `SHOW PARTITIONS` + +**More Information** +- [Documentation on TRUNCATE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/truncate/) +- [Documentation on CREATE OR REPLACE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-or-replace-table) +- [Documentation on ALTER TABLE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#alter-table) +- [FLIP-302: Support TRUNCATE TABLE statement in batch mode](https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement+in+batch+mode) +- [FLIP-303: Support REPLACE TABLE AS SELECT statement](https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement) +- [FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement](https://cwiki.apache.org/confluence/display/FLINK
Re: [PR] add announcement blog post for Flink 1.18 [flink-web]
snuyanzin commented on code in PR #680: URL: https://github.com/apache/flink-web/pull/680#discussion_r1348759823 ## docs/content/posts/2023-10-10-release-1.18.0.md: ## @@ -0,0 +1,542 @@ +--- +authors: +- JingGe: + name: "Jing Ge" + twitter: jingengineer +- KonstantinKnauf: + name: "Konstantin Knauf" + twitter: snntrable +- SergeyNuyanzin: + name: "Sergey Nuyanzin" + twitter: uckamello +- QingshengRen: + name: "Qingsheng Ren" + twitter: renqstuite +date: "2023-10-10T08:00:00Z" +subtitle: "" +title: Announcing the Release of Apache Flink 1.18 +aliases: +- /news/2023/10/10/release-1.18.0.html +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 1.18.0. As usual, we are looking at a packed +release with a wide variety of improvements and new features. Overall, 176 people contributed to this release completing +18 FLIPS and 700+ issues. Thank you! + +Let's dive into the highlights. + +# Towards a Streaming Lakehouse + +## Flink SQL Improvements + +### Introduce Flink JDBC Driver For Sql Gateway + +Flink 1.18 comes with a JDBC Driver for the Flink SQL Gateway. So, you can now use any SQL Client that supports JDBC to +interact with your tables via Flink SQL. Here is an example using [SQLLine](https://julianhyde.github.io/sqlline/manual.html). + +```shell +sqlline> !connect jdbc:flink://localhost:8083 +``` + +```shell +sqlline version 1.12.0 +sqlline> !connect jdbc:flink://localhost:8083 +Enter username for jdbc:flink://localhost:8083: +Enter password for jdbc:flink://localhost:8083: +0: jdbc:flink://localhost:8083> CREATE TABLE T( +. . . . . . . . . . . . . . .)> a INT, +. . . . . . . . . . . . . . .)> b VARCHAR(10) +. . . . . . . . . . . . . . .)> ) WITH ( +. . . . . . . . . . . . . . .)> 'connector' = 'filesystem', +. . . . . . . . . . . . . . .)> 'path' = 'file:///tmp/T.csv', +. . . . . . . . . . . . . . .)> 'format' = 'csv' +. . . . . . . . . . . . . . .)> ); +No rows affected (0.122 seconds) +0: jdbc:flink://localhost:8083> INSERT INTO T VALUES (1, 'Hi'), (2, 'Hello'); ++--+ +| job id | ++--+ +| fbade1ab4450fc57ebd5269fdf60dcfd | ++--+ +1 row selected (1.282 seconds) +0: jdbc:flink://localhost:8083> SELECT * FROM T; ++---+---+ +| a | b | ++---+---+ +| 1 | Hi| +| 2 | Hello | ++---+---+ +2 rows selected (1.955 seconds) +0: jdbc:flink://localhost:8083> +``` + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/) +* [FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway](https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway) + + +### Stored Procedures + +Stored Procedures provide a convenient way to encapsulate complex logic to perform data manipulation or administrative +tasks in Apache Flink itself. Therefore， Flink introduces the support for calling stored procedures. +Flink now allows catalog developers to develop their own built-in stored procedures and then enables users to call these +predefined stored procedures. + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/procedures/) +* [FLIP-311: Support Call Stored Procedure](https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure) + +### Extended DDL Support + +From this release onwards, Flink supports + +- `REPLACE TABLE AS SELECT` +- `CREATE OR REPLACE TABLE AS SELECT` + +and both these commands and previously supported `CREATE TABLE AS` can now support atomicity provided the underlying +connector supports this. + +Moreover, Apache Flink now supports TRUNCATE TABLE in batch execution mode. As before, the underlying connector needs +to implement and provide this capability + +And, finally, we have also added support for adding, dropping and listing partitions via + +- `ALTER TABLE ADD PARTITION` +- `ALTER TABLE DROP PARTITION` +- `SHOW PARTITIONS` + +**More Information** +- [Documentation on TRUNCATE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/truncate/) +- [Documentation on CREATE OR REPLACE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-or-replace-table) +- [Documentation on ALTER TABLE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#alter-table) +- [FLIP-302: Support TRUNCATE TABLE statement in batch mode](https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement+in+batch+mode) +- [FLIP-303: Support REPLACE TABLE AS SELECT statement](https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement) +- [FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement](https://cwiki.apache.org/confluence/display/FLINK
[jira] [Created] (FLINK-33198) Add timestamp with local time zone support in Avro converters
Zhenqiu Huang created FLINK-33198: - Summary: Add timestamp with local time zone support in Avro converters Key: FLINK-33198 URL: https://issues.apache.org/jira/browse/FLINK-33198 Project: Flink Issue Type: Improvement Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Zhenqiu Huang Fix For: 1.18.1 Currently, RowDataToAvroConverters doesn't handle with LogicType TIMESTAMP_WITH_LOCAL_TIME_ZONE. We should add the corresponding conversion. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33182) Allow metadata columns in NduAnalyzer with ChangelogNormalize
[ https://issues.apache.org/jira/browse/FLINK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772592#comment-17772592 ] lincoln lee commented on FLINK-33182: - [~twalthr] I've thought about this carefully, and for upsert sources, while the non-determinism of the metadata column doesn't affect the +U message, if there is a -D message, there is no guarantee of correctness if we relax the strict restriction on the metadata (since the metadata column in the -D message is carried by the message itself, not as the changelognormalize node is handled). So, if we want to relax the restriction, we need to additionally make sure that the source does not contain -D message, WDYT? > Allow metadata columns in NduAnalyzer with ChangelogNormalize > - > > Key: FLINK-33182 > URL: https://issues.apache.org/jira/browse/FLINK-33182 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > Currently, the NduAnalyzer is very strict about metadata columns in updating > sources. However, for upsert sources (like Kafka) that contain an incomplete > changelog, the planner always adds a ChangelogNormalize node. > ChangelogNormalize will make sure that metadata columns can be considered > deterministic. So the NduAnalyzer should be satisfied in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33198) Add timestamp with local time zone support in Avro converters
[ https://issues.apache.org/jira/browse/FLINK-33198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora reassigned FLINK-33198: -- Assignee: Zhenqiu Huang > Add timestamp with local time zone support in Avro converters > - > > Key: FLINK-33198 > URL: https://issues.apache.org/jira/browse/FLINK-33198 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Minor > Fix For: 1.18.1 > > > Currently, RowDataToAvroConverters doesn't handle with LogicType > TIMESTAMP_WITH_LOCAL_TIME_ZONE. We should add the corresponding conversion. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33109) Watermark alignment not applied after recovery from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-33109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yordan Pavlov updated FLINK-33109:
--
Attachment: WatermarkTest.scala
> Watermark alignment not applied after recovery from checkpoint
> --
>
> Key: FLINK-33109
> URL: https://issues.apache.org/jira/browse/FLINK-33109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.1
>Reporter: Yordan Pavlov
>Priority: Major
> Attachments: WatermarkTest.scala, image-2023-09-18-15-40-06-868.png,
> image-2023-09-18-15-46-16-106.png
>
>
> I am observing a problem where after recovery from a checkpoint the Kafka
> source watermarks would start to diverge not honoring the watermark alignment
> setting I have applied.
> I have a Kafka source which reads a topic with 32 partitions. I am applying
> the following watermark strategy:
> {code:java}
> new EventAwareWatermarkStrategy[KeyedKafkaSourceMessage]](msg =>
> msg.value.getTimestamp)
> .withWatermarkAlignment("alignment-sources-group",
> time.Duration.ofMillis(sourceWatermarkAlignmentBlocks)){code}
>
> This works great up until my job needs to recover from checkpoint. Once the
> recovery takes place, no alignment is taking place any more. This can best be
> illustrated by looking at the watermark metrics for various operators in the
> image:
> !image-2023-09-18-15-40-06-868.png!
>
> You can see how the watermarks disperse after the recovery. Trying to debug
> the problem I noticed that before the failure there would be calls in
>
> {code:java}
> SourceCoordinator::announceCombinedWatermark()
> {code}
> after the recovery, no calls get there, so no value for
> {code:java}
> watermarkAlignmentParams.getMaxAllowedWatermarkDrift(){code}
> is ever read. I can manually fix the problem If I stop the job, clear all
> state from Zookeeper and then manually start Flink providing the last
> checkpoint with
> {code:java}
> '–fromSavepoint'{code}
> flag. This would cause the SourceCoordinator to be constructed properly and
> watermark drift to be checked. Once recovery manually watermarks would again
> converge to the allowed drift as seen in the metrics:
> !image-2023-09-18-15-46-16-106.png!
>
> Let me know If I can be helpful by providing any more information.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33187] Don't record duplicate event if no change [flink-kubernetes-operator]
gyfora commented on code in PR #683:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/683#discussion_r1348864871
##
flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/ScalingExecutor.java:
##
@@ -94,14 +94,15 @@ public boolean scaleResource(
var scalingEnabled = conf.get(SCALING_ENABLED);
var scalingReport = scalingReport(scalingSummaries, scalingEnabled);
-eventRecorder.triggerEvent(
+eventRecorder.triggerEventByInterval(
Review Comment:
We should only control the events by interval if `scalingEnabled == false`
(in advisor mode). Regular scaling should always trigger the event
##
flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/config/AutoScalerOptions.java:
##
@@ -168,4 +168,10 @@ private static ConfigOptions.OptionBuilder
autoScalerConfig(String key) {
.defaultValues()
.withDescription(
"A (semicolon-separated) list of vertex ids in
hexstring for which to disable scaling. Caution: For non-sink vertices this
will still scale their downstream operators until
https://issues.apache.org/jira/browse/FLINK-31215 is implemented.");
+
+public static final ConfigOption SCALING_REPORT_INTERVAL =
+autoScalerConfig("scaling.report.interval")
+.longType()
Review Comment:
This should be a `Duration` type config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-33109) Watermark alignment not applied after recovery from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-33109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772616#comment-17772616
]
Yordan Pavlov commented on FLINK-33109:
---
Hi [~fanrui]
sorry for the delay, running this image gives me the following error, shortly
after start
{code:java}
2023-10-06 14:54:20.725 [AsyncOperations-thread-1] INFO
o.a.flink.streaming.runtime.tasks.AsyncCheckpointRunnable -
TumblingEventTimeWindows (10/32)#0 - asynchronous part of checkpoint 1 could
not be completed.
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: 'void
org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle.(java.util.UUID,
org.apache.flink.runtime.state.KeyGroupRange, long, java.util.List,
java.util.List, org.apache.flink.runtime.state.StreamStateHandle, long)'
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:544)
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:54)
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:191)
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoSuchMethodError: 'void
org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle.(java.util.UUID,
org.apache.flink.runtime.state.KeyGroupRange, long, java.util.List,
java.util.List, org.apache.flink.runtime.state.StreamStateHandle, long)'
at
org.apache.flink.contrib.streaming.state.snapshot.RocksNativeFullSnapshotStrategy$RocksDBNativeFullSnapshotOperation.get(RocksNativeFullSnapshotStrategy.java:198)
at
org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:91)
at
org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:88)
at
org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:78)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:541)
... 6 common frames omitted{code}
so I couldn't test if has the problem I am facing.
Attached you can find a simple program which would exhibit the problem with the
latest stable image 1.17.1: [^WatermarkTest.scala]
what I would observe is that different sources would converge shortly after
start having a small watermark alignment value. However if I kill the task
manager and restart from a checkpoint they would start to diverge as if no
watermark alignment is present.
> Watermark alignment not applied after recovery from checkpoint
> --
>
> Key: FLINK-33109
> URL: https://issues.apache.org/jira/browse/FLINK-33109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.1
>Reporter: Yordan Pavlov
>Priority: Major
> Attachments: WatermarkTest.scala, image-2023-09-18-15-40-06-868.png,
> image-2023-09-18-15-46-16-106.png
>
>
> I am observing a problem where after recovery from a checkpoint the Kafka
> source watermarks would start to diverge not honoring the watermark alignment
> setting I have applied.
> I have a Kafka source which reads a topic with 32 partitions. I am applying
> the following watermark strategy:
> {code:java}
> new EventAwareWatermarkStrategy[KeyedKafkaSourceMessage]](msg =>
> msg.value.getTimestamp)
> .withWatermarkAlignment("alignment-sources-group",
> time.Duration.ofMillis(sourceWatermarkAlignmentBlocks)){code}
>
> This works great up until my job needs to recover from checkpoint. Once the
> recovery takes place, no alignment is taking place any more. This can best be
> illustrated by looking at the watermark metrics for various operators in the
> image:
> !image-2023-09-18-15-40-06-868.png!
>
> You can see how the watermarks disperse after the recovery. Trying to debug
> the problem I noticed that before the failure there would be calls in
>
> {code:java}
> SourceCoordinator::announceCombinedWatermark()
> {code}
> after the recovery, no calls get there, so no value for
> {code:java}
> watermarkAlignmentParams.getMaxAllowedWatermarkDrift(){code}
> is ever read. I can manually fix the problem If I stop the job, clear all
> state from Zookeeper and then manually start Flink prov
[jira] [Updated] (FLINK-33109) Watermark alignment not applied after recovery from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-33109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yordan Pavlov updated FLINK-33109:
--
Attachment: WatermarkTest-1.scala
> Watermark alignment not applied after recovery from checkpoint
> --
>
> Key: FLINK-33109
> URL: https://issues.apache.org/jira/browse/FLINK-33109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.1
>Reporter: Yordan Pavlov
>Priority: Major
> Attachments: WatermarkTest-1.scala,
> image-2023-09-18-15-40-06-868.png, image-2023-09-18-15-46-16-106.png
>
>
> I am observing a problem where after recovery from a checkpoint the Kafka
> source watermarks would start to diverge not honoring the watermark alignment
> setting I have applied.
> I have a Kafka source which reads a topic with 32 partitions. I am applying
> the following watermark strategy:
> {code:java}
> new EventAwareWatermarkStrategy[KeyedKafkaSourceMessage]](msg =>
> msg.value.getTimestamp)
> .withWatermarkAlignment("alignment-sources-group",
> time.Duration.ofMillis(sourceWatermarkAlignmentBlocks)){code}
>
> This works great up until my job needs to recover from checkpoint. Once the
> recovery takes place, no alignment is taking place any more. This can best be
> illustrated by looking at the watermark metrics for various operators in the
> image:
> !image-2023-09-18-15-40-06-868.png!
>
> You can see how the watermarks disperse after the recovery. Trying to debug
> the problem I noticed that before the failure there would be calls in
>
> {code:java}
> SourceCoordinator::announceCombinedWatermark()
> {code}
> after the recovery, no calls get there, so no value for
> {code:java}
> watermarkAlignmentParams.getMaxAllowedWatermarkDrift(){code}
> is ever read. I can manually fix the problem If I stop the job, clear all
> state from Zookeeper and then manually start Flink providing the last
> checkpoint with
> {code:java}
> '–fromSavepoint'{code}
> flag. This would cause the SourceCoordinator to be constructed properly and
> watermark drift to be checked. Once recovery manually watermarks would again
> converge to the allowed drift as seen in the metrics:
> !image-2023-09-18-15-46-16-106.png!
>
> Let me know If I can be helpful by providing any more information.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33109) Watermark alignment not applied after recovery from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-33109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yordan Pavlov updated FLINK-33109:
--
Attachment: (was: WatermarkTest.scala)
> Watermark alignment not applied after recovery from checkpoint
> --
>
> Key: FLINK-33109
> URL: https://issues.apache.org/jira/browse/FLINK-33109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.1
>Reporter: Yordan Pavlov
>Priority: Major
> Attachments: WatermarkTest-1.scala,
> image-2023-09-18-15-40-06-868.png, image-2023-09-18-15-46-16-106.png
>
>
> I am observing a problem where after recovery from a checkpoint the Kafka
> source watermarks would start to diverge not honoring the watermark alignment
> setting I have applied.
> I have a Kafka source which reads a topic with 32 partitions. I am applying
> the following watermark strategy:
> {code:java}
> new EventAwareWatermarkStrategy[KeyedKafkaSourceMessage]](msg =>
> msg.value.getTimestamp)
> .withWatermarkAlignment("alignment-sources-group",
> time.Duration.ofMillis(sourceWatermarkAlignmentBlocks)){code}
>
> This works great up until my job needs to recover from checkpoint. Once the
> recovery takes place, no alignment is taking place any more. This can best be
> illustrated by looking at the watermark metrics for various operators in the
> image:
> !image-2023-09-18-15-40-06-868.png!
>
> You can see how the watermarks disperse after the recovery. Trying to debug
> the problem I noticed that before the failure there would be calls in
>
> {code:java}
> SourceCoordinator::announceCombinedWatermark()
> {code}
> after the recovery, no calls get there, so no value for
> {code:java}
> watermarkAlignmentParams.getMaxAllowedWatermarkDrift(){code}
> is ever read. I can manually fix the problem If I stop the job, clear all
> state from Zookeeper and then manually start Flink providing the last
> checkpoint with
> {code:java}
> '–fromSavepoint'{code}
> flag. This would cause the SourceCoordinator to be constructed properly and
> watermark drift to be checked. Once recovery manually watermarks would again
> converge to the allowed drift as seen in the metrics:
> !image-2023-09-18-15-46-16-106.png!
>
> Let me know If I can be helpful by providing any more information.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33109) Watermark alignment not applied after recovery from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-33109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772616#comment-17772616
]
Yordan Pavlov edited comment on FLINK-33109 at 10/6/23 3:44 PM:
Hi [~fanrui]
sorry for the delay, running this image gives me the following error, shortly
after start
{code:java}
2023-10-06 14:54:20.725 [AsyncOperations-thread-1] INFO
o.a.flink.streaming.runtime.tasks.AsyncCheckpointRunnable -
TumblingEventTimeWindows (10/32)#0 - asynchronous part of checkpoint 1 could
not be completed.
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: 'void
org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle.(java.util.UUID,
org.apache.flink.runtime.state.KeyGroupRange, long, java.util.List,
java.util.List, org.apache.flink.runtime.state.StreamStateHandle, long)'
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:544)
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:54)
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:191)
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoSuchMethodError: 'void
org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle.(java.util.UUID,
org.apache.flink.runtime.state.KeyGroupRange, long, java.util.List,
java.util.List, org.apache.flink.runtime.state.StreamStateHandle, long)'
at
org.apache.flink.contrib.streaming.state.snapshot.RocksNativeFullSnapshotStrategy$RocksDBNativeFullSnapshotOperation.get(RocksNativeFullSnapshotStrategy.java:198)
at
org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:91)
at
org.apache.flink.runtime.state.SnapshotStrategyRunner$1.callInternal(SnapshotStrategyRunner.java:88)
at
org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:78)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:541)
... 6 common frames omitted{code}
so I couldn't test if has the problem I am facing.
Attached you can find a simple program which would exhibit the problem with the
latest stable image 1.17.1: [^WatermarkTest-1.scala]
what I would observe is that different sources would converge shortly after
start having a small watermark alignment value. However if I kill the task
manager and restart from a checkpoint they would start to diverge as if no
watermark alignment is present.
was (Author: yordanpavlov):
Hi [~fanrui]
sorry for the delay, running this image gives me the following error, shortly
after start
{code:java}
2023-10-06 14:54:20.725 [AsyncOperations-thread-1] INFO
o.a.flink.streaming.runtime.tasks.AsyncCheckpointRunnable -
TumblingEventTimeWindows (10/32)#0 - asynchronous part of checkpoint 1 could
not be completed.
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: 'void
org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle.(java.util.UUID,
org.apache.flink.runtime.state.KeyGroupRange, long, java.util.List,
java.util.List, org.apache.flink.runtime.state.StreamStateHandle, long)'
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:544)
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:54)
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:191)
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoSuchMethodError: 'void
org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle.(java.util.UUID,
org.apache.flink.runtime.state.KeyGroupRange, long, java.util.List,
java.util.List, org.apache.flink.runtime.state.StreamStateHandle, long)'
at
org.apache.flink.contrib.streaming.state.snapshot.RocksNativeFullSnap
[jira] [Created] (FLINK-33199) ArchitectureTests should test for canonical class names instead of Class objects
Alexander Fedulov created FLINK-33199: - Summary: ArchitectureTests should test for canonical class names instead of Class objects Key: FLINK-33199 URL: https://issues.apache.org/jira/browse/FLINK-33199 Project: Flink Issue Type: Improvement Components: Tests Reporter: Alexander Fedulov Currently architecture tests rely on importing such classes as MiniClusterExtension. This introduces a production scope dependency on flink-test-utils which in turn depends on flink-streaming-java. This is problematic because adding architecture tests to any direct or transitive dependency of flink-streaming-java creates a dependency cycle. Example: https://github.com/apache/flink/pull/22850#discussion_r1243343382 In general, since architecture tests are supposed to be used freely in any submodule, it is desirable to reduce its dependency surface as much as possible to prevent such cycles. This can be achieved by moving away from using Class objects and employing fully qualified type names checks instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33199) ArchitectureTests should test for canonical class names instead of Class objects
[ https://issues.apache.org/jira/browse/FLINK-33199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-33199: - Assignee: Alexander Fedulov > ArchitectureTests should test for canonical class names instead of Class > objects > > > Key: FLINK-33199 > URL: https://issues.apache.org/jira/browse/FLINK-33199 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > Currently architecture tests rely on importing such classes as > MiniClusterExtension. This introduces a production scope dependency on > flink-test-utils which in turn depends on flink-streaming-java. This is > problematic because adding architecture tests to any direct or transitive > dependency of flink-streaming-java creates a dependency cycle. > Example: https://github.com/apache/flink/pull/22850#discussion_r1243343382 > In general, since architecture tests are supposed to be used freely in any > submodule, it is desirable to reduce its dependency surface as much as > possible to prevent such cycles. > This can be achieved by moving away from using Class objects and employing > fully qualified type names checks instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33199) ArchitectureTests should test for fully qualified class names instead of Class objects
[ https://issues.apache.org/jira/browse/FLINK-33199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33199: -- Summary: ArchitectureTests should test for fully qualified class names instead of Class objects (was: ArchitectureTests should test for canonical class names instead of Class objects) > ArchitectureTests should test for fully qualified class names instead of > Class objects > -- > > Key: FLINK-33199 > URL: https://issues.apache.org/jira/browse/FLINK-33199 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > Currently architecture tests rely on importing such classes as > MiniClusterExtension. This introduces a production scope dependency on > flink-test-utils which in turn depends on flink-streaming-java. This is > problematic because adding architecture tests to any direct or transitive > dependency of flink-streaming-java creates a dependency cycle. > Example: https://github.com/apache/flink/pull/22850#discussion_r1243343382 > In general, since architecture tests are supposed to be used freely in any > submodule, it is desirable to reduce its dependency surface as much as > possible to prevent such cycles. > This can be achieved by moving away from using Class objects and employing > fully qualified type names checks instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-31631][FileSystems] shade guava in gs-fs filesystem [flink]
singhravidutt commented on PR #23489: URL: https://github.com/apache/flink/pull/23489#issuecomment-1751097059 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33187] Don't record duplicate event if no change [flink-kubernetes-operator]
clarax commented on code in PR #683:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/683#discussion_r1349036555
##
flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/config/AutoScalerOptions.java:
##
@@ -168,4 +168,10 @@ private static ConfigOptions.OptionBuilder
autoScalerConfig(String key) {
.defaultValues()
.withDescription(
"A (semicolon-separated) list of vertex ids in
hexstring for which to disable scaling. Caution: For non-sink vertices this
will still scale their downstream operators until
https://issues.apache.org/jira/browse/FLINK-31215 is implemented.");
+
+public static final ConfigOption SCALING_REPORT_INTERVAL =
+autoScalerConfig("scaling.report.interval")
+.longType()
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33187] Don't record duplicate event if no change [flink-kubernetes-operator]
clarax commented on code in PR #683: URL: https://github.com/apache/flink-kubernetes-operator/pull/683#discussion_r1349038012 ## flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler/ScalingExecutor.java: ## @@ -94,14 +94,15 @@ public boolean scaleResource( var scalingEnabled = conf.get(SCALING_ENABLED); var scalingReport = scalingReport(scalingSummaries, scalingEnabled); -eventRecorder.triggerEvent( +eventRecorder.triggerEventByInterval( Review Comment: updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31371) Support subscribing non-existed topics in Pulsar source
[
https://issues.apache.org/jira/browse/FLINK-31371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772666#comment-17772666
]
Enzo Dechaene commented on FLINK-31371:
---
Hi [~syhily], any news for this ticket ?
> Support subscribing non-existed topics in Pulsar source
> ---
>
> Key: FLINK-31371
> URL: https://issues.apache.org/jira/browse/FLINK-31371
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.15.3
>Reporter: Enzo Dechaene
>Assignee: Yufan Sheng
>Priority: Major
>
> *Describe the bug*
> With a Pulsar 2.8.4 server, a Flink stream containing Pulsar sources or sinks
> will fail at startup if the topic doesn't exist.
>
> *To Reproduce*
> Create a stream with :
> * Flink 1.15.2
> * Pulsar 2.8.4
> * with a Pulsar source or sink linked to a non existant topic
> * Start the stream
>
> *Expected behavior*
> If the topic doesn't exist, it should be created at the first connection of
> the source or sink without error.
>
> *Additional context*
> In the TopicListSubscriber class of the connector, the method
> getSubscribedTopicPartitions() try to get the metadata of a topic by doing
> that :
>
> {code:java}
> TopicMetadata metadata = queryTopicMetadata(pulsarAdmin, topic);{code}
>
> If the topic doesn't exist, I get a NullPointerException on the metadata
> We created a previous
> [ticket|https://github.com/streamnative/pulsar-flink/issues/366] on the
> Pulsar connector and it was fixed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33011) Operator deletes HA data unexpectedly
[
https://issues.apache.org/jira/browse/FLINK-33011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772350#comment-17772350
]
Mason Chen edited comment on FLINK-33011 at 10/6/23 7:06 PM:
-
[~gyfora] -could you backport this to 1.6? We are also hitting this bug in 1.6-
It has been already backported. I'll update the ticket fix versions. Thanks for
the fix!
was (Author: mason6345):
[~gyfora] could you backport this to 1.6? We are also hitting this bug in 1.6
> Operator deletes HA data unexpectedly
> -
>
> Key: FLINK-33011
> URL: https://issues.apache.org/jira/browse/FLINK-33011
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: 1.17.1, kubernetes-operator-1.6.0
> Environment: Flink: 1.17.1
> Flink Kubernetes Operator: 1.6.0
>Reporter: Ruibin Xing
>Assignee: Gyula Fora
>Priority: Blocker
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.7.0
>
> Attachments: flink_operator_logs_0831.csv
>
>
> We encountered a problem where the operator unexpectedly deleted HA data.
> The timeline is as follows:
> 12:08 We submitted the first spec, which suspended the job with savepoint
> upgrade mode.
> 12:08 The job was suspended, while the HA data was preserved, and the log
> showed the observed job deployment status was MISSING.
> 12:10 We submitted the second spec, which deployed the job with the last
> state upgrade mode.
> 12:10 Logs showed the operator deleted both the Flink deployment and the HA
> data again.
> 12:10 The job failed to start because the HA data was missing.
> According to the log, the deletion was triggered by
> https://github.com/apache/flink-kubernetes-operator/blob/a728ba768e20236184e2b9e9e45163304b8b196c/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ApplicationReconciler.java#L168
> I think this would only be triggered if the job deployment status wasn't
> MISSING. But the log before the deletion showed the observed job status was
> MISSING at that moment.
> Related logs:
>
> {code:java}
> 2023-08-30 12:08:48.190 + o.a.f.k.o.s.AbstractFlinkService [INFO
> ][default/pipeline-pipeline-se-3] Cluster shutdown completed.
> 2023-08-30 12:10:27.010 + o.a.f.k.o.o.d.ApplicationObserver [INFO
> ][default/pipeline-pipeline-se-3] Observing JobManager deployment. Previous
> status: MISSING
> 2023-08-30 12:10:27.533 + o.a.f.k.o.l.AuditUtils [INFO
> ][default/pipeline-pipeline-se-3] >>> Event | Info | SPECCHANGED |
> UPGRADE change(s) detected (Diff: FlinkDeploymentSpec[image :
> docker-registry.randomcompany.com/octopus/pipeline-pipeline-online:0835137c-362
> ->
> docker-registry.randomcompany.com/octopus/pipeline-pipeline-online:23db7ae8-365,
> podTemplate.metadata.labels.app.kubernetes.io~1version :
> 0835137cd803b7258695eb53a6ec520cb62a48a7 ->
> 23db7ae84bdab8d91fa527fe2f8f2fce292d0abc, job.state : suspended -> running,
> job.upgradeMode : last-state -> savepoint, restartNonce : 1545 -> 1547]),
> starting reconciliation.
> 2023-08-30 12:10:27.679 + o.a.f.k.o.s.NativeFlinkService [INFO
> ][default/pipeline-pipeline-se-3] Deleting JobManager deployment and HA
> metadata.
> {code}
> A more complete log file is attached. Thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33011) Operator deletes HA data unexpectedly
[
https://issues.apache.org/jira/browse/FLINK-33011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772350#comment-17772350
]
Mason Chen edited comment on FLINK-33011 at 10/6/23 7:07 PM:
-
[~gyfora] -could you backport this to 1.6? We are also hitting this bug in 1.6-
It has been already backported. Would be good to update the ticket fix version.
Thanks for the fix!
was (Author: mason6345):
[~gyfora] -could you backport this to 1.6? We are also hitting this bug in 1.6-
It has been already backported. I'll update the ticket fix versions. Thanks for
the fix!
> Operator deletes HA data unexpectedly
> -
>
> Key: FLINK-33011
> URL: https://issues.apache.org/jira/browse/FLINK-33011
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: 1.17.1, kubernetes-operator-1.6.0
> Environment: Flink: 1.17.1
> Flink Kubernetes Operator: 1.6.0
>Reporter: Ruibin Xing
>Assignee: Gyula Fora
>Priority: Blocker
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.7.0
>
> Attachments: flink_operator_logs_0831.csv
>
>
> We encountered a problem where the operator unexpectedly deleted HA data.
> The timeline is as follows:
> 12:08 We submitted the first spec, which suspended the job with savepoint
> upgrade mode.
> 12:08 The job was suspended, while the HA data was preserved, and the log
> showed the observed job deployment status was MISSING.
> 12:10 We submitted the second spec, which deployed the job with the last
> state upgrade mode.
> 12:10 Logs showed the operator deleted both the Flink deployment and the HA
> data again.
> 12:10 The job failed to start because the HA data was missing.
> According to the log, the deletion was triggered by
> https://github.com/apache/flink-kubernetes-operator/blob/a728ba768e20236184e2b9e9e45163304b8b196c/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ApplicationReconciler.java#L168
> I think this would only be triggered if the job deployment status wasn't
> MISSING. But the log before the deletion showed the observed job status was
> MISSING at that moment.
> Related logs:
>
> {code:java}
> 2023-08-30 12:08:48.190 + o.a.f.k.o.s.AbstractFlinkService [INFO
> ][default/pipeline-pipeline-se-3] Cluster shutdown completed.
> 2023-08-30 12:10:27.010 + o.a.f.k.o.o.d.ApplicationObserver [INFO
> ][default/pipeline-pipeline-se-3] Observing JobManager deployment. Previous
> status: MISSING
> 2023-08-30 12:10:27.533 + o.a.f.k.o.l.AuditUtils [INFO
> ][default/pipeline-pipeline-se-3] >>> Event | Info | SPECCHANGED |
> UPGRADE change(s) detected (Diff: FlinkDeploymentSpec[image :
> docker-registry.randomcompany.com/octopus/pipeline-pipeline-online:0835137c-362
> ->
> docker-registry.randomcompany.com/octopus/pipeline-pipeline-online:23db7ae8-365,
> podTemplate.metadata.labels.app.kubernetes.io~1version :
> 0835137cd803b7258695eb53a6ec520cb62a48a7 ->
> 23db7ae84bdab8d91fa527fe2f8f2fce292d0abc, job.state : suspended -> running,
> job.upgradeMode : last-state -> savepoint, restartNonce : 1545 -> 1547]),
> starting reconciliation.
> 2023-08-30 12:10:27.679 + o.a.f.k.o.s.NativeFlinkService [INFO
> ][default/pipeline-pipeline-se-3] Deleting JobManager deployment and HA
> metadata.
> {code}
> A more complete log file is attached. Thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33175) Nightly builds from S3 are not available for download, breaking all connector tests
[
https://issues.apache.org/jira/browse/FLINK-33175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772680#comment-17772680
]
Jing Ge commented on FLINK-33175:
-
[~mapohl] the file is ok and not public accessible:
[https://apache-flink.slack.com/archives/C051F2AFNJW/p1695206313618819?thread_ts=1695122850.132129&cid=C051F2AFNJW]
I just followed the release guideline and didn't change anything else. Do you
know why the file can be downloaded from S3 previously? How is the credential
configured?
> Nightly builds from S3 are not available for download, breaking all connector
> tests
> ---
>
> Key: FLINK-33175
> URL: https://issues.apache.org/jira/browse/FLINK-33175
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Reporter: Martijn Visser
>Priority: Blocker
>
> All downloads of Flink binaries fail with:
> {code:java}
> Run wget -q -c
> https://s3.amazonaws.com/flink-nightly/flink-1.18-SNAPSHOT-bin-scala_2.12.tgz
> -O - | tar -xz
> gzip: stdin: unexpected end of file
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> Error: Process completed with exit code 2.
> {code}
> This goes for 1.18, but also 1.17 and 1.16
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33175) Nightly builds from S3 are not available for download, breaking all connector tests
[
https://issues.apache.org/jira/browse/FLINK-33175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772680#comment-17772680
]
Jing Ge edited comment on FLINK-33175 at 10/6/23 8:18 PM:
--
[~mapohl] the file is ok and not public accessible:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53296&view=l[…]-51bc-8163-03e54ba9ce5b&t=2f31de30-3dd6-5bd6-d86d-dcdc2d7cc3de|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53296&view=logs&j=585d8b77-fa33-51bc-8163-03e54ba9ce5b&t=2f31de30-3dd6-5bd6-d86d-dcdc2d7cc3de]
I just followed the release guideline and didn't change anything else. Do you
know why the file can be downloaded from S3 previously? How is the credential
configured?
was (Author: jingge):
[~mapohl] the file is ok and not public accessible:
[https://apache-flink.slack.com/archives/C051F2AFNJW/p1695206313618819?thread_ts=1695122850.132129&cid=C051F2AFNJW]
I just followed the release guideline and didn't change anything else. Do you
know why the file can be downloaded from S3 previously? How is the credential
configured?
> Nightly builds from S3 are not available for download, breaking all connector
> tests
> ---
>
> Key: FLINK-33175
> URL: https://issues.apache.org/jira/browse/FLINK-33175
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Reporter: Martijn Visser
>Priority: Blocker
>
> All downloads of Flink binaries fail with:
> {code:java}
> Run wget -q -c
> https://s3.amazonaws.com/flink-nightly/flink-1.18-SNAPSHOT-bin-scala_2.12.tgz
> -O - | tar -xz
> gzip: stdin: unexpected end of file
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> Error: Process completed with exit code 2.
> {code}
> This goes for 1.18, but also 1.17 and 1.16
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32732) auto offset reset should be exposed to user
[
https://issues.apache.org/jira/browse/FLINK-32732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32732:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> auto offset reset should be exposed to user
> ---
>
> Key: FLINK-32732
> URL: https://issues.apache.org/jira/browse/FLINK-32732
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: 1.16.1
>Reporter: xiaogang zhou
>Priority: Major
> Labels: pull-request-available, stale-major
>
> {code:java}
> // code placeholder
> maybeOverride(
> ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
>
> startingOffsetsInitializer.getAutoOffsetResetStrategy().name().toLowerCase(),
> true); {code}
> now flink override the auto.offset.reset with the scan.startup.mode config,
> and user's explicit config does not take effect. I think maybe we should
> expose this to customer?
>
> I think after consuming kafka records from earliest to latest, the
> scan.startup.mode should no longer influence the kafka scan behave. So I
> suggest change the override to false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31615) Fix some parts forgot to translate in "Table API" page of "Table API & SQL"
[ https://issues.apache.org/jira/browse/FLINK-31615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31615: --- Labels: auto-deprioritized-minor chinese-translation (was: chinese-translation stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Fix some parts forgot to translate in "Table API" page of "Table API & SQL" > > > Key: FLINK-31615 > URL: https://issues.apache.org/jira/browse/FLINK-31615 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Hang Ruan >Priority: Not a Priority > Labels: auto-deprioritized-minor, chinese-translation > > The query_state_warning in "Table API" page of "Table API & SQL" is still in > English. And some comments in codes are in English. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32293) Support vector with long index
[ https://issues.apache.org/jira/browse/FLINK-32293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32293: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support vector with long index > -- > > Key: FLINK-32293 > URL: https://issues.apache.org/jira/browse/FLINK-32293 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: Zhipeng Zhang >Priority: Major > Labels: pull-request-available, stale-major > > Currently in Flink ML, we only support sparse and dense vector with `int` as > index and `double` as value. > > However, there are real-world cases that the index of a vector could exceed > the range of `INT.MAX`. Thus we need to support vector with `long` index. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
Zhenqiu Huang created FLINK-33200:
-
Summary: ItemAt Expression validation fail in Table API due to
type mismatch
Key: FLINK-33200
URL: https://issues.apache.org/jira/browse/FLINK-33200
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: Zhenqiu Huang
Fix For: 1.8.4
The table schema is defined as below:
public static final DataType DATA_TYPE = DataTypes.ROW(
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("events",
DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(
);
public static final Schema SCHEMA =
Schema.newBuilder().fromRowDataType(DATA_TYPE).build();
inTable.select(Expressions.$("events").at(1).at("eventType").as("firstEventType")
The validation fail as "eventType" is inferred as
BasicTypeInfo.STRING_TYPE_INFO, the table key internally is a
StringDataTypeInfo. The validation fail at
case mti: MapTypeInfo[_, _] =>
if (key.resultType == mti.getKeyTypeInfo) {
ValidationSuccess
} else {
ValidationFailure(
s"Map entry access needs a valid key of type " +
s"'${mti.getKeyTypeInfo}', found '${key.resultType}'.")
}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772723#comment-17772723
]
Zhenqiu Huang commented on FLINK-33200:
---
[~lzljs3620320]
As StringDataTypeInfo is the default key type in Flink table, shall we loose
the validation to
if (key.resultType == StringDataTypeInfo || key.resultType ==
BasicTypeInfo.STRING_TYPE_INFO)
> ItemAt Expression validation fail in Table API due to type mismatch
> ---
>
> Key: FLINK-33200
> URL: https://issues.apache.org/jira/browse/FLINK-33200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Zhenqiu Huang
>Priority: Minor
> Fix For: 1.8.4
>
>
> The table schema is defined as below:
> public static final DataType DATA_TYPE = DataTypes.ROW(
> DataTypes.FIELD("id", DataTypes.STRING()),
> DataTypes.FIELD("events",
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(
> );
> public static final Schema SCHEMA =
> Schema.newBuilder().fromRowDataType(DATA_TYPE).build();
> inTable.select(Expressions.$("events").at(1).at("eventType").as("firstEventType")
> The validation fail as "eventType" is inferred as
> BasicTypeInfo.STRING_TYPE_INFO, the table key internally is a
> StringDataTypeInfo. The validation fail at
> case mti: MapTypeInfo[_, _] =>
> if (key.resultType == mti.getKeyTypeInfo) {
> ValidationSuccess
> } else {
> ValidationFailure(
> s"Map entry access needs a valid key of type " +
> s"'${mti.getKeyTypeInfo}', found '${key.resultType}'.")
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] add announcement blog post for Flink 1.18 [flink-web]
lsyldliu commented on code in PR #680: URL: https://github.com/apache/flink-web/pull/680#discussion_r1349437120 ## docs/content/posts/2023-10-10-release-1.18.0.md: ## @@ -0,0 +1,542 @@ +--- +authors: +- JingGe: + name: "Jing Ge" + twitter: jingengineer +- KonstantinKnauf: + name: "Konstantin Knauf" + twitter: snntrable +- SergeyNuyanzin: + name: "Sergey Nuyanzin" + twitter: uckamello +- QingshengRen: + name: "Qingsheng Ren" + twitter: renqstuite +date: "2023-10-10T08:00:00Z" +subtitle: "" +title: Announcing the Release of Apache Flink 1.18 +aliases: +- /news/2023/10/10/release-1.18.0.html +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 1.18.0. As usual, we are looking at a packed +release with a wide variety of improvements and new features. Overall, 176 people contributed to this release completing +18 FLIPS and 700+ issues. Thank you! + +Let's dive into the highlights. + +# Towards a Streaming Lakehouse + +## Flink SQL Improvements + +### Introduce Flink JDBC Driver For Sql Gateway + +Flink 1.18 comes with a JDBC Driver for the Flink SQL Gateway. So, you can now use any SQL Client that supports JDBC to +interact with your tables via Flink SQL. Here is an example using [SQLLine](https://julianhyde.github.io/sqlline/manual.html). + +```shell +sqlline> !connect jdbc:flink://localhost:8083 +``` + +```shell +sqlline version 1.12.0 +sqlline> !connect jdbc:flink://localhost:8083 +Enter username for jdbc:flink://localhost:8083: +Enter password for jdbc:flink://localhost:8083: +0: jdbc:flink://localhost:8083> CREATE TABLE T( +. . . . . . . . . . . . . . .)> a INT, +. . . . . . . . . . . . . . .)> b VARCHAR(10) +. . . . . . . . . . . . . . .)> ) WITH ( +. . . . . . . . . . . . . . .)> 'connector' = 'filesystem', +. . . . . . . . . . . . . . .)> 'path' = 'file:///tmp/T.csv', +. . . . . . . . . . . . . . .)> 'format' = 'csv' +. . . . . . . . . . . . . . .)> ); +No rows affected (0.122 seconds) +0: jdbc:flink://localhost:8083> INSERT INTO T VALUES (1, 'Hi'), (2, 'Hello'); ++--+ +| job id | ++--+ +| fbade1ab4450fc57ebd5269fdf60dcfd | ++--+ +1 row selected (1.282 seconds) +0: jdbc:flink://localhost:8083> SELECT * FROM T; ++---+---+ +| a | b | ++---+---+ +| 1 | Hi| +| 2 | Hello | ++---+---+ +2 rows selected (1.955 seconds) +0: jdbc:flink://localhost:8083> +``` + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/) +* [FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway](https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway) + + +### Stored Procedures + +Stored Procedures provide a convenient way to encapsulate complex logic to perform data manipulation or administrative +tasks in Apache Flink itself. Therefore， Flink introduces the support for calling stored procedures. +Flink now allows catalog developers to develop their own built-in stored procedures and then enables users to call these +predefined stored procedures. + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/procedures/) +* [FLIP-311: Support Call Stored Procedure](https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure) + +### Extended DDL Support + +From this release onwards, Flink supports + +- `REPLACE TABLE AS SELECT` +- `CREATE OR REPLACE TABLE AS SELECT` + +and both these commands and previously supported `CREATE TABLE AS` can now support atomicity provided the underlying +connector supports this. + +Moreover, Apache Flink now supports TRUNCATE TABLE in batch execution mode. As before, the underlying connector needs +to implement and provide this capability + +And, finally, we have also added support for adding, dropping and listing partitions via + +- `ALTER TABLE ADD PARTITION` +- `ALTER TABLE DROP PARTITION` +- `SHOW PARTITIONS` + +**More Information** +- [Documentation on TRUNCATE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/truncate/) +- [Documentation on CREATE OR REPLACE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-or-replace-table) +- [Documentation on ALTER TABLE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#alter-table) +- [FLIP-302: Support TRUNCATE TABLE statement in batch mode](https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement+in+batch+mode) +- [FLIP-303: Support REPLACE TABLE AS SELECT statement](https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement) +- [FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement](https://cwiki.apache.org/confluence/display/FLINK/
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
rinkako commented on PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53#issuecomment-1751562763 hi @mtfelisb , I read the code and found that it has no implementation for `DynamicTableSink`, hence this es8 sink cannot be used in flink-sql. Would you have any plan to work on this point? 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33046][FLIP-333] Redesign Apache Flink website [flink-web]
Myasuka commented on PR #676: URL: https://github.com/apache/flink-web/pull/676#issuecomment-1751567081 I found a strange behavior that when I click the nav title on the right, it will hide the head of lines:  What I expect:  The problematic link https://website-refresh.d193kg429zpv7e.amplifyapp.com/what-is-flink/flink-applications/#state -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33053][zookeeper] Manually remove the leader watcher after ret… [flink]
KarmaGYZ commented on code in PR #23415:
URL: https://github.com/apache/flink/pull/23415#discussion_r1349440903
##
flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/ZooKeeperLeaderRetrievalDriver.java:
##
@@ -122,6 +125,19 @@ public void close() throws Exception {
client.getConnectionStateListenable().removeListener(connectionStateListener);
cache.close();
+
+try {
+if (client.getZookeeperClient().isConnected()
+&&
!connectionInformationPath.contains(RESOURCE_MANAGER_NODE)) {
Review Comment:
Also an exclusive curator framework will definitely harm the performance and
add additional pressure to ZooKeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33053][zookeeper] Manually remove the leader watcher after ret… [flink]
KarmaGYZ commented on code in PR #23415:
URL: https://github.com/apache/flink/pull/23415#discussion_r1349440740
##
flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/ZooKeeperLeaderRetrievalDriver.java:
##
@@ -122,6 +125,19 @@ public void close() throws Exception {
client.getConnectionStateListenable().removeListener(connectionStateListener);
cache.close();
+
+try {
+if (client.getZookeeperClient().isConnected()
+&&
!connectionInformationPath.contains(RESOURCE_MANAGER_NODE)) {
Review Comment:
I agree that this is a rather tricky fix. As Tison mentioned, JobMaster and
Dispatcher will use the same curator framework at the JobManager side. Close
the watch on resource manager when JobMaster exit will influence other
JobMasters and Dispatcher, thus I arbitrarily exclude the RM here. We can also
construct an exclusive curator framework for each JobMaster, but I think a
cleaner solution should be implemented within curator or zookeeper, then we
should remove this fix. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33053][zookeeper] Manually remove the leader watcher after ret… [flink]
KarmaGYZ commented on code in PR #23415:
URL: https://github.com/apache/flink/pull/23415#discussion_r1349440903
##
flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/ZooKeeperLeaderRetrievalDriver.java:
##
@@ -122,6 +125,19 @@ public void close() throws Exception {
client.getConnectionStateListenable().removeListener(connectionStateListener);
cache.close();
+
+try {
+if (client.getZookeeperClient().isConnected()
+&&
!connectionInformationPath.contains(RESOURCE_MANAGER_NODE)) {
Review Comment:
Also an exclusive curator framework per JobMaster will definitely harm the
performance and add additional pressure to ZooKeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31275) Flink supports reporting and storage of source/sink tables relationship
[ https://issues.apache.org/jira/browse/FLINK-31275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772734#comment-17772734 ] Fang Yong commented on FLINK-31275: --- [~ZhenqiuHuang] Thanks for your attention, FLIP-314 is on voting https://lists.apache.org/thread/dxdqjc0dd8rf1vbdg755zo1n2zj1tj8d > Flink supports reporting and storage of source/sink tables relationship > --- > > Key: FLINK-31275 > URL: https://issues.apache.org/jira/browse/FLINK-31275 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Fang Yong >Priority: Major > > Currently flink generates job id in `JobGraph` which can identify a job. On > the other hand, flink create source/sink table in planner. We need to create > relations between source and sink tables for the job with an identifier id -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] add announcement blog post for Flink 1.18 [flink-web]
Myasuka commented on code in PR #680: URL: https://github.com/apache/flink-web/pull/680#discussion_r1349442579 ## docs/content/posts/2023-10-10-release-1.18.0.md: ## @@ -0,0 +1,542 @@ +--- +authors: +- JingGe: + name: "Jing Ge" + twitter: jingengineer +- KonstantinKnauf: + name: "Konstantin Knauf" + twitter: snntrable +- SergeyNuyanzin: + name: "Sergey Nuyanzin" + twitter: uckamello +- QingshengRen: + name: "Qingsheng Ren" + twitter: renqstuite +date: "2023-10-10T08:00:00Z" +subtitle: "" +title: Announcing the Release of Apache Flink 1.18 +aliases: +- /news/2023/10/10/release-1.18.0.html +--- + +The Apache Flink PMC is pleased to announce the release of Apache Flink 1.18.0. As usual, we are looking at a packed +release with a wide variety of improvements and new features. Overall, 176 people contributed to this release completing +18 FLIPS and 700+ issues. Thank you! + +Let's dive into the highlights. + +# Towards a Streaming Lakehouse + +## Flink SQL Improvements + +### Introduce Flink JDBC Driver For Sql Gateway + +Flink 1.18 comes with a JDBC Driver for the Flink SQL Gateway. So, you can now use any SQL Client that supports JDBC to +interact with your tables via Flink SQL. Here is an example using [SQLLine](https://julianhyde.github.io/sqlline/manual.html). + +```shell +sqlline> !connect jdbc:flink://localhost:8083 +``` + +```shell +sqlline version 1.12.0 +sqlline> !connect jdbc:flink://localhost:8083 +Enter username for jdbc:flink://localhost:8083: +Enter password for jdbc:flink://localhost:8083: +0: jdbc:flink://localhost:8083> CREATE TABLE T( +. . . . . . . . . . . . . . .)> a INT, +. . . . . . . . . . . . . . .)> b VARCHAR(10) +. . . . . . . . . . . . . . .)> ) WITH ( +. . . . . . . . . . . . . . .)> 'connector' = 'filesystem', +. . . . . . . . . . . . . . .)> 'path' = 'file:///tmp/T.csv', +. . . . . . . . . . . . . . .)> 'format' = 'csv' +. . . . . . . . . . . . . . .)> ); +No rows affected (0.122 seconds) +0: jdbc:flink://localhost:8083> INSERT INTO T VALUES (1, 'Hi'), (2, 'Hello'); ++--+ +| job id | ++--+ +| fbade1ab4450fc57ebd5269fdf60dcfd | ++--+ +1 row selected (1.282 seconds) +0: jdbc:flink://localhost:8083> SELECT * FROM T; ++---+---+ +| a | b | ++---+---+ +| 1 | Hi| +| 2 | Hello | ++---+---+ +2 rows selected (1.955 seconds) +0: jdbc:flink://localhost:8083> +``` + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/) +* [FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway](https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway) + + +### Stored Procedures + +Stored Procedures provide a convenient way to encapsulate complex logic to perform data manipulation or administrative +tasks in Apache Flink itself. Therefore， Flink introduces the support for calling stored procedures. +Flink now allows catalog developers to develop their own built-in stored procedures and then enables users to call these +predefined stored procedures. + +**More Information** +* [Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/procedures/) +* [FLIP-311: Support Call Stored Procedure](https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure) + +### Extended DDL Support + +From this release onwards, Flink supports + +- `REPLACE TABLE AS SELECT` +- `CREATE OR REPLACE TABLE AS SELECT` + +and both these commands and previously supported `CREATE TABLE AS` can now support atomicity provided the underlying +connector supports this. + +Moreover, Apache Flink now supports TRUNCATE TABLE in batch execution mode. As before, the underlying connector needs +to implement and provide this capability + +And, finally, we have also added support for adding, dropping and listing partitions via + +- `ALTER TABLE ADD PARTITION` +- `ALTER TABLE DROP PARTITION` +- `SHOW PARTITIONS` + +**More Information** +- [Documentation on TRUNCATE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/truncate/) +- [Documentation on CREATE OR REPLACE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-or-replace-table) +- [Documentation on ALTER TABLE](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/alter/#alter-table) +- [FLIP-302: Support TRUNCATE TABLE statement in batch mode](https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement+in+batch+mode) +- [FLIP-303: Support REPLACE TABLE AS SELECT statement](https://cwiki.apache.org/confluence/display/FLINK/FLIP-303%3A+Support+REPLACE+TABLE+AS+SELECT+statement) +- [FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement](https://cwiki.apache.org/confluence/display/FLINK/F
[jira] [Updated] (FLINK-31275) Flink supports reporting and storage of source/sink tables relationship
[ https://issues.apache.org/jira/browse/FLINK-31275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fang Yong updated FLINK-31275: -- Description: FLIP-314 has been accepted https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener (was: Currently flink generates job id in `JobGraph` which can identify a job. On the other hand, flink create source/sink table in planner. We need to create relations between source and sink tables for the job with an identifier id) > Flink supports reporting and storage of source/sink tables relationship > --- > > Key: FLINK-31275 > URL: https://issues.apache.org/jira/browse/FLINK-31275 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Fang Yong >Priority: Major > > FLIP-314 has been accepted > https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-31275) Flink supports reporting and storage of source/sink tables relationship
[ https://issues.apache.org/jira/browse/FLINK-31275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fang Yong reassigned FLINK-31275: - Assignee: Fang Yong > Flink supports reporting and storage of source/sink tables relationship > --- > > Key: FLINK-31275 > URL: https://issues.apache.org/jira/browse/FLINK-31275 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.18.0 >Reporter: Fang Yong >Assignee: Fang Yong >Priority: Major > > FLIP-314 has been accepted > https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33166][core][config] Make root logger level configurable [flink]
X-czh commented on PR #23476: URL: https://github.com/apache/flink/pull/23476#issuecomment-1751601862 @FangYongs, could you help review it when you have time? Many thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33201) Memory leak in CachingTopicSelector
Sunyeop Lee created FLINK-33201: --- Summary: Memory leak in CachingTopicSelector Key: FLINK-33201 URL: https://issues.apache.org/jira/browse/FLINK-33201 Project: Flink Issue Type: Bug Components: Connectors / Kafka Environment: I am out of office now, so this is what I remember (the flink version may not be correct). Because I already identified the cause, this should not matter anyway. EKS 1.24, x86_64, Bottlerocket OS, flink 1.14, scala 2.12 Reporter: Sunyeop Lee Attachments: 273084767-29bc0d8a-7445-4a74-a6e1-7c836775c7b1.png Pull Request available at: https://github.com/apache/flink-connector-kafka/pull/55 In the CachingTopicSelector, a memory leak may occur when the internal logic fails to check the cache size due to a race condition. ([https://github.com/apache/flink-connector-kafka/blob/d89a082180232bb79e3c764228c4e7dbb9eb6b8b/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaRecordSerializationSchemaBuilder.java#L287-L289)] By analyzing a Java heap dump, I identified a memory leak in the CachingTopicSelector. As in the screenshot, cache has 47,769 elements. If the internal logic were functioning correctly, the number of elements should be less than or equal to CACHE_RESET_SIZE (which is 5). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33202) FLIP-327: Support switching from batch to stream mode to improve throughput when processing backlog data
Xuannan Su created FLINK-33202: -- Summary: FLIP-327: Support switching from batch to stream mode to improve throughput when processing backlog data Key: FLINK-33202 URL: https://issues.apache.org/jira/browse/FLINK-33202 Project: Flink Issue Type: New Feature Components: Runtime / Task Reporter: Xuannan Su Fix For: 1.19.0 Umbrella issue for [https://cwiki.apache.org/confluence/display/FLINK/FLIP-327%3A+Support+switching+from+batch+to+stream+mode+to+improve+throughput+when+processing+backlog+data|https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog] h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33060] Fix the javadoc of ListState interfaces about not allowing null value [flink]
Zakelly commented on code in PR #23441:
URL: https://github.com/apache/flink/pull/23441#discussion_r1349473352
##
flink-core/src/main/java/org/apache/flink/api/common/state/AppendingState.java:
##
@@ -60,8 +60,11 @@ public interface AppendingState extends State {
*
* If null is passed in, the state value will remain unchanged.
*
+ * Except for list state, where null value is not allowed to be passed
in.
Review Comment:
Agreed. I believe it would be better to eliminate the description of the
null value at the moment, considering the implementations have already become
varied. We can consider refactoring the interfaces in the upcoming major
version. What do you think? @davidradl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33161] Java17 profile for benchmarking [flink-benchmarks]
Zakelly commented on PR #80: URL: https://github.com/apache/flink-benchmarks/pull/80#issuecomment-1751624381 I'm not sure if I have the right person, @pnowojski , but would you please review this one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
