[jira] [Created] (FLINK-35748) DeduplicateITCase.testLastRowWithoutAllChangelogOnRowtime with MiniBatch mode and RocksDB backend enabled
Matthias Pohl created FLINK-35748:
-
Summary: DeduplicateITCase.testLastRowWithoutAllChangelogOnRowtime
with MiniBatch mode and RocksDB backend enabled
Key: FLINK-35748
URL: https://issues.apache.org/jira/browse/FLINK-35748
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.20.0
Reporter: Matthias Pohl
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60613&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4&l=12259
{code}
Jul 02 14:44:36 14:44:36.737 [ERROR] Tests run: 40, Failures: 1, Errors: 0,
Skipped: 4, Time elapsed: 18.45 s <<< FAILURE! -- in
org.apache.flink.table.planner.runtime.stream.sql.DeduplicateITCase
Jul 02 14:44:36 14:44:36.737 [ERROR]
org.apache.flink.table.planner.runtime.stream.sql.DeduplicateITCase.testLastRowWithoutAllChangelogOnRowtime
-- Time elapsed: 0.860 s <<< FAILURE!
Jul 02 14:44:36 org.opentest4j.AssertionFailedError:
Jul 02 14:44:36
Jul 02 14:44:36 expected: List(+I(1,1,Hi,1970-01-01T00:00:00.001), +I(1,2,Hello
world,1970-01-01T00:00:00.002), +I(2,3,I am fine.,1970-01-01T00:00:00.003),
+I(2,6,Comment#1,1970-01-01T00:00:00.006),
+I(3,5,Comment#2,1970-01-01T00:00:00.005),
+I(4,4,Comment#3,1970-01-01T00:00:00.004))
Jul 02 14:44:36 but was: ArrayBuffer(+I(1,1,Hi,1970-01-01T00:00:00.001),
+I(1,2,Hello world,1970-01-01T00:00:00.002),
+I(1,3,Hello,1970-01-01T00:00:00.003),
+I(2,6,Comment#1,1970-01-01T00:00:00.006),
+I(3,5,Comment#2,1970-01-01T00:00:00.005),
+I(4,4,Comment#3,1970-01-01T00:00:00.004), +U(2,3,I am
fine.,1970-01-01T00:00:00.003), -U(1,3,Hello,1970-01-01T00:00:00.003))
Jul 02 14:44:36 at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jul 02 14:44:36 at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Jul 02 14:44:36 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jul 02 14:44:36 at
org.apache.flink.table.planner.runtime.stream.sql.DeduplicateITCase.testLastRowWithoutAllChangelogOnRowtime(DeduplicateITCase.scala:364)
Jul 02 14:44:36 at java.lang.reflect.Method.invoke(Method.java:498)
[...]
{code}
The test failure appeared in a CI run for FLINK-35553. Which does some changes
to how checkpointing is triggered. I checked the logs and couldn't find any
evidence that the test run included the FLINK-35553 change (no restoring from
checkpoint happens in the failed and successful of the test; see attached logs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35749) Kafka sink component will lose data when kafka cluster is unavailable for a while
Jimmy Zhao created FLINK-35749: -- Summary: Kafka sink component will lose data when kafka cluster is unavailable for a while Key: FLINK-35749 URL: https://issues.apache.org/jira/browse/FLINK-35749 Project: Flink Issue Type: Bug Components: Connectors / Kafka Affects Versions: 1.17.1, 1.16.2, 1.20.0 Reporter: Jimmy Zhao As the title described, here is the procedure to reproduce the problem: 1. develop a simple flink stream job to consume from one kafka topic and sink to anthoer kafka sever and topic 2. make amount of kafka message and produce to the source kafka topic, record the message number 3. start the flink stream job, and config to cosume from earliest source topic offset 4. during the job cosuming the source topic, restart the kafka cluster(we use aws MSK) 5. the flink job will not throw any Exception like nothing happened, but only print error log like : [kafka-producer-network-thread | producer-2] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Node 2 disconnected. 6. wait for the kafka cluster finished restarting and all the source kafka message consumed 7. count the target kafka topic message number, compare to the source, there is a high probability of data loss（more than 50%） -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-456: CompiledPlan support for Batch Execution Mode
+1 (binding) Thanks, Timo On 02.07.24 19:46, Sergey Nuyanzin wrote: Thanks for driving this +1 (binding) On Tue, Jul 2, 2024, 11:21 Martijn Visser wrote: +1 (binding) On Mon, Jul 1, 2024 at 7:00 PM Jim Hughes wrote: Hi Alexey, +1 (non-binding) I'm looking forward to parity between streaming and batch bound for compiled plans! Cheers, Jim On Mon, Jul 1, 2024 at 12:55 PM Alexey Leonov-Vendrovskiy < vendrov...@gmail.com> wrote: Hello everyone, We had a good discussion of FLIP-456: CompiledPlan support for Batch Execution Mode [1]. Discussion thread is here: [2]. Let's start voting on it. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. The FLIP will be considered accepted if 3 binding votes (from active committers according to the Flink bylaws [3]) are gathered by the community. Thanks, Alexey [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode [2] https://lists.apache.org/thread/7gpyqvdnnbjwbh3vbk6b0pj38l91crvv [3] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals < https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals)
[jira] [Created] (FLINK-35750) The latency marker metrics aren't updated after failover
RocMarshal created FLINK-35750: -- Summary: The latency marker metrics aren't updated after failover Key: FLINK-35750 URL: https://issues.apache.org/jira/browse/FLINK-35750 Project: Flink Issue Type: Bug Components: Runtime / Metrics Affects Versions: 1.19.1, 1.18.1, 1.20.0 Reporter: RocMarshal Described in https://docs.google.com/document/d/1WbAgdj8NdrSVChUuEywPgWZo3bAc3JKeVsN27AWDiL4/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Release flink-shaded 19.0, release candidate #1
+1 (binding) Thanks all for your votes. I'll conclude the vote in a separate email On Tue, 2 Jul 2024 at 10:39, Yuxin Tan wrote: > +1 (non-binding) > > - build from source > - verified website pr > - verified hash and signature > > Best, > Yuxin > > > weijie guo 于2024年7月2日周二 15:27写道： > > > +1 (binding) > > > > - Verified hash and signature > > - Build from source > > - Verified website PR > > > > Best regards, > > > > Weijie > > > > > > Martijn Visser 于2024年7月1日周一 21:30写道： > > > > > +1 (binding) > > > > > > - Validated hashes > > > - Verified signature > > > - Verified that no binaries exist in the source archive > > > - Build the source with Maven > > > - Verified licenses > > > - Verified web PRs > > > > > > On Fri, Jun 28, 2024 at 2:02 PM Timo Walther > wrote: > > > > > > > +1 (binding) > > > > > > > > Thanks for fixing the JSON functions! > > > > > > > > Timo > > > > > > > > On 28.06.24 12:54, Dawid Wysakowicz wrote: > > > > > Hi everyone, > > > > > Please review and vote on the release candidate 1 for the version > > 19.0, > > > > as > > > > > follows: > > > > > [ ] +1, Approve the release > > > > > [ ] -1, Do not approve the release (please provide specific > comments) > > > > > > > > > > > > > > > The complete staging area is available for your review, which > > includes: > > > > > * JIRA release notes [1], > > > > > * the official Apache source release to be deployed to > > dist.apache.org > > > > [2], > > > > > which are signed with the key with fingerprint > > > > > EA93A435B4E2C9B4C9F533F631D2DD10BFC15A2D [3], > > > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > > > * source code tag release-19.0-rc1 [5], > > > > > * website pull request listing the new release [6]. > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > majority > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > Thanks, > > > > > Dawid > > > > > > > > > > [1] > https://issues.apache.org/jira/projects/FLINK/versions/12353853 > > > > > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-shaded-19.0-rc1 > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > > [4] > > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1743/ > > > > > [5] > > > https://github.com/apache/flink-shaded/releases/tag/release-19.0-rc1 > > > > > [6] https://github.com/apache/flink-web/pull/749 > > > > > > > > > > > > > > > > > > >
[RESULT][VOTE] Release flink-shaded 19.0, release candidate #1
I'm happy to announce that we have unanimously approved this release. There are 6 approving votes, 4 of which are binding: * Timo Walther (binding) * Martijn Visser (binding) * Weijie Guo (binding) * Dawid Wysakowicz (binding) * Sergey Nuyanzin (non-binding) * Yuxin Tan (non-binding) There are no disapproving votes. Thanks everyone! Dawid
[jira] [Created] (FLINK-35751) Migrate SplitPythonConditionFromCorrelateRule
Jacky Lau created FLINK-35751: - Summary: Migrate SplitPythonConditionFromCorrelateRule Key: FLINK-35751 URL: https://issues.apache.org/jira/browse/FLINK-35751 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 2.0.0 Reporter: Jacky Lau Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35752) Migrate PythonUtil
Jacky Lau created FLINK-35752: - Summary: Migrate PythonUtil Key: FLINK-35752 URL: https://issues.apache.org/jira/browse/FLINK-35752 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 2.0.0 Reporter: Jacky Lau Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35753) ParquetColumnarRowInputFormatTest.testContinuousRepetition(int) failed due to ArrayIndexOutOfBoundsException
Weijie Guo created FLINK-35753: -- Summary: ParquetColumnarRowInputFormatTest.testContinuousRepetition(int) failed due to ArrayIndexOutOfBoundsException Key: FLINK-35753 URL: https://issues.apache.org/jira/browse/FLINK-35753 Project: Flink Issue Type: Bug Components: Build System / CI Affects Versions: 1.20.0 Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35754) SqlGatewayE2ECase.testMaterializedTableInFullMode failed due to Internal Server Error
Weijie Guo created FLINK-35754: -- Summary: SqlGatewayE2ECase.testMaterializedTableInFullMode failed due to Internal Server Error Key: FLINK-35754 URL: https://issues.apache.org/jira/browse/FLINK-35754 Project: Flink Issue Type: Bug Affects Versions: 1.20.0 Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35755) Upgrade flink-shaded to 19.0
Dawid Wysakowicz created FLINK-35755: Summary: Upgrade flink-shaded to 19.0 Key: FLINK-35755 URL: https://issues.apache.org/jira/browse/FLINK-35755 Project: Flink Issue Type: Sub-task Reporter: Dawid Wysakowicz Assignee: Dawid Wysakowicz Fix For: 1.20.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released
The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.9.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Release blogpost: https://flink.apache.org/2024/07/02/apache-flink-kubernetes-operator-1.9.0-release-announcement/ The release is available for download at: https://flink.apache.org/downloads.html Maven artifacts for Flink Kubernetes Operator can be found at: https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator Official Docker image for Flink Kubernetes Operator can be found at: https://hub.docker.com/r/apache/flink-kubernetes-operator The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354417 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Gyula Fora
[VOTE] FLIP-465: Introduce DESCRIBE FUNCTION
Hello everyone, I would like to start a vote on FLIP-465 [1]. It proposes adding SQL syntax that would allow users to describe the metadata of a given function. The vote will be open for at least 72 hours (Saturday, July 6th of July 2024, 12:30 PST) unless there is an objection or insufficient votes. Thanks, Natea
Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION
Sorry I forgot to include the FLIP [1] and the mailing thread discussion link [2] in my previous email. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph Thanks, Natea On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada wrote: > Hello everyone, > > I would like to start a vote on FLIP-465 [1]. It proposes adding SQL > syntax that would allow users to describe the metadata of a given function. > > The vote will be open for at least 72 hours (Saturday, July 6th of July > 2024, > 12:30 PST) unless there is an objection or insufficient votes. > > Thanks, > Natea > >
Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released
Thank you, Gyula! 🥳 Cheers On Wed, Jul 3, 2024 at 8:00 AM Gyula Fóra wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.9.0. > > The Flink Kubernetes Operator allows users to manage their Apache Flink > applications and their lifecycle through native k8s tooling like kubectl. > > Release blogpost: > https://flink.apache.org/2024/07/02/apache-flink-kubernetes-operator-1.9.0-release-announcement/ > > The release is available for download at: > https://flink.apache.org/downloads.html > > Maven artifacts for Flink Kubernetes Operator can be found at: > https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator > > Official Docker image for Flink Kubernetes Operator can be found at: > https://hub.docker.com/r/apache/flink-kubernetes-operator > > The full release notes are available in Jira: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354417 > > We would like to thank all contributors of the Apache Flink community who > made this release possible! > > Regards, > Gyula Fora >
Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION
Hi Natea, Looks good to me! +1 (non-binding). Cheers, Jim On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada wrote: > Sorry I forgot to include the FLIP [1] and the mailing thread discussion > link [2] in my previous email. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION > [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph > > Thanks, > Natea > > On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada < > nbesh...@confluent.io> > wrote: > > > Hello everyone, > > > > I would like to start a vote on FLIP-465 [1]. It proposes adding SQL > > syntax that would allow users to describe the metadata of a given > function. > > > > The vote will be open for at least 72 hours (Saturday, July 6th of July > > 2024, > > 12:30 PST) unless there is an objection or insufficient votes. > > > > Thanks, > > Natea > > > > >
Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released
Thanks for the effort, Guyla! Best Regards Peter Huang On Wed, Jul 3, 2024 at 12:48 PM Őrhidi Mátyás wrote: > Thank you, Gyula! 🥳 > Cheers > On Wed, Jul 3, 2024 at 8:00 AM Gyula Fóra wrote: > > > The Apache Flink community is very happy to announce the release of > Apache > > Flink Kubernetes Operator 1.9.0. > > > > The Flink Kubernetes Operator allows users to manage their Apache Flink > > applications and their lifecycle through native k8s tooling like kubectl. > > > > Release blogpost: > > > https://flink.apache.org/2024/07/02/apache-flink-kubernetes-operator-1.9.0-release-announcement/ > > > > The release is available for download at: > > https://flink.apache.org/downloads.html > > > > Maven artifacts for Flink Kubernetes Operator can be found at: > > > https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator > > > > Official Docker image for Flink Kubernetes Operator can be found at: > > https://hub.docker.com/r/apache/flink-kubernetes-operator > > > > The full release notes are available in Jira: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354417 > > > > We would like to thank all contributors of the Apache Flink community who > > made this release possible! > > > > Regards, > > Gyula Fora > > >
Re: [VOTE] FLIP-456: CompiledPlan support for Batch Execution Mode
+1 (non-binding) Best regards, Yuepeng Pan At 2024-07-03 01:46:13, "Sergey Nuyanzin" wrote: >Thanks for driving this > >+1 (binding) > >On Tue, Jul 2, 2024, 11:21 Martijn Visser wrote: > >> +1 (binding) >> >> On Mon, Jul 1, 2024 at 7:00 PM Jim Hughes >> wrote: >> >> > Hi Alexey, >> > >> > +1 (non-binding) >> > >> > I'm looking forward to parity between streaming and batch bound for >> > compiled plans! >> > >> > Cheers, >> > >> > Jim >> > >> > On Mon, Jul 1, 2024 at 12:55 PM Alexey Leonov-Vendrovskiy < >> > vendrov...@gmail.com> wrote: >> > >> > > Hello everyone, >> > > >> > > We had a good discussion of FLIP-456: CompiledPlan support for Batch >> > > Execution Mode [1]. Discussion thread is here: [2]. >> > > >> > > Let's start voting on it. The vote will be open for at least 72 >> > > hours unless there is an objection or insufficient votes. The FLIP will >> > be >> > > considered accepted if 3 binding votes (from active committers >> according >> > to >> > > the Flink bylaws [3]) are gathered by the community. >> > > >> > > Thanks, >> > > Alexey >> > > >> > > [1] >> > > >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode >> > > [2] https://lists.apache.org/thread/7gpyqvdnnbjwbh3vbk6b0pj38l91crvv >> > > [3] >> > > >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals >> > > < >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals) >> > > > >> > > >> > >>
Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION
Hi Natea, thanks for driving it. +1 (non-binding). Jim Hughes 于2024年7月4日周四 04:41写道： > Hi Natea, > > Looks good to me! > > +1 (non-binding). > > Cheers, > > Jim > > On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada > wrote: > > > Sorry I forgot to include the FLIP [1] and the mailing thread discussion > > link [2] in my previous email. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION > > [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph > > > > Thanks, > > Natea > > > > On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada < > > nbesh...@confluent.io> > > wrote: > > > > > Hello everyone, > > > > > > I would like to start a vote on FLIP-465 [1]. It proposes adding SQL > > > syntax that would allow users to describe the metadata of a given > > function. > > > > > > The vote will be open for at least 72 hours (Saturday, July 6th of July > > > 2024, > > > 12:30 PST) unless there is an objection or insufficient votes. > > > > > > Thanks, > > > Natea > > > > > > > > >
[jira] [Created] (FLINK-35756) FileSystemTableSourceStreamingITCase.testSourceWithRegexPattern produced no output for 900 seconds
Weijie Guo created FLINK-35756: -- Summary: FileSystemTableSourceStreamingITCase.testSourceWithRegexPattern produced no output for 900 seconds Key: FLINK-35756 URL: https://issues.apache.org/jira/browse/FLINK-35756 Project: Flink Issue Type: Bug Components: Build System / CI Affects Versions: 1.18.1 Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35757) support read new tables when scanNewlyTableAdded is enabled
shikai.wang created FLINK-35757: --- Summary: support read new tables when scanNewlyTableAdded is enabled Key: FLINK-35757 URL: https://issues.apache.org/jira/browse/FLINK-35757 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: shikai.wang Fix For: cdc-3.2.0 currently if we set tableList with regex like db.*, after snapshot finished, we can't capture tables when create new tables in db. instead,we need to restart to snapshot new table. this is very troublesome. so we should support dynamically capture new table that match the regex when enable scanNewlyTableAdded. https://github.com/apache/flink-cdc/discussions/3338 -- This message was sent by Atlassian Jira (v8.20.10#820010)
