[jira] [Created] (FLINK-35184) Hash collision inside MiniBatchStreamingJoin operator
Roman Boyko created FLINK-35184:
---
Summary: Hash collision inside MiniBatchStreamingJoin operator
Key: FLINK-35184
URL: https://issues.apache.org/jira/browse/FLINK-35184
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.19.0
Reporter: Roman Boyko
The hash collision is possible for InputSideHasNoUniqueKeyBundle. To reproduce
it just launch the following test within StreamingMiniBatchJoinOperatorTest:
{code:java}
@Tag("miniBatchSize=6")
@Test
public void testInnerJoinWithNoUniqueKeyHashCollision(TestInfo testInfo) throws
Exception {
leftTypeInfo =
InternalTypeInfo.of(
RowType.of(
new LogicalType[] {new IntType(), new BigIntType()},
new String[] {"id1", "val1"}));
rightTypeInfo =
InternalTypeInfo.of(
RowType.of(
new LogicalType[] {new IntType(), new BigIntType()},
new String[] {"id2", "val2"}));
leftKeySelector =
HandwrittenSelectorUtil.getRowDataSelector(
new int[] {0},
leftTypeInfo.toRowType().getChildren().toArray(new
LogicalType[0]));
rightKeySelector =
HandwrittenSelectorUtil.getRowDataSelector(
new int[] {0},
rightTypeInfo.toRowType().getChildren().toArray(new
LogicalType[0]));
joinKeyTypeInfo = InternalTypeInfo.of(new IntType());
super.beforeEach(testInfo);
testHarness.setStateTtlProcessingTime(1);
testHarness.processElement2(insertRecord(1, 1L));
testHarness.processElement1(insertRecord(1, 4294967296L));
testHarness.processElement2(insertRecord(1, 4294967296L));
testHarness.processElement2(deleteRecord(1, 1L));
testHarness.close();
assertor.shouldEmit(testHarness, rowOfKind(RowKind.INSERT, 1, 4294967296L,
1, 4294967296L));
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] Release flink-connector-aws v4.3.0, release candidate #2
+1 (non-binding) - Validated checksum hash - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven and jdk8 - Verified web PR - Check that the jar is built by jdk8 Best, Hang Danny Cranmer 于2024年4月19日周五 18:08写道： > Hi everyone, > > Please review and vote on release candidate #2 for flink-connector-aws > v4.3.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > This version supports Flink 1.18 and 1.19. > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > which are signed with the key with fingerprint 125FD8DB [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v4.3.0-rc2 [5], > * website pull request listing the new release [6]. > * CI build of the tag [7]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Release Manager > > [1] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353793 > [2] > https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.3.0-rc2 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] > https://repository.apache.org/content/repositories/orgapacheflink-1721/ > [5] https://github.com/apache/flink-connector-aws/releases/tag/v4.3.0-rc2 > [6] https://github.com/apache/flink-web/pull/733 > [7] https://github.com/apache/flink-connector-aws/actions/runs/8751694197 >
Re: [VOTE] Release flink-connector-mongodb v1.2.0, release candidate #2
+1 (non-binding) - Validated checksum hash - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven and jdk8 - Verified web PR - Check that the jar is built by jdk8 Best, Hang Ahmed Hamdy 于2024年4月18日周四 21:40写道： > +1 (non-binding) > > - verified hashes and checksums > - verified signature > - verified source contains no binaries > - tag exists in github > - reviewed web PR > > > Best Regards > Ahmed Hamdy > > > On Thu, 18 Apr 2024 at 11:21, Danny Cranmer > wrote: > > > Hi everyone, > > > > Please review and vote on the release candidate #2 for v1.2.0, as > follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This release supports Flink 1.18 and 1.19. > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], > > which are signed with the key with fingerprint 125FD8DB [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v1.2.0-rc2 [5], > > * website pull request listing the new release [6]. > > * CI build of tag [7]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Danny > > > > [1] > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354192 > > [2] > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-mongodb-1.2.0-rc2 > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1719/ > > [5] > > > https://github.com/apache/flink-connector-mongodb/releases/tag/v1.2.0-rc2 > > [6] https://github.com/apache/flink-web/pull/735 > > [7] > > > https://github.com/apache/flink-connector-mongodb/actions/runs/8735987710 > > >
Re: [VOTE] Release flink-connector-mongodb v1.2.0, release candidate #2
+1 (non-binding) - Validated checksum hash - Verified signature - Tag is present - Build successful with jdk8, jdk11 and jdk17 - Checked the dist jar was built by jdk8 - Reviewed web PR Best, Jiabao Hang Ruan 于2024年4月21日周日 21:33写道： > +1 (non-binding) > > - Validated checksum hash > - Verified signature > - Verified that no binaries exist in the source archive > - Build the source with Maven and jdk8 > - Verified web PR > - Check that the jar is built by jdk8 > > Best, > Hang > > Ahmed Hamdy 于2024年4月18日周四 21:40写道： > > > +1 (non-binding) > > > > - verified hashes and checksums > > - verified signature > > - verified source contains no binaries > > - tag exists in github > > - reviewed web PR > > > > > > Best Regards > > Ahmed Hamdy > > > > > > On Thu, 18 Apr 2024 at 11:21, Danny Cranmer > > wrote: > > > > > Hi everyone, > > > > > > Please review and vote on the release candidate #2 for v1.2.0, as > > follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > This release supports Flink 1.18 and 1.19. > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source release to be deployed to dist.apache.org > > > [2], > > > which are signed with the key with fingerprint 125FD8DB [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tag v1.2.0-rc2 [5], > > > * website pull request listing the new release [6]. > > > * CI build of tag [7]. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Danny > > > > > > [1] > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354192 > > > [2] > > > > > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-mongodb-1.2.0-rc2 > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] > > > > https://repository.apache.org/content/repositories/orgapacheflink-1719/ > > > [5] > > > > > > https://github.com/apache/flink-connector-mongodb/releases/tag/v1.2.0-rc2 > > > [6] https://github.com/apache/flink-web/pull/735 > > > [7] > > > > > > https://github.com/apache/flink-connector-mongodb/actions/runs/8735987710 > > > > > >
Re: [VOTE] Release flink-connector-jdbc v3.2.0, release candidate #2
+1 (non-binding) - Validated checksum hash - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven and jdk8 - Verified web PR - Check that the jar is built by jdk8 Best, Hang Ahmed Hamdy 于2024年4月18日周四 21:37写道： > +1 (non-binding) > > - Verified Checksums and hashes > - Verified Signatures > - No binaries in source > - Build source > - Github tag exists > - Reviewed Web PR > > > Best Regards > Ahmed Hamdy > > > On Thu, 18 Apr 2024 at 11:22, Danny Cranmer > wrote: > > > Sorry for typos: > > > > > Please review and vote on the release candidate #1 for the version > 3.2.0, > > as follows: > > Should be "release candidate #2" > > > > > * source code tag v3.2.0-rc1 [5], > > Should be "source code tag v3.2.0-rc2" > > > > Thanks, > > Danny > > > > On Thu, Apr 18, 2024 at 11:19 AM Danny Cranmer > > wrote: > > > > > Hi everyone, > > > > > > Please review and vote on the release candidate #1 for the version > 3.2.0, > > > as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > This release supports Flink 1.18 and 1.19. > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source release to be deployed to dist.apache.org > > > [2], which are signed with the key with fingerprint 125FD8DB [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tag v3.2.0-rc1 [5], > > > * website pull request listing the new release [6]. > > > * CI run of tag [7]. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Danny > > > > > > [1] > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353143 > > > [2] > > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.2.0-rc2 > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] > > > > https://repository.apache.org/content/repositories/orgapacheflink-1718/ > > > [5] > > https://github.com/apache/flink-connector-jdbc/releases/tag/v3.2.0-rc2 > > > [6] https://github.com/apache/flink-web/pull/734 > > > [7] > > https://github.com/apache/flink-connector-jdbc/actions/runs/8736019099 > > > > > >
Re: [VOTE] Release flink-connector-gcp-pubsub v3.1.0, release candidate #1
+1 (non-binding) - Validated checksum hash - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven and jdk8 - Verified web PR - Check that the jar is built by jdk8 Best, Hang Ahmed Hamdy 于2024年4月18日周四 20:01写道： > Hi Danny, > +1 (non-binding) > > - verified hashes and checksums > - verified signature > - verified source contains no binaries > - tag exists in github > - reviewed web PR > > Best Regards > Ahmed Hamdy > > > On Thu, 18 Apr 2024 at 11:32, Danny Cranmer > wrote: > > > Hi everyone, > > > > Please review and vote on release candidate #1 for > > flink-connector-gcp-pubsub v3.1.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This release supports Flink 1.18 and 1.19. > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], > > which are signed with the key with fingerprint 125FD8DB [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v3.1.0-rc1 [5], > > * website pull request listing the new release [6]. > > * CI build of the tag [7]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Danny > > > > [1] > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353813 > > [2] > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-gcp-pubsub-3.1.0-rc1 > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > https://repository.apache.org/content/repositories/orgapacheflink-1720 > > [5] > > > > > https://github.com/apache/flink-connector-gcp-pubsub/releases/tag/v3.1.0-rc1 > > [6] https://github.com/apache/flink-web/pull/736/files > > [7] > > > > > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8735952883 > > >
[jira] [Created] (FLINK-35185) Resuming Externalized Checkpoint(rocks, incremental, no parallelism change) end-to-end test failed
Weijie Guo created FLINK-35185: -- Summary: Resuming Externalized Checkpoint(rocks, incremental, no parallelism change) end-to-end test failed Key: FLINK-35185 URL: https://issues.apache.org/jira/browse/FLINK-35185 Project: Flink Issue Type: Bug Components: Build System / CI Affects Versions: 1.20.0 Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0
Hi Flink devs, I would like to start a discussion on FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0 https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0 This FLIP proposes upgrading the version of FRocksDB in the Flink Project from 6.20.3 to 8.10.0. The FLIP mainly introduces the main benefits of upgrading FRocksDB, including the use of IngestDB which can improve Rescaling performance by more than 10 times in certain scenarios, as well as other potential optimization points such as async_io, blob db, and tiered storage.The FLIP also presented test results based on RocksDB 8.10, including StateBenchmark and Nexmark tests. Overall, upgrading FRocksDB may result in a small regression of write performance( which is a very small part of the overall overhead), but it can bring many important performance benefits. So we hope to upgrade the version of FRocksDB through this FLIP. Looking forward to everyone's feedback and suggestions. Thank you! -- Best regards, Yue
[RESULT][VOTE] FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines
Hi, Dev I'm happy to announce that FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines[1] has been accepted with 13 approving votes (8 binding) [2] - Ron Liu(binding) - Feng Jin - Rui Fan(binding) - Yuepeng Pan - Ahmed Hamdy - Ferenc Csaky - Lincoln Lee(binding) - Leonard Xu(binding) - Jark Wu(binding) - Yun Tang(binding) - Jinsong Li(binding) - Zhongqiang Gong - Martijn Visser(binding) [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines [2] https://lists.apache.org/thread/woj27nsmx5xd7p87ryfo8h6gx37n3wlx Best, Ron
[DISCUSS] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table
Hi, Dev I would like to start a discussion about FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table. In FLIP-435[1], we proposed Materialized Table, which has two types of data refresh modes: Full Refresh & Continuous Refresh Mode. In Full Refresh mode, the Materialized Table relies on a workflow scheduler to perform periodic refresh operation to achieve the desired data freshness. There are numerous open-source workflow schedulers available, with popular ones including Airflow and DolphinScheduler. To enable Materialized Table to work with different workflow schedulers, we propose a pluggable workflow scheduler interface for Materialized Table in this FLIP. For more details, see FLIP-448 [2]. Looking forward to your feedback. [1] https://lists.apache.org/thread/c1gnn3bvbfs8v1trlf975t327s4rsffs [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-448%3A+Introduce+Pluggable+Workflow+Scheduler+Interface+for+Materialized+Table Best, Ron
[jira] [Created] (FLINK-35186) Create State V2 from new StateDescriptor
Zakelly Lan created FLINK-35186: --- Summary: Create State V2 from new StateDescriptor Key: FLINK-35186 URL: https://issues.apache.org/jira/browse/FLINK-35186 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends, Runtime / Task Reporter: Zakelly Lan Assignee: Zakelly Lan -- This message was sent by Atlassian Jira (v8.20.10#820010)
