[DISCUSS] FLIP-278: Hybrid Source Connector

2022-12-15 Thread Ran Tao
Hi guys. HybridSource is a good feature, but now released version did not support table & sql api for a long time. I have wrote a discussed FLIP. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235836225 Sorry for my unclear subject of previous email, so here i have copied the r

Re: [DISCUSS] FLIP-278: Hybrid Source Connector

2022-12-15 Thread Ran Tao
Fyi. This flip using index as child source option prefix because we may use the same connector as hybrid child sources. e.g. create table hybrid_source( f0 varchar, f1 varchar, f2 bigint ) with( 'connector'='hybrid', 'sources'='filesystem,filesystem', '0.path' = '/tmp/a.csv', '0.format' =

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2022-12-15 Thread Piotr Nowojski
Hi Shammon, > The following is a simple example. Data is transferred between ETL1, ETL2 and ETL3 in Intermediate Table by Timestamp. > [image: simple_example.jpg] This time it's your image that doesn't want to load :) > Timestamp Barrier Does it have to be combining watermarks and checkpoint b

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

2022-12-15 Thread Hang Ruan
Hi, Zhu Zhu, Thanks for your feedback! The OperatorCoordinator implementations are different. And their metrics are much different too. We try to find the common metrics and put them in the OperatorCoordinatorMetricGroup. If most developers think we do not need these common metrics, removing the

[jira] [Created] (FLINK-30425) Generalize token receive side

2022-12-15 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-30425: - Summary: Generalize token receive side Key: FLINK-30425 URL: https://issues.apache.org/jira/browse/FLINK-30425 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-30426) Archunit can't find any tests in Pulsar repository

2022-12-15 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30426: -- Summary: Archunit can't find any tests in Pulsar repository Key: FLINK-30426 URL: https://issues.apache.org/jira/browse/FLINK-30426 Project: Flink Issue

[jira] [Created] (FLINK-30427) Pulsar SQL connector lists not bundled dependencies

2022-12-15 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-30427: Summary: Pulsar SQL connector lists not bundled dependencies Key: FLINK-30427 URL: https://issues.apache.org/jira/browse/FLINK-30427 Project: Flink I

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2022-12-15 Thread Shammon FY
Hi Piotr, It's kind of amazing about the image, it's a simple example and I have to put it in a document https://bytedance.feishu.cn/docx/FC6zdq0eqoWxHXxli71cOxe9nEe?from=from_copylink :) > Does it have to be combining watermarks and checkpoint barriers together? It's an interesting question. As

[jira] [Created] (FLINK-30428) SparkReadITCase.testSnapshotsTable is unstable

2022-12-15 Thread Shammon (Jira)
Shammon created FLINK-30428: --- Summary: SparkReadITCase.testSnapshotsTable is unstable Key: FLINK-30428 URL: https://issues.apache.org/jira/browse/FLINK-30428 Project: Flink Issue Type: Bug

Re: [VOTE] Release flink-connector-pulsar, release candidate #3

2022-12-15 Thread Martijn Visser
Hi all, Unfortunately an issue with dependency bundling for the SQL version was found [1]. Cancelling this RC and will create another one shortly... Best regards, Martijn [1] https://issues.apache.org/jira/browse/FLINK-30427 On Tue, Dec 13, 2022 at 1:29 PM Martijn Visser wrote: > Hi everyone

[VOTE] Release flink-connector-pulsar, release candidate #4

2022-12-15 Thread Martijn Visser
Hi everyone, Please review and vote on the release candidate #4 for the version 3.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

[jira] [Created] (FLINK-30429) Flink command stops with IllegalArgumentException

2022-12-15 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-30429: - Summary: Flink command stops with IllegalArgumentException Key: FLINK-30429 URL: https://issues.apache.org/jira/browse/FLINK-30429 Project: Flink Issue Typ

Re: [VOTE] Release flink-connector-pulsar, release candidate #4

2022-12-15 Thread Martijn Visser
It seems that some files got accidentally added during the tagging process of the source :( Checking once more... On Thu, Dec 15, 2022 at 3:46 PM Martijn Visser wrote: > Hi everyone, > Please review and vote on the release candidate #4 for the version 3.0.0, > as follows: > [ ] +1, Approve the r

[DICSUSS] Releasing Flink Opensearch Connector v1.0.0

2022-12-15 Thread Andriy Redko
Hello All, I would like to start discussing the first release of the new Flink Opensearch Connector [1] 1.0.0. As per the connector versioning strategy [2] from the new flink-connector-opensearch repository [3]. As of today, no outstanding issues are left for the first release to happen. I propos

Re: [DICSUSS] Releasing Flink Opensearch Connector v1.0.0

2022-12-15 Thread Martijn Visser
Hi Andriy, +1 to release the first version. Happy to help with the release (feels like I'm doing nothing else anyway :D) I would indeed go for version 1.0.0 to start since it's a new connector. Cheers, Martijn On Thu, Dec 15, 2022 at 4:16 PM Andriy Redko wrote: > Hello All, > > I would like

[jira] [Created] (FLINK-30430) Remove the useless dependencies in flink-connector-pulsar's dependencyManagement

2022-12-15 Thread Yufan Sheng (Jira)
Yufan Sheng created FLINK-30430: --- Summary: Remove the useless dependencies in flink-connector-pulsar's dependencyManagement Key: FLINK-30430 URL: https://issues.apache.org/jira/browse/FLINK-30430 Projec

[VOTE] Release flink-connector-pulsar, release candidate #5

2022-12-15 Thread Martijn Visser
Hi everyone, Please review and vote on the release candidate #5 for the version 3.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

Re: [VOTE] Release flink-connector-pulsar, release candidate #5

2022-12-15 Thread Yufan Sheng
+1. And I can see that the pending issue FLINK-30430 is in the 3.0 release. Since the first 3.0 release only externalize the flink-connector-pulsar, we can remove the FLINK-30430 from the 3.0.0 and add it back after 3.0.0 getting released. On Thu, Dec 15, 2022 at 11:41 PM Martijn Visser wrote: >

Re: [DICSUSS] Releasing Flink Opensearch Connector v1.0.0

2022-12-15 Thread Danny Cranmer
Hey, I will also volunteer to be release manager, since Martijn has been doing all of them I am happy to take some of the work! We launched the new flink-connector-dynamodb connector with support for Flink 1.15. The versioning strategy says to support the latest 2 Flink versions [1]. If this conn

Re: [DICSUSS] Releasing Flink Opensearch Connector v1.0.0

2022-12-15 Thread Andriy Redko
Thanks a lot Danny. The connector builds against 1.16 and 1.17 (snapshot) right now, I think we could explore 1.15 support. The Opensearch is backed by AWS as of today, right, but it is a community driven project under OpenSearch Project organization, so I would not tight it to AWS anyhow (and

[DISCUSS] Release Flink 1.16.1

2022-12-15 Thread Martijn Visser
Hi everyone, I would like to open a discussion about releasing Flink 1.16.1. We've released Flink 1.16 at the end of October, but we already have 58 fixes listed for 1.16.1, including a blocker [1] on the environment variables and a number of critical issues. Some of the critical issues are relate

Re: [DISCUSS] Release Flink 1.16.1

2022-12-15 Thread David Anderson
Martijn, Thank you for bringing this up. From my (admittedly narrow) perspective, I'd like to see a release sooner rather than later, as there's an already merged bug fix I'd like to get released. David On Thu, Dec 15, 2022 at 1:53 PM Martijn Visser wrote: > Hi everyone, > > I would like to

Re: [SURVEY] Drop Share and Key_Shared subscription support in Pulsar connector

2022-12-15 Thread Martijn Visser
Hi Yufan, I'm also looping in the Dev mailing list for awareness. >From my perspective, it sounds like it makes more sense to drop the Shared and Key_Shared subscription. If it's currently unstable/not usable and there are alternatives that you can leverage with Flink (like the increased parallel

[jira] [Created] (FLINK-30431) JDBC Connector fails to reestablish the lost DB connection.

2022-12-15 Thread Sai Nadendla (Jira)
Sai Nadendla created FLINK-30431: Summary: JDBC Connector fails to reestablish the lost DB connection. Key: FLINK-30431 URL: https://issues.apache.org/jira/browse/FLINK-30431 Project: Flink

[jira] [Created] (FLINK-30432) HiveSoure doesn't need to maintain all splits when it's for parallelism inference

2022-12-15 Thread luoyuxia (Jira)
luoyuxia created FLINK-30432: Summary: HiveSoure doesn't need to maintain all splits when it's for parallelism inference Key: FLINK-30432 URL: https://issues.apache.org/jira/browse/FLINK-30432 Project: Fl

Re: [DISCUSS] Release Flink 1.16.1

2022-12-15 Thread ConradJam
Hi Martijn, Thank you for bringing this up. I'm in favor of this proposal, by the way, FLINK-30116 has been fixed Martijn Visser 于2022年12月16日周五 02:52写道: > Hi everyone, > > I would like to open a discussion about releasing Flink 1.16.1. We've > released Flink 1.16 at the end of October, but we

Re: [DISCUSS] FLIP-271: Autoscaling

2022-12-15 Thread ConradJam
Thanks to Gyula and Max for a great start, I'll try this feature out and I'll raise it on issue 😀 Maximilian Michels 于2022年12月15日周四 02:37写道: > A heads-up: Gyula just opened a PR with the code contribution based on the > design: https://github.com/apache/flink-kubernetes-operator/pull/484 > > We