Re: [DISCUSSION] Scan mode in Table Store for Flink Stream and Batch job

Thanks Shammon! Your summary was very good and very detailed. I thought about it again. ## Solution 1 Actually, according to what you said, there should be so many modes in theory. - Runtime-mode: streaming or batch. - Range: full or incremental. - Position: Latest, timestamp, snapshot-id, compa

Re: [DISCUSS] Cleaning up HighAvailabilityServices interface to reflect the per-JM-process LeaderElection

Hi Dong, see my answers below. Regarding "Interface change might affect other projects that customize HA > services", are you referring to those projects which hack into Flink's > source code (as opposed to using Flink's public API) to customize HA > services? Yes, the proposed change might affe

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi Shammon, Do I understand it correctly, that you effectively want to expand the checkpoint alignment mechanism across many different jobs and hand over checkpoint barriers from upstream to downstream jobs using the intermediate tables? Re the watermarks for the "Rejected Alternatives". I don't

[jira] [Created] (FLINK-30349) Sync missing HBase e2e tests to external repo

Martijn Visser created FLINK-30349: -- Summary: Sync missing HBase e2e tests to external repo Key: FLINK-30349 URL: https://issues.apache.org/jira/browse/FLINK-30349 Project: Flink Issue Type:

[jira] [Created] (FLINK-30350) Write dependency-reduced pom to default directory

Chesnay Schepler created FLINK-30350: Summary: Write dependency-reduced pom to default directory Key: FLINK-30350 URL: https://issues.apache.org/jira/browse/FLINK-30350 Project: Flink Iss

Re: [VOTE] Release flink-connector-aws v4.0.0, release candidate #1

+1 (binding) - clean source release - builds from source - source matches git tag - all expected maven artifacts present - maven artifacts have correct Flink version suffix - releases notes are good - PR is good Not a blocking issue, but the source NOTICE currently says "Apache Flink", where it

[jira] [Created] (FLINK-30351) Enable test cases again

Matthias Pohl created FLINK-30351: - Summary: Enable test cases again Key: FLINK-30351 URL: https://issues.apache.org/jira/browse/FLINK-30351 Project: Flink Issue Type: Sub-task Comp

Re: [VOTE] Release flink-connector-aws v4.0.0, release candidate #1

+1 (binding) - Validated hashes - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven - Verified licenses - Verified web PR On Fri, Dec 9, 2022 at 12:16 PM Chesnay Schepler wrote: > +1 (binding) > > - clean source release > - builds from sour

[jira] [Created] (FLINK-30352) [Connectors][Elasticsearch] Document missing configuration properties

Andriy Redko created FLINK-30352: Summary: [Connectors][Elasticsearch] Document missing configuration properties Key: FLINK-30352 URL: https://issues.apache.org/jira/browse/FLINK-30352 Project: Flink

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

Hi Hang, Thanks for the FLIP! The FLIP looks good and it is pretty informative. I have just two minor comments regarding names: - Would it be useful to rename the config key as *metrics.scope.jm.job.operator-coordinator* for consistency with *metrics.scope.jm.job *(which is not named as *jm-job)?

[jira] [Created] (FLINK-30353) Enable concurrency for external connector repositories

Martijn Visser created FLINK-30353: -- Summary: Enable concurrency for external connector repositories Key: FLINK-30353 URL: https://issues.apache.org/jira/browse/FLINK-30353 Project: Flink Is

Re: [DISCUSS] FLIP-277: Native GlueCatalog Support in Flink

Hi Samrat, Thanks a lot for driving the new catalog, and sorry for jumping into the discussion late. As Flink SQL is becoming the first-class citizen of the Flink API, we are planning to push Catalog to become the first-class citizen of the connector instead of Source & Sink. For Flink SQL users,

Re: [VOTE] Release flink-connector-jdbc v3.0.0, release candidate #1

That was because the number of changes between 1.16.0 and master were limited.

Re: [DISCUSS] Cleaning up HighAvailabilityServices interface to reflect the per-JM-process LeaderElection

Hi Matthias, Thanks for the proposal! I am in favor of cleaning up this interface, and It seems a bit cumbersome now. Especially, the implementation of per-component leader election has been removed from our current code path. To be honest, I don't like the per-component approach. I'm even often

[jira] [Created] (FLINK-30354) Reducing the number of ThreadPools in LookupFullCache and related cache-loading classes

Matthias Pohl created FLINK-30354: - Summary: Reducing the number of ThreadPools in LookupFullCache and related cache-loading classes Key: FLINK-30354 URL: https://issues.apache.org/jira/browse/FLINK-30354

Re: [VOTE] Release flink-connector-aws v4.0.0, release candidate #1

+1 (binding) - Validated hashes - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven - Verified licenses - Run the following DataStream apps: - KDS > KDS (v1 legacy) - KDS > KDS (v2) - KDS > KDF - KDS (EFO) > DDB On Fri, Dec 9, 2022 at

[jira] [Created] (FLINK-30355) crictl causes long wait in e2e tests

Matthias Pohl created FLINK-30355: - Summary: crictl causes long wait in e2e tests Key: FLINK-30355 URL: https://issues.apache.org/jira/browse/FLINK-30355 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-30356) Update NOTICE files to say "Apache Flink AWS connectors"

Danny Cranmer created FLINK-30356: - Summary: Update NOTICE files to say "Apache Flink AWS connectors" Key: FLINK-30356 URL: https://issues.apache.org/jira/browse/FLINK-30356 Project: Flink Is

Re: [DISCUSS] Cleaning up HighAvailabilityServices interface to reflect the per-JM-process LeaderElection

I generally agree that the internals of the HA services are currently too complex, but I'm wondering if the proposal doesn't go a bit too far to resolve those. Is there maybe some way we can refactor things internally to reduce complexity while keeping the per-component semantics? Ultimately,

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

As a whole I feel like this FLIP is overly complicated. A dedicated coordinator MG implementation is overkill; it could just re-use the existing Task/OperatorMGs to create the same structure we have on TMs, similar to what we did with the Job MG. However, I'm not convinced that this is require

Re: [VOTE] Release flink-connector-aws v4.0.0, release candidate #1

Thanks all, this vote thread is now closed. I have raised an issue [1] to fix the NOTICE file issue Chesnay detected. Thanks, Danny [1] https://issues.apache.org/jira/browse/FLINK-30356 On Fri, Dec 9, 2022 at 4:03 PM Danny Cranmer wrote: > +1 (binding) > > - Validated hashes > - Verified sign

[RESULT] [VOTE] flink-connector-aws v4.0.0, release candidate #1

I'm happy to announce that we have unanimously approved this release. There are 4 approving votes, 3 of which are binding: * Hong (non-binding) * Chesnary (binding) * Martijn (binding) * Danny (binding) There are no disapproving votes. Thanks everyone! Danny

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

Hi, Dong, Thanks for your suggestion. I plan to rename this scope like this : public static final ConfigOption SCOPE_NAMING_OPERATOR_COORDINATOR = key("metrics.scope.operator-coordinator") .stringType() .defaultValue(".jobmanager...coordinator") .withDescr

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

Hi, Chesnay, Thanks for your reply. Actually we can not reuse the Task/OperatorMG for the OperatorCoordinator. There are mainly two reasons. First of all, the scopes of these metric groups are not suitable for the OperatorCoordinator. It should be ".jobmanager...coordinator". Secondly, there are

Re: [DISCUSS] Cleaning up HighAvailabilityServices interface to reflect the per-JM-process LeaderElection

Hi Chesnay, I like the use-cases (e.g. running multiple UIs for load-balancing purposes) mentioned. On the other hand, these are probably not high-priority features, and we don't know when the community will get to implement these features. It seems a bit over-design to add implementation complexi

Re: [DISCUSS] Cleaning up HighAvailabilityServices interface to reflect the per-JM-process LeaderElection

Hi Matthias, Thanks for the explanation. I was trying to understand the concrete user-facing benefits of preserving the flexibility of per-component leader election. Now I get that maybe they want to scale those components independently, and maybe run the UI in an environment that is more accessib

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

Hi Chesney, Just to double check with you, OperatorCoordinatorMetricGroup (annotated as @PublicEvolving) has already been introduced into Flink by FLIP-179 . And that FLIP has got you +1.. Do you me

[jira] [Created] (FLINK-30357) Wrong link in connector/jdbc doc.

Aiden Gong created FLINK-30357: -- Summary: Wrong link in connector/jdbc doc. Key: FLINK-30357 URL: https://issues.apache.org/jira/browse/FLINK-30357 Project: Flink Issue Type: Improvement