Re: Working improving the REST API

Just adding the updates from the Slack thread - @SamratDeb and @JulietLee have expressed interest, and there is a PR ready for review! https://github.com/apache/flink/pull/22901 Regards, Hong On 3 Jul 2023, at 16:56, Teoh, Hong wrote: CAUTION: This email originated from outside of the

Working improving the REST API

Hi all, (This is a cross-post from a Slack message posted in #dev [1]. Posting here for visibility) Recently I’ve been looking into improving the REST API that Flink has, to make it more generally available for programmatic access (e.g. control functions) rather than just for the Flink dashboa

Re: [DISCUSS] Release 2.0 Work Items

Thanks Xintong for driving the effort. I’d add a +1 to reworking configs, as suggested by @Jark and @Chesnay, especially the types. We have various configs that encode Time / MemorySize that are Long instead! Regards, Hong > On 29 Jun 2023, at 16:19, Yuan Mei wrote: > > CAUTION: This email

Re: Async I/O: preserve stream order for requests on key level

Hi Juho, Thank you for bringing this up! Definitely +1 to this. We have had similar requests for the AsyncSink as well. As a side note, it would be useful to share the same implementation for both somehow, to prevent duplicate code. Happy to help with the implementation here. For the AsyncSink

[DISCUSS] Flink REST API improvements

Hi all, I have been looking at the Flink REST API implementation, and had some question on potential improvements. Looking to gather some thoughts: 1. Only use what is necessary. The GET /checkpoints API seems to be using the cached version of the entire Execution graph (stale data), when it co

Re: AsyncFunction vs Async Sink

share the same underlying implementation and the features like batching and rate limiting can benefit both? Best Lu On Wed, Jun 14, 2023 at 2:20 PM Teoh, Hong mailto:lian...@amazon.co.uk>> wrote: Hi Lu, Thanks for your question. See below for my understanding. I would recommend using the

Re: AsyncFunction vs Async Sink

Hi Lu, Thanks for your question. See below for my understanding. I would recommend using the Async Sink if you are writing to the external service as the final output of your job graph, and if you don’t have the ordered requirement that updates to the external system must be done before update

Re: [DISCUSSION] Improved USER/SYSTEM exception wrapping in Flink code base

was a related discussion [1], >> FYI. >> >> I’m in favor of the heuristic approach to classify the exceptions by which >> classloader it comes from. In addition, we could introduce extra >> configurations >> to allow manual execution classification based on the

Re: Questions on checkpointing mechanism for FLIP-27 Source API

utor of CheckpointCoordinator. It triggers the checkpoint in SourceCoordinator (which is passed to SplitEnumerator) and then in SourceOperator. The checkpoint event is put in SplitEnumerator's event loop to be executed. You can see the details here. Yours Hongshun On Wed, May 17, 2023 at 11:39 PM Te

Re: Questions on checkpointing mechanism for FLIP-27 Source API

utor of CheckpointCoordinator. It triggers the checkpoint in SourceCoordinator (which is passed to SplitEnumerator) and then in SourceOperator. The checkpoint event is put in SplitEnumerator's event loop to be executed. You can see the details here. Yours Hongshun On Wed, May 17, 2023 at 11:39 PM Te

Re: Questions on checkpointing mechanism for FLIP-27 Source API

utor of CheckpointCoordinator. It triggers the checkpoint in SourceCoordinator (which is passed to SplitEnumerator) and then in SourceOperator. The checkpoint event is put in SplitEnumerator's event loop to be executed. You can see the details here. Yours Hongshun On Wed, May 17, 2023 at 11:39 PM Te

[DISCUSSION] Improved USER/SYSTEM exception wrapping in Flink code base

Hi all, This discussion thread is to gauge community opinion and gather feedback on implementing a better exception hierarchy in Flink to identify exceptions that come from running “User job code” and exceptions coming from “Flink engine code”. Problem: Flink provides a distributed processing

Questions on checkpointing mechanism for FLIP-27 Source API

Hi all, I’m writing a new source based on the FLIP-27 Source API, and I had some questions on the checkpointing mechanisms and associated guarantees. Would appreciate if someone more familiar with the API would be able to provide insights here! In FLIP-27 Source, we now have a SplitEnumerator

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

Thanks Karthi for creating the FLIP! Re: Martijn, As I understand it, the motivation for the Prometheus Sink is for users who want to write metrics to a Prometheus remote_write endpoint as an output of their job graph, so it would be good to treat it as a first-class citizen and do it as part

Re: [VOTE] Release flink-connector-jdbc v3.1.0, release candidate #1

Thanks Danny +1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven Cheers, Hong > On 14 Apr 2023, at 05:21, Elphas Toringepi wrote:

Re: [VOTE] Release flink-connector-mongodb v1.0.1, release candidate #1

Thanks Danny +1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven Cheers, Hong > On 14 Apr 2023, at 06:36, Elphas Toringepi wrote:

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

Hi Panagiotis, Thank you for the update. Looks great! Just one suggestion below: 1. We seem to be waiting for the future(s) to complete before restarting the job - should we add a configurable timeout for the enrichment? Since each failure enricher are run in parallel, we could probably settle

Re: [VOTE] Release flink-connector-mongodb, release candidate #1

+1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven Cheers, Hong > On 28 Mar 2023, at 12:56, Samrat Deb wrote: > > CAUTION: This e

Re: [VOTE] Release flink-connector-aws 4.1.0, release candidate #1

+1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven Cheers, Hong > On 22 Mar 2023, at 07:21, Robert Metzger wrote: > > CAUTION: Thi

Re: [VOTE] Release 1.15.4, release candidate #1

Thanks Danny for driving this +1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Source archive builds using maven * Started packaged example WordCountSQLExample job * Web PR looks good. Cheers, Hong > On 24 Feb 2023, at 05:36, Weihua Hu wrote: >

[RESULT][VOTE] Externalize AWS formats to flink-connector-aws

Hi all, I’m happy to announce we have unanimously approved this proposal, with 4 binding votes and 5 non-binding votes. +1 votes: * Danny (binding) * Jing (non-binding) * Ahmed (non-binding) * Yuxia (non-binding) * Martijn (binding) * Matthias (binding) * Samrat (non-binding) * Daren (non-bindin

Re: [VOTE] Externalize AWS formats to flink-connector-aws

" Cc: "Teoh, Hong" , Chesnay Schepler Subject: RE: [EXTERNAL][VOTE] Externalize AWS formats to flink-connector-aws CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Re: [VOTE] Externalize AWS formats to flink-connector-aws

Got it. Thanks for flagging it up. I have created a JIRA for the followup to make flink-avro Public API more clear. https://issues.apache.org/jira/browse/FLINK-30853 Will close this vote thread and announce in a separate thread. Regards, Hong

Re: [VOTE] Externalize AWS formats to flink-connector-aws

Hi @Dong Lin @Chesnay Schepler, We have deviated from the original purpose of this thread. Let's keep this thread to the original proposal of externalising flink-avro-glue-schema-registry and flink-json-glue-schema-registry formats to the flink-connector-aws repository. The way I see it, @Che

Re: [VOTE] Externalize AWS formats to flink-connector-aws

Hi @Chesnay @Dong Lin, Thanks for flagging up the concerns. I agree that we should make clear which APIs are public in flink-avro so that we can make the connector code more independent. IMO that moves in the right direction for Flink in general! 1. Public classes/interfaces in flink-avro -

Re: [VOTE] Release 1.16.1, release candidate #1

+1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Source archive builds using maven * Started packaged example WordCountSQLExample job. Cheers, Hong On 27/01/2023, 07:52, "Gabor Somogyi" wrote: CAUTION: This email originated from outside of the

[VOTE] Externalize AWS formats to flink-connector-aws

Hi all, As discussed in the discussion thread [1], I would like to propose we externalize flink-avro-glue-schema-registry and flink-json-glue-schema-registry formats to the flink-connector-aws repository [2]. Motivation: 1. We can unify and upgrade the AWS SDK versions more easily. 2. We can no

Re: [DISCUSS] Externalize AWS formats to flink-connector-aws

; > Jing > > > > On Tue, Jan 24, 2023 at 9:44 AM Ahmed Hamdy > wrote: > > > > > Hi Hong > > > +1 for the proposal. Additionally I believe it serves the same value > for > > > the schema classes as it does fo

[DISCUSS] Externalize AWS formats to flink-connector-aws

Hi all, As part of the effort to externalize connectors, I would like to propose that we also externalize flink-avro-glue-schema-registry and flink-json-glue-schema-registry formats to the flink-connector-aws repository. Benefits of this being: 1. We can unify and upgrade the AWS SDK versio

Re: [VOTE] Release flink-connector-aws v4.0.0, release candidate #1

+1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven * Started packaged example SQL job using SQL client. Verified that the following co

Re: [VOTE] Release flink-connector-dynamodb v3.0.0, release candidate #0

+1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven * Started packaged example SQL job using SQL client. Verified that it writes succes

Re: [VOTE] Dedicated AWS externalized connector repo

+1 (non-binding) Thanks for driving this, Danny! Hong On 26/10/2022, 08:14, "Martijn Visser" wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. +1 binding

Re: [DISCUSS] Create a dedicated aws-base connector repository

I like the single repo with single version idea. Pros: - Better discoverability for connectors for AWS services means a better experience for Flink users - Natural placement of AWS-related utils (Credentials, SDK Retry strategy) Caveats: - As you mentioned, it is not desirable if we have to evol

Re: [VOTE] Release 1.16.0, release candidate #2

+1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven * Deployed standalone session cluster and ran TopSpeedWindowing example in streamin

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

Congratulations Danny! Hong On 10/10/2022, 08:06, "Robert Metzger" wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi everyone, I'm very happy and e

Re: [VOTE] Release 1.16.0, release candidate #1

+1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Tag is present in Github * Verified source archive does not contain any binary files * Source archive builds using maven * Deployed standalone session cluster and ran TopSpeedWindowing example in streami

Re: [VOTE] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 (non-binding) Thanks for driving this! Regards, Hong On 11/09/2022, 19:23, "Gyula Fóra" wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. +1 (binding)

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

sense to mention that this FLIP is not about the prioritization of config options. so this feature maybe do this filp and after discuss , not now After the addition is complete, if there are no new comments, we will enter the voting stage and start it ~ what do you think? Teoh, Hong 于2022年8月27日周六

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

e relevant ideas in the future, please continue to discuss. If not, we will open voting at a later date. Teoh, Hong 于2022年8月22日周一 16:02写道： > > Hi Zheng Yu, > > We would have to take the same "cluster configuration" (cannot be set on job submission) vs

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

e inverted override flag should be a cluster setting > and not something the user can override at will. I fear that this might > defeat the purpose of the feature itself. > So I think we should add a config for this but not expose it on the rest api > &g

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

I feel that the inverted override flag should be a cluster setting and not something the user can override at will. I fear that this might defeat the purpose of the feature itself. So I think we should add a config for this but not expose it on the rest api Gyula On Thu, Aug 18, 2022 a

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

+1 to this FLIP. This is very useful for teams building a Flink platform to run jobs from an external user +1 on Danny's comment on adding a configuration to allow inverted order of overrides. However, might it be better to include an "override" toggle in the REST API itself? That way we can

Re: [VOTE] FLIP-243: Dedicated Opensearch connectors

Hi all, I'd also like to add my non-binding +1 for this FLIP. I think it's a great idea to make onboarding easier for new data analysts/engineers, and IMO adding more dedicated connectors is a step in the right direction. Regards, Hong On 27/07/2022, 19:15, "Ryan Skraba" wrote: CAUTI

[RESULT][VOTE] FLIP-242: Introduce configurable RateLimitingStrategy for Async Sink

Hi Flink community! FLIP-242 [1] has been accepted. There are 3 binding votes [2] * Piotrek Nowojski (binding) * Martijn Visser (binding) * Danny Cranmer (binding) None against. Thanks everyone for the support! [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-242%3A+In

[VOTE] FLIP-242: Introduce configurable RateLimitingStrategy for Async Sink

Hi everyone, Thanks for all the feedback so far. Based on the discussion [1], we seem to have consensus. So, I would like to start a vote on FLIP-242 [2]. The vote will last for at least 72 hours unless there is an objection or insufficient votes. [1] https://lists.apache.org/thread/k5s970xlqo

Re: [DISCUSS] FLIP-242: Introduce configurable CongestionControlStrategy for Async Sink

ible for us. All in all +1 for your current > proposal. Ok, let’s keep it separate for now since the complexity is encapsulated in the AsyncSinkWriter. Regards, Hong From: Piotr Nowojski Date: Monday, 27 June 2022 at 09:27 To: "Teoh, Hong" Cc: "dev@flink.apache.org

Re: [DISCUSS] FLIP-242: Introduce configurable CongestionControlStrategy for Async Sink

equests currently in flight. `maxBufferedRequests` on the other hand measures how many messages are in the AsyncSinkWriter’s buffer (or queue). We flush when the number of buffered messages > ` maxBufferedRequests`. Regards, Hong From: Piotr Nowojski Date: Monday, 27 June 2022 at 08:22

Re: [DISCUSS] FLIP-242: Introduce configurable CongestionControlStrategy for Async Sink

;t this variable misnamed? It suggests checking the max number of requests (one async request = one batch?), but from the code it looks like it's actually `maxBufferedMessages`? Best, Piotrek pt., 24 cze 2022 o 09:34 Teoh, Hong napisał(a): > Hi Piotr, > > T

Re: [DISCUSS] FLIP-242: Introduce configurable CongestionControlStrategy for Async Sink

ou proposed in "Example configuring the Congestion Control Strategy using the new interfaces", Or am I missing something? Best Piotrek pon., 20 cze 2022 o 09:52 Teoh, Hong napisał(a): > Hi all, > > I would like to open a discussion on FLIP-2

Re: Permission to create a FLIP wiki page

ng forward to your FLIP. Best regards, Jing On Sat, Jun 18, 2022 at 4:51 PM Teoh, Hong wrote: > Thank you! __ > > Hong > > On 18/06/2022, 15:11, "Jing Ge" wrote: > > CAUTION: This email originated from outside of t

[DISCUSS] FLIP-242: Introduce configurable CongestionControlStrategy for Async Sink

Hi all, I would like to open a discussion on FLIP-242: Introduce configurable CongestionControlStrategy for Async Sink. The Async Sink was introduced as part of FLIP-171 [1], and implements a non-configurable congestion control strategy to reduce network congestion when the destination rejects

Re: Permission to create a FLIP wiki page

eek during the working days. I will let you know once it has been completed. Best regards, Jing On Sat, Jun 18, 2022 at 3:15 PM Teoh, Hong wrote: > Hi all, > > I’m looking to start discussion for a Flink improvement proposal, but I > don’t seem

Re: Flink Kinesis Connector GetRecords Frequency

Hi Aqib, To add to Ahmed's response: * The specific GetRecords Configurations you can control for the KinesisConsumer can be seen here