Flink connector sinks

Hello everyone, I have a question more on the curiosity side, unsure if this has been discussed before (sorry if It has). Given the recent deprecation of the `FlinkKafkaProducer` in favour of the new `KafkaSink` where we get away from the `SinkFunction` interface in favour of the “new” `Sink`

Re: Flink connector sinks

Hi Carlos, Yes, the new Sink interface is ultimately the target for all Sink connectors. A discussion on that topic has started recently, which you can see at https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx. Currently Kafka, Elasticsearch, Opensearch, Kinesis, Firehose, DynamoDB a

[jira] [Created] (FLINK-30768) flink-web version cleanup

Matthias Pohl created FLINK-30768: - Summary: flink-web version cleanup Key: FLINK-30768 URL: https://issues.apache.org/jira/browse/FLINK-30768 Project: Flink Issue Type: Improvement

Re: Flink connector sinks

Hi, I would recommend consolidating all those discussions like using Sink over SinkFunction, SinkFunktion deprecation, etc. in [1] to keep everyone on the same page. Martijn has already shared a very good big picture with more details in that thread. Let's continue our discussion there. Thanks! B

Re: Flink connector sinks

Hello Martijn, Thanks a lot for the answer. Other question I have if you could help me. Is there a proper way to define sequential sinks? Meaning if we could have a sequence of sinks that operate in order instead of operating in parallel Best regards, Carlos Freitas > On 23 Jan 2023, at 12:52,

Re: [VOTE] Release 1.16.1, release candidate #1

Thanks Martijn for pushing this one. +1 (non-binding) * Downloaded artifacts & built Flink from sources * Verified SHA512 checksums & GPG signatures * Compared checkout with provided sources * Verified pom file versions * Went over NOTICE file/pom files changes without finding anything suspicious

[jira] [Created] (FLINK-30769) Use Operator SDK RuntimeInfo in Health Probe

Gyula Fora created FLINK-30769: -- Summary: Use Operator SDK RuntimeInfo in Health Probe Key: FLINK-30769 URL: https://issues.apache.org/jira/browse/FLINK-30769 Project: Flink Issue Type: New Feat

[jira] [Created] (FLINK-30770) OUTER join generates partial results

Thomas Erlenbach created FLINK-30770: Summary: OUTER join generates partial results Key: FLINK-30770 URL: https://issues.apache.org/jira/browse/FLINK-30770 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-30771) different performance of conversion between datetime type in mysql and timestamp in flink

jingjing dong created FLINK-30771: - Summary: different performance of conversion between datetime type in mysql and timestamp in flink Key: FLINK-30771 URL: https://issues.apache.org/jira/browse/FLINK-30771

Re: [DISCUSS] FLIP-285: Refactoring the leader election code base in Flink

Thanks for updating the design. From my side this looks good. On 18/01/2023 17:59, Matthias Pohl wrote: After another round of discussion, I came up with a (hopefully) final proposal. The previously discussed approach was still not optimal because the contender ID lived in the LeaderContender ev

[jira] [Created] (FLINK-30772) Update Flink Shaded dependencies to the latest versions

Martijn Visser created FLINK-30772: -- Summary: Update Flink Shaded dependencies to the latest versions Key: FLINK-30772 URL: https://issues.apache.org/jira/browse/FLINK-30772 Project: Flink I

[jira] [Created] (FLINK-30773) Allow rescaling of jobs based on per-vertex parallelism overrides

Maximilian Michels created FLINK-30773: -- Summary: Allow rescaling of jobs based on per-vertex parallelism overrides Key: FLINK-30773 URL: https://issues.apache.org/jira/browse/FLINK-30773 Project

[jira] [Created] (FLINK-30774) flink-utils module

Matthias Pohl created FLINK-30774: - Summary: flink-utils module Key: FLINK-30774 URL: https://issues.apache.org/jira/browse/FLINK-30774 Project: Flink Issue Type: Improvement Compon

[jira] [Created] (FLINK-30775) Support pluggable reconciler modules

Maximilian Michels created FLINK-30775: -- Summary: Support pluggable reconciler modules Key: FLINK-30775 URL: https://issues.apache.org/jira/browse/FLINK-30775 Project: Flink Issue Type:

[jira] [Created] (FLINK-30776) Move autoscaler code into a seperate module

Maximilian Michels created FLINK-30776: -- Summary: Move autoscaler code into a seperate module Key: FLINK-30776 URL: https://issues.apache.org/jira/browse/FLINK-30776 Project: Flink Issue

Reworking the Rescale API

Hi, The current rescale API appears to be a work in progress. A couple years ago, we disabled access to the API [1]. I'm looking into this problem as part of working on autoscaling [2] where we currently require a full restart of the job to apply the parallelism overrides. This adds additional de

[jira] [Created] (FLINK-30777) Allow Kinesis Table API Connector to specify shard assigner

Karthi Thyagarajan created FLINK-30777: -- Summary: Allow Kinesis Table API Connector to specify shard assigner Key: FLINK-30777 URL: https://issues.apache.org/jira/browse/FLINK-30777 Project: Flin

Re: Reworking the Rescale API

Hello max Thanks for driving it, I think there is no problem with your previous suggestion of [1] FLINK-30773. Here I just put forward some supplements and doubts.I have some suggestions and insights for this I have experienced the autoscaling of Flink K8S Operator for a part of the time. The cu

[DISCUSS] Externalize AWS formats to flink-connector-aws

Hi all, As part of the effort to externalize connectors, I would like to propose that we also externalize flink-avro-glue-schema-registry and flink-json-glue-schema-registry formats to the flink-connector-aws repository. Benefits of this being: 1. We can unify and upgrade the AWS SDK versio