Re: [DISCUSS] FLIP-307: Flink connector Redshift

2023-05-29 Thread Yaroslav Tkachenko
Hi Samrat, Regarding the sink, will it only support append-only tables (no changelog mode support)? It looks like it's the case, so IMO, it should be mentioned in the limitations section. On Sun, May 28, 2023 at 9:52 PM Samrat Deb wrote: > Hello all , > > Context: > Amazon Redshift [1] is a ful

Re: [VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax

2022-12-01 Thread Yaroslav Tkachenko
+1 (non-binding). Looking forward to it! On Thu, Dec 1, 2022 at 5:06 AM Dong Lin wrote: > +1 (binding) > > Thanks for the FLIP! > > On Thu, Dec 1, 2022 at 12:20 PM Shengkai Fang wrote: > > > Hi All, > > > > Thanks for all the feedback so far. Based on the discussion[1] we seem > > to have a co

Re: Flink Operator - Supporting Recovery from Snapshot

2023-02-10 Thread Yaroslav Tkachenko
Hi Kevin! In my case, I automated this workflow by first deleting the current Flink deployment and then creating a new one. So, if the initialSavepointPath is different it'll use it for recovery. This approach is indeed irreversible, but so far it's been working well. On Fri, Feb 10, 2023 at 8:1

Re: Impact of redacting UPDATE_BEFORE fields?

2024-02-07 Thread Yaroslav Tkachenko
Hey Kevin, In my experience it mostly depends on the type of your sinks. If all of your sinks can leverage primary keys and support upsert semantics, you don't really need UPDATE_BEFOREs altogether (you can even filter them out). But if you have sinks with append-only semantics (OR if you don't ha

Discussion: Kafka Consumer and client.id.prefix usage

2022-02-16 Thread Yaroslav Tkachenko
Hey everyone! I'm currently investigating the way Flink configures client.id values for the Kafka consumer and I'd appreciate some help. *Old FlinkKafkaConsumer* It doesn't look like client.id receives any special treatment when using deprecated FlinkKafkaConsumer: if client.id is not specified,

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Yaroslav Tkachenko
Hi Martijn, As a Scala user, this change would affect me a lot and I'm not looking forward to rewriting my codebase, and it's not even a very large one :) I'd like to suggest supporting Java 17 as a prerequisite ( https://issues.apache.org/jira/browse/FLINK-15736). Things like switch expressions

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Yaroslav Tkachenko
ility guarantees already (depending if it's > Public/PublicEvolving/Experimental). If a change would happen there, I > think it would be smaller refactoring. > > Best regards, > > Martijn > > [1] https://issues.apache.org/jira/browse/FLINK-25000 > > On Tue, Oct 4, 2

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Yaroslav Tkachenko
to deprecate and > remove the Scala APIs? > > Best regards, > > Martijn > > > On Tue, Oct 4, 2022 at 2:41 PM Yaroslav Tkachenko > wrote: > >> Hi Martijn, >> >> The 2.0 argument makes sense (I agree it's easier to introduce more >> breaking ch

[DISCUSS] Adding client.id.prefix to the KafkaSink

2022-10-12 Thread Yaroslav Tkachenko
Hi everyone, I'd like to propose adding client.id.prefix to the KafkaSink to mirror the functionality provided by the KafkaSource. Defining client.id is very important when running workloads with many different Kafka clients: they help with identification and enforcing quotas. Due to the specific

Re: Utilizing Kafka headers in Flink Kafka connector

2022-10-12 Thread Yaroslav Tkachenko
Hi, You can implement a custom KafkaRecordDeserializationSchema (example https://docs.immerok.cloud/docs/cookbook/reading-apache-kafka-headers-with-apache-flink/#the-custom-deserializer) and just avoid emitting the record if the header value matches what you need. On Wed, Oct 12, 2022 at 11:04 AM

Re: Can we make flink operator using golang?

2022-11-17 Thread Yaroslav Tkachenko
Hey Mark, Apache Flink doesn't support Golang, but you can look at Beam's Golang SDK: https://beam.apache.org/documentation/sdks/go/. Beam jobs can use Apache Flink as a runner: https://beam.apache.org/documentation/runners/flink/ On Wed, Nov 16, 2022 at 8:47 PM Mark Lee wrote: > Hi, > > I fo

[jira] [Created] (FLINK-32814) Support disabling failed JM shutdown in the operator

2023-08-08 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-32814: -- Summary: Support disabling failed JM shutdown in the operator Key: FLINK-32814 URL: https://issues.apache.org/jira/browse/FLINK-32814 Project: Flink

[jira] [Created] (FLINK-33622) table.optimizer.reuse-sub-plan-enabled doesn't work when a Table is converted to a DataStream

2023-11-22 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-33622: -- Summary: table.optimizer.reuse-sub-plan-enabled doesn't work when a Table is converted to a DataStream Key: FLINK-33622 URL: https://issues.apache.org/jira/b

[jira] [Created] (FLINK-30873) FileSystem Table API connector doesn't discover new files when partitions are specified

2023-02-01 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-30873: -- Summary: FileSystem Table API connector doesn't discover new files when partitions are specified Key: FLINK-30873 URL: https://issues.apache.org/jira/browse/

[jira] [Created] (FLINK-31285) FileSource should support reading files in order

2023-03-01 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-31285: -- Summary: FileSource should support reading files in order Key: FLINK-31285 URL: https://issues.apache.org/jira/browse/FLINK-31285 Project: Flink

[jira] [Created] (FLINK-27873) Primary key is lost after calling addColumns

2022-06-01 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-27873: -- Summary: Primary key is lost after calling addColumns Key: FLINK-27873 URL: https://issues.apache.org/jira/browse/FLINK-27873 Project: Flink

[jira] [Created] (FLINK-25164) DogStatsD Metrics Reporter

2021-12-03 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-25164: -- Summary: DogStatsD Metrics Reporter Key: FLINK-25164 URL: https://issues.apache.org/jira/browse/FLINK-25164 Project: Flink Issue Type: New

[jira] [Created] (FLINK-23127) Can't use plugins for GCS filesystem

2021-06-23 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-23127: -- Summary: Can't use plugins for GCS filesystem Key: FLINK-23127 URL: https://issues.apache.org/jira/browse/FLINK-23127 Project: Flink Issue

[jira] [Created] (FLINK-28842) Add client.id.prefix for the KafkaSink

2022-08-05 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-28842: -- Summary: Add client.id.prefix for the KafkaSink Key: FLINK-28842 URL: https://issues.apache.org/jira/browse/FLINK-28842 Project: Flink Issue

[jira] [Created] (FLINK-29288) Can't start a job with a jar in the system classpath

2022-09-13 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-29288: -- Summary: Can't start a job with a jar in the system classpath Key: FLINK-29288 URL: https://issues.apache.org/jira/browse/FLINK-29288 Project:

[jira] [Created] (FLINK-29633) Operator doesn't pass initialSavepointPath as fromSavepoint argument

2022-10-13 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-29633: -- Summary: Operator doesn't pass initialSavepointPath as fromSavepoint argument Key: FLINK-29633 URL: https://issues.apache.org/jira/browse/FLINK-

[jira] [Created] (FLINK-36673) Operator is not properly handling failed deployments without savepoints

2024-11-07 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created FLINK-36673: -- Summary: Operator is not properly handling failed deployments without savepoints Key: FLINK-36673 URL: https://issues.apache.org/jira/browse/FLINK-36673