Re: Kafka connector doesn't support consuming update and delete changes in Table SQL API

Unfortunately using row_ts doesn't help. Setting the kafka topic cleanup.policy to compact is not a very good idea as it increases cpu, memory and might lead to other problems. So for now I'll just ignore the null records. Is there anyone who is successfully deduplicating CDC records into either ka

Re: Re: Connecting Pyflink DDL to Azure Eventhubs using Kafka connector

My JAR files included in the same folder i run the python code: flink-connector-kafka_2.11-1.13-SNAPSHOT.JAR flink-sql-connector-kafka_2.11-1.13-SNAPSHOT.JAR kafka-clients-2.7.0.JAR -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Any plans to make Flink configurable with pure data?

Hi Pilgrim, Currently table indeed could not using low level api like timer, would a mixture of sql & datastream could satisfy the requirements? A job might be created via multiple sqls, and connected via datastream operations. Best, Yun ---

Re: Re: Re: Connecting Pyflink DDL to Azure Eventhubs using Kafka connector

Hi, Could you have a try to add the jar via python configuration explicitly? It might refer to [1]. Best, Yun [1]https://ci.apache.org/projects/flink/flink-docs-master/dev/python/table-api-users-guide/dependency_management.html#java-dependency-in-python-program -

State Access Beyond RichCoFlatMapFunction

Hello, I am creating a class that extends RichCoFlatMapFunction. I need to connect() two streams to basically share the state of one stream in another. This is what I do: private transient MapState state; @Override public void open(Configuration parameters) throws Exception { MapStateDescri

Re: Native Kubernetes annotation parsing problem

> > > is there a way I can load my own ConfigMap and specify it via the > dynamic argument? Flink client will automatically ship the "flink-conf.yaml" and "log4j-console.properties" under FLINK_HOME/conf directory on the client side. After the application is submitted successfully, you could find

Custom source with multiple, differently typed outputs

Hi everyone, I want to connect to a proprietary data stream, which sends different types of messages (maybe interpreted as a table), intertwined in the stream. Every type of message (or table) can have a different schema, but for each type this schema is known when connecting (i.e., at runtime) an

Re: Re: Re: Connecting Pyflink DDL to Azure Eventhubs using Kafka connector

hi Yun, thanks for the help! if i direct the Kafka connector in the DDL to a local Kafka cluster, it works fine. So i assume access to the JAR files should not be the issue. This is how i referred to the JAR files from Python: t_env.get_config().get_configuration().set_string("pipeline.jars", "

What is the difference between RuntimeContext state and ProcessWindowFunction.Context state when using a ProcessWindowFunction?

Hi, I am having a difficult time distinguishing the difference between RuntimeContext state and global state when using a ProcessWindowFunction. A ProcessWindowFunction has three access different kinds of state. 1. RuntimeContext state. 2. ProcessWindowFunction.Context global state 3. ProcessWind

Join two streams from Kafka

Hello Flink Users, I am newbie and have question on join of two streams (stream1 and stream2 ) from Kafka topic based on some key. In my use case I need to join with stream2 data which might be year old and more. Now if on stream1 the data gets arrived today and I need to join with stream2 base

Re: "upsert-kafka" connector not working with Avro confluent schema registry

Hello Flink Users, Request you to please help. I am facing issue with "KafkaAvroDeserializer" by using "upsert-kafka" connector. Regards, Shamit Jain -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

ClassLoader leak when using s3a upload through DataSet.output

Hello folks, We see threads from https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/internal/TransferManagerUtils.java#L49 outlive a batch job that writes Parquet Files to S3, causing a ClassLoader Leak. Is this a known

Re: State Access Beyond RichCoFlatMapFunction

(a) It is by design. For keyed state, you can only access state for that key, not others. If you want one value per key, ValueState fits more appropriate that MapState. (b) state-processor-api aims to access/create/modify/upgrade offline savepoint but not running state. Queryable state may meet you

Re: State Access Beyond RichCoFlatMapFunction

Hello, Thanks a lot for the response. I will try to check Queryable-state for this purpose. Actually, my use case is that I want to share the state of one stream in two other streams. Right now, I can think of connecting this stream independently with each of the two other streams and manage

Re: State Access Beyond RichCoFlatMapFunction

Flink has broadcast state to broadcast one stream to other in case you are not aware of it. It actually duplicates state. 1. Broadcast state: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html Best, Kezhu Wang On February 10, 2021 at 13:03:36, Sandeep kh