Avoiding data shuffling when reading pre-partitioned data from Kafka

Hello, My team has a Flink streaming job that does a stateful join across two high throughput kafka topics. This results in a large amount of data ser/de and shuffling (about 1gb/s for context). We're running into a bottleneck on this shuffling step. We've attempted to optimize our flink configura

Question regarding Flink-CDC for postgres

Hi Flink Users, We are looking to use Flink Postgres CDC but then we noticed it only supports single thread reading during the snapshot reading phase. Is there any plan to support parallel snapshot reading for postgres in the near future? Looks like this is not an issue for mysql. Thanks Leon

Re: [EXTERNAL] Re: Secure Azure Credential Configuration

Thanks Alexis, I will be trying that out today. If it works I will share back and try adding it to the docs. From: Alexis Sarda-Espinosa Sent: Thursday, March 2, 2023 3:33:03 PM To: Ivan Webber Cc: user Subject: Re: [EXTERNAL] Re: Secure Azure Credential Conf

Can I dedup over an upsert topic?

Hi Flink User group, I bumped into the following error when using FlinkSQL, version 1.15.3 * StreamPhysicalDeduplicate doesn't support consuming update and delete changes which is produced by node ChangelogNormalize Essentially I was trying to dedup over an upsert kafka topic. I wonder if

aws zeppelin glue table connector

Hi- Hoping someone can give me more understanding and a sense of timing flink leveraging glue tables. Specifically I would like to leverage a glue table data source via an AWS hosted Zeppelin notebook (instead of referencing a kinesis data stream or s3 bucket directly). Additionally I will ne

Re: debezium/flink-oracle-cdc-connector large memory consumption for transactions

Hello,Try Infinispan - https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-event-buffering15:12, 3 марта 2023 г., Alexandr Dmitrov :Hello! Maybe you could help with debezium/vervica connectors in optimizing memory consumption for large changes in a single transaction us

debezium/flink-oracle-cdc-connector large memory consumption for transactions

Hello! Maybe you could help with debezium/vervica connectors in optimizing memory consumption for large changes in a single transaction using vervica flink-oracle-cdc-connector.Environment:flink - 1.15.1oracle-cdc - 2.3oracle - 19.3When updating large number of rows in a single transaction I get th

Re: Customisable Triggers on Table API (Java)

Gentle Reminder on the query. On Wed, Mar 1, 2023 at 8:32 PM Kartikey Pant wrote: > Hi, > > We are using Table API with Flink (Java). Our use case requires us to use > Multiple Aggregate Functions most of which the Table API already supports. > However, when we went ahead with the Table APIs, we

unsubscribe