RE: Huge number of GAX-Threads in Flink

2022-04-21 Thread SHREEKANT ANKALA
Hi, Can somebody please help with this issue? This is really blocking us in Production. If needed, I can provide the thread dump as well. Thanks, Shreekant A. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows From: SHREEKANT

Huge number of GAX-Threads in Flink

2022-04-17 Thread SHREEKANT ANKALA
Hi, we are using Flink 1.13.2 for one of our datalake project to process json files from Google Cloud storage and write to Iceberg tables in S3. After running the application in production for a couple of days, we are noticing that the GAX-Threads in "Flink Task Threadgoup" are growing with out

Re: How to Fan Out to 100s of Sinks

2021-11-29 Thread SHREEKANT ANKALA
: Fabian Paul Sent: Monday, November 29, 2021 1:57 AM To: SHREEKANT ANKALA Cc: user@flink.apache.org Subject: Re: How to Fan Out to 100s of Sinks Hi, What do you mean by "fan out" to 100 different sinks? Do you want to replicate the data in all buckets or is there some conditional branc

How to Fan Out to 100s of Sinks

2021-11-28 Thread SHREEKANT ANKALA
Hi all, we current have a Flink job that retrieves jsonl data from GCS and writes to Iceberg Tables. We are using Flink 13.2 and things are working fine. We now have to fan out that same data in to 100 different sinks - Iceberg Tables on s3. There will be 100 buckets and the data needs to be sen