Re: How to Fan Out to 100s of Sinks

2021-11-30 Thread Fabian Paul
> Sent: Monday, November 29, 2021 1:57 AM > To: SHREEKANT ANKALA > Cc: user@flink.apache.org > Subject: Re: How to Fan Out to 100s of Sinks > > Hi, > > What do you mean by "fan out" to 100 different sinks? Do you want to > replicate the data in all buckets or is t

Re: How to Fan Out to 100s of Sinks

2021-11-29 Thread SHREEKANT ANKALA
: Fabian Paul Sent: Monday, November 29, 2021 1:57 AM To: SHREEKANT ANKALA Cc: user@flink.apache.org Subject: Re: How to Fan Out to 100s of Sinks Hi, What do you mean by "fan out" to 100 different sinks? Do you want to replicate the data in all buckets or is there some conditional branc

Re: How to Fan Out to 100s of Sinks

2021-11-29 Thread Fabian Paul
Hi, What do you mean by "fan out" to 100 different sinks? Do you want to replicate the data in all buckets or is there some conditional branching logic? In general, Flink can easily support 100 different sinks but I am not sure if this is the right approach for your use case. Can you clarify your

How to Fan Out to 100s of Sinks

2021-11-28 Thread SHREEKANT ANKALA
Hi all, we current have a Flink job that retrieves jsonl data from GCS and writes to Iceberg Tables. We are using Flink 13.2 and things are working fine. We now have to fan out that same data in to 100 different sinks - Iceberg Tables on s3. There will be 100 buckets and the data needs to be sen