I'm going to follow the guidelines you mentioned. On the other hand, do you
know if there is any real performance comparison between Flink SQL and
Flink? By using 100% Flink SQL, I think we gain a lot of ease of use, but
we might be losing performance. Additionally, there are features available
in
Hi,
There can be various reasons for performance issues, such as:
1. Bottlenecks in the Kafka connector when reading data
2. Time-consuming filter conditions
3. Bottlenecks in the Kafka connector when writing data
To further diagnose the problem, you can try:
1. Set the Flink configuration `
I have been configuring the Flink cluster (application mode) to process the
Kafka data volume. The current configuration consists of 16 pods, each
with *12GB
of memory and 2 CPUs*. Each TaskManager has *4 slots*.
All processing is done using *Flink SQL*, and since Kafka topics may
contain out-of-o
> The average output rate needed to avoid lag after filtering messages
should be around 60K messages per second. I’ve been testing different
configurations of parallelism, slots and pods (everything runs on
Kubernetes), but I’m far from achieving those numbers.
How are you partitioning your query?
After last checking it uses about 200-400 millicores each pod and 2.2Gb.
El mié, 29 ene 2025 a las 21:41, Guillermo Ortiz Fernández (<
guillermo.ortiz.f...@gmail.com>) escribió:
> I have a job entirely written in Flink SQL. The first part of the program
> processes 10 input topics and generates o
I have a job entirely written in Flink SQL. The first part of the program
processes 10 input topics and generates one output topic with normalized
messages and some filtering applied (really easy, some where by fields and
substring). Nine of the topics produce between hundreds and thousands of
mess
