Hi Fabian, Can you maybe share more about the setup and how you use the AsyncFunction > with > the Kafka client?
Oops, I didn't mention in the reply to David that the kafka producer has nothing to do with the AsyncFunction! I interact with Redis and a Spring boot app. in the AsyncFunction, not Kafka. My pipeline looks like "KafkaSource -> AsyncFunction -> Window -> (KafkaSink1, KafkaSink2)" and the kafka producer is the last one. As David already pointed out it could be indeed a Kafka bug but it could > also > mean that your defined async function leaks direct memory by not freeing > some > resources. The problem seems to be the KafkaSink2 which I recently added to the pipeline in order to write large records (~10MB) for a debugging purpose. I just launched the pipeline without KafkaSink2 to see whether or not my conjecture is right or wrong. If so, I'd rather give more direct memory to each task manager to avoid this problem. We can definitely improve the metrics for the AsyncFunction and expose the > current queue size as a followup. Really looking forward to it. I spent many hours debugging AsyncFunction without metrics. It would be great to have how many records are timed out as a metric as well. Thanks, Dongwon On Tue, Nov 9, 2021 at 7:55 PM Fabian Paul <fabianp...@ververica.com> wrote: > Hi Dongwan, > > Can you maybe share more about the setup and how you use the AsyncFunction > with > the Kafka client? > > As David already pointed out it could be indeed a Kafka bug but it could > also > mean that your defined async function leaks direct memory by not freeing > some > resources. > > We can definitely improve the metrics for the AsyncFunction and expose the > current queue size as a followup. > > Best, > Fabian
