Re: Flink local mini cluster is causing memory leak when triggered multiple times

2023-09-01 Thread Bashir Sadjad via user
An update: The extra memory that is being acquired/kept after each pipeline run seems to be coming from off-heap space, especially using `Unsafe.allocateMemory`. I have added some notes here but the TL;DR; is that adding

Flink local mini cluster is causing memory leak when triggered multiple times

2023-08-30 Thread Chandrashekar Sankarapu via user
Hi Team, We have a data pipeline which is built using Apache Beam SDK and we use Apache Flink Runner to execute Beam pipelines. We use the local embedded execution mode of Flink for running the pipelines. Currently,