Help with Memory Leak in DataStream API V2

Hi all! I'm trying to use the new DataStream API V2, but I have noticed that the Java heap usage keeps increasing infinitely. Upon analyzing a heap dump, I found that a HashSet named 'keySet' is consuming 97% of the heap. Here is the relevant field in the source code I traced it to: https://gi

Re: Memory Leak

We also faced the same issue with Flink 1.16.1. Please enable jemalloc as a memory allocator, it fixed the issue for us. On Wed, Sep 6, 2023 at 9:07 PM Kenan Kılıçtepe wrote: > Hi, > Thanks for the answer. > I will try the documents you have shared. > But still it would be great if you can take

Re: Memory Leak

Hi, Thanks for the answer. I will try the documents you have shared. But still it would be great if you can take a look at the numbers below and give some tips. At the moment RSS is 46.6GB although taskmanager.memory.process.size is set to 4m GC Statistics: 2023-09-06 15:15:03,785 INFO org.

Re: Memory Leak

Hi Kenan, If you have confirmed the heap memory is ok(e.g. no Java OOM exception and no frequent GC), then the cause may be off-heap memory over usage, especially when your flink job uses some native library. To diagnose such problem, you can refer to [1][2] for more details about using NMT and jep

Memory Leak

Hi, I have Flink 1.16.2 on a single server with 64GB Ram. Although taskmanager.memory.process.size is set to 4m, I can see memory usage of the task manager exceed 59GB and OS kills it because of OOM. I check the RSS column of application top for memory usage. I don`t see any heap memory p

Re: Flink local mini cluster is causing memory leak when triggered multiple times

An update: The extra memory that is being acquired/kept after each pipeline run seems to be coming from off-heap space, especially using `Unsafe.allocateMemory`. I have added some notes here but the TL;DR; is that adding

Flink local mini cluster is causing memory leak when triggered multiple times

Hi Team, We have a data pipeline which is built using Apache Beam SDK and we use Apache Flink Runner to execute Beam pipelines. We use the local embedded execution mode of Flink for running the pipelines. Currently,

State Memory Leak Issue in Flink CEP v1.17: Discussion & Proposed Solution

Hi, We have encountered a state memory leak issue while working with the Flink CEP (Complex Event Processing) in version 1.17. Based on code, it seems like the issue is persistent across all versions of CEP library. I've already raised a JIRA issue detailing the problem we've face

Re: Possible memory leak in JobManager (Flink 1.10.0)?

>>> too often and would not make much sense in production environment. >>> >>> Best >>> Yun Tang >>> -- >>> *From:* Till Rohrmann >>> *Sent:* Thursday, April 9, 2020 17:41 >>> *To:* Marc LEGER

Re: Possible memory leak in JobManager (Flink 1.10.0)?

gt;> *From:* Till Rohrmann >> *Sent:* Thursday, April 9, 2020 17:41 >> *To:* Marc LEGER >> *Cc:* Yun Tang ; user@flink.apache.org < >> user@flink.apache.org> >> *Subject:* Re: Possible memory leak in JobManager (Flink 1.10.0)? >> >> Thanks for reporting thi

Re: Possible memory leak in JobManager (Flink 1.10.0)?

20 17:41 > *To:* Marc LEGER > *Cc:* Yun Tang ; user@flink.apache.org < > user@flink.apache.org> > *Subject:* Re: Possible memory leak in JobManager (Flink 1.10.0)? > > Thanks for reporting this issue Marc. From what you've reported, I think > Yun is right and that th

Re: Possible memory leak in JobManager (Flink 1.10.0)?

Sent: Thursday, April 9, 2020 17:41 To: Marc LEGER Cc: Yun Tang ; user@flink.apache.org Subject: Re: Possible memory leak in JobManager (Flink 1.10.0)? Thanks for reporting this issue Marc. From what you've reported, I think Yun is right and that the large memory footprint is caused by

Re: Possible memory leak in JobManager (Flink 1.10.0)?

github.com/apache/flink/blob/d7e247209358779b6485062b69965b83043fb59d/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java#L234 >> >> Best >> Yun Tang >> >> -- >> *From:* Marc LEGER >

Re: Possible memory leak in JobManager (Flink 1.10.0)?

untime/checkpoint/ZooKeeperCompletedCheckpointStore.java#L234 > > Best > Yun Tang > > -- > *From:* Marc LEGER > *Sent:* Wednesday, April 8, 2020 16:50 > *To:* user@flink.apache.org > *Subject:* Possible memory leak in JobManager (Flink 1.10.

Re: Possible memory leak in JobManager (Flink 1.10.0)?

, April 8, 2020 16:50 To: user@flink.apache.org Subject: Possible memory leak in JobManager (Flink 1.10.0)? Hello, I am currently testing Flink 1.10.0 but I am facing memory issues with JobManagers deployed in a standalone cluster configured in HA mode with 3 TaskManagers (and 3 running jobs). I do

Re: Memory Leak in ProcessingTimeSessionWindow

Hi, for most windows, all state is cleared through FIRE_AND_PURGE, except for windows that are subtypes of merging windows, such as session windows. Here, the state still remembers the window itself until the watermark passes the session timeout+allowed lateness. This is done so that elements t

Re: Memory Leak in ProcessingTimeSessionWindow

One more attempt to get some feedback on this. It basically boils down to using High-Level Window API in scenarios where keys are unbounded / infinite but can be expired after certain time. From what we have observed (solution 2 below), some properties of keys are still in state (guessing key it

Re: Memory Leak in ProcessingTimeSessionWindow

from Yahoo Mail for iPhone On Monday, June 18, 2018, 3:37 AM, Stefan Richter wrote: Hi, can you take a heap dump from a JVM that runs into the problem and share it with us? That would make finding the cause a lot easier. Best,Stefan Am 15.06.2018 um 23:01 schrieb ashish pok : All, I have ano

Re: Memory Leak in ProcessingTimeSessionWindow

iPhone On Monday, June 18, 2018, 3:37 AM, Stefan Richter wrote: Hi, can you take a heap dump from a JVM that runs into the problem and share it with us? That would make finding the cause a lot easier. Best,Stefan Am 15.06.2018 um 23:01 schrieb ashish pok : All, I have another slow Memory Lea

Re: Memory Leak in ProcessingTimeSessionWindow

AM, Stefan Richter wrote: Hi, can you take a heap dump from a JVM that runs into the problem and share it with us? That would make finding the cause a lot easier. Best,Stefan Am 15.06.2018 um 23:01 schrieb ashish pok : All, I have another slow Memory Leak situation using basic TimeSession Windo

Re: Memory Leak in ProcessingTimeSessionWindow

t > with us? That would make finding the cause a lot easier. > > Best, > Stefan > >> Am 15.06.2018 um 23:01 schrieb ashish pok > <mailto:ashish...@yahoo.com>>: >> >> All, >> >> I have another slow Memory Leak situation using b

Re: Memory Leak in ProcessingTimeSessionWindow

re it with us? That would make finding the cause a lot easier. Best,Stefan Am 15.06.2018 um 23:01 schrieb ashish pok : All, I have another slow Memory Leak situation using basic TimeSession Window (earlier it was GlobalWindow related that Fabian helped clarify).  I have a very simple data pip

Re: Memory Leak in ProcessingTimeSessionWindow

runs into the problem and share it with us? That would make finding the cause a lot easier. Best,Stefan Am 15.06.2018 um 23:01 schrieb ashish pok : All, I have another slow Memory Leak situation using basic TimeSession Window (earlier it was GlobalWindow related that Fabian helped clarify).  I

Re: Memory Leak in ProcessingTimeSessionWindow

Hi, can you take a heap dump from a JVM that runs into the problem and share it with us? That would make finding the cause a lot easier. Best, Stefan > Am 15.06.2018 um 23:01 schrieb ashish pok : > > All, > > I have another slow Memory Leak situation using basic Tim

Memory Leak in ProcessingTimeSessionWindow

All, I have another slow Memory Leak situation using basic TimeSession Window (earlier it was GlobalWindow related that Fabian helped clarify).  I have a very simple data pipeline: DataStream processedData = rawTuples .window(ProcessingTimeSessionWindows.withGap(Time.seconds

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Hi, If the TM is not responding check the TM logs if there is some long gap in logs. There might be three main reasons for such gaps: 1. Machine is swapping - setup/configure your machine/processes that machine never swap (best to disable swap altogether) 2. Long GC full stops - look how to ana

Re: Flink memory leak

Thank you for those screenshots, they help a lot. However I do not see any obvious candidate for a memory leak. There is a slight upward trend in "G1 Old Gen”, but this can be misleading. To further analyse what’s going you need to run your test case for a longer time. Also you will ne

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Sorry, the "killed" I mean here is JM lost the TM. The TM instance is still running inside kubernetes, but it is not responding to any requests, probably due to high load. And from JM side, JM lost heartbeat tracking of the TM, so it marked the TM as died. The „volume“ of Kafka topics, I mean, the

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Hi, > In addition to your comments, what are the items retained by > NetworkEnvironment? They grew seems like indefinitely, do they ever reduce? > Mostly the network buffers, which should be ok. They are always recycled and should not be released until the network environment is GCed. > I think

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Thanks a lot! This is very helpful. In addition to your comments, what are the items retained by NetworkEnvironment? They grew seems like indefinitely, do they ever reduce? I think there is a GC issue because my task manager is killed somehow after a job run. The duration correlates to the volume

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Hi, I cannot spot anything that indicates a leak from your screenshots. Maybe you misinterpret the numbers? In your heap dump, there is only a single instance of org.apache.flink.runtime.io.network.NetworkEnvironment and it retains about 400,000,000 bytes from being GCed because it holds refere

Re: Flink memory leak

2017 at 4:02 PM, Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Ebru, Javier, Flavio: > > I tried to reproduce memory leak by submitting a job, that was generating > classes with random names. And indeed I have found one. Memory was > accumulating

Re: Flink memory leak

What should we do to confirm it? Do you have any github repo start from? On Tue, Nov 14, 2017 at 4:02 PM, Piotr Nowojski wrote: > Ebru, Javier, Flavio: > > I tried to reproduce memory leak by submitting a job, that was generating > classes with random names. And indeed I have found

Re: Flink memory leak

Ebru, Javier, Flavio: I tried to reproduce memory leak by submitting a job, that was generating classes with random names. And indeed I have found one. Memory was accumulating in `char[]` instances that belonged to `java.lang.ClassLoader#parallelLockMap`. OldGen memory pool was growing in size

Re: Flink memory leak

ilters there shouldn’t be any network transfers involved, >>>>>>>>> aside >>>>>>>>>> from Source and Sink functions. >>>>>>>>>> Piotrek >>>>>>>>>> On 8 Nov 2017, at 12:54,

Re: Flink memory leak

use case, you have to manually call `clear()` on the state instance in order to release the managed state. Best, Ufuk On Tue, Nov 7, 2017 at 12:43 PM, ebru wrote: Begin forwarded message: From: ebru Subject: Re: Flink memory leak Date: 7 November 2017 at 14:09:17 GMT+3 To: Ufuk Celebi Hi Ufuk, Ther

Re: Flink memory leak

>>> Kien, but it didn't work. We have a workaround similar to >>>>>> the one >>>>>>> that Flavio has, we restart the taskmanagers once they reach >>>>>> a >>>>>>> memory threshold. We created a small test to

Re: Flink memory leak

;>>> per node. We have one job that uses 56 slots, and we cannot >>>> execute >>>>> that job 5 times in a row because the whole cluster dies. If >>>> you >>>>> want, we can publish our test job. >>>>> Regards, >>>>

Re: Flink memory leak

gt; On 8. Nov 2017, at 10:25, Flavio Pompermaier >> >>> wrote: >>> We also have the same problem in production. At the moment >> the >>> solution is to restart the entire Flink cluster after every >> job.. >>> We've tried to reproduce this problem wi

Re: Flink memory leak

& @Piotr Could you please have a look at this? You >>>> both >>>>>>>>> recently worked on the network stack and might be most >>>> familiar with >>>>>>>>> this. >>>>>>>>> On 8. Nov 2017, at 1

Re: Flink memory leak

Ufuk On Tue, Nov 7, 2017 at 12:43 PM, ebru wrote: Begin forwarded message: From: ebru Subject: Re: Flink memory leak Date: 7 November 2017 at 14:09:17 GMT+3 To: Ufuk Celebi Hi Ufuk, There are there snapshots of htop output. 1. snapshot is initial state. 2. snapshot is after submitted one job.

Re: Flink memory leak

;> >> >>>>> - Ebru >> >>>>> >> >>>>> On 7 Nov 2017, at 16:23, Ufuk Celebi > >>>>> <mailto:u...@apache.org>> wrote: >> >>>>> >> >>>>> Hey Ebru, the memory usage might be incr

Re: Flink memory leak

ak are > >>>>> correlated.. > >>>>> > >>>>> Best, > >>>>> Flavio > >>>>> > >>>>> On Wed, Nov 8, 2017 at 9:51 AM, ÇETİNKAYA EBRU ÇETİNKAYA EBRU > >>>>> mailto:b20926...@cs.hacettepe

Re: Flink memory leak

t;>> On 2017-11-07 16:53, Ufuk Celebi wrote: >>>>> Do you use any windowing? If yes, could you please share that code? >>>>> If >>>>> there is no stateful operation at all, it's strange where the list >>>>> state instances a

Re: Flink memory leak

state instances are coming from. >> >> On Tue, Nov 7, 2017 at 2:35 PM, ebru > <mailto:b20926...@cs.hacettepe.edu.tr>> >> wrote: >> Hi Ufuk, >> >> We don’t explicitly define any state descriptor. We only use map >> and filters >> operato

Re: Flink memory leak

>>>> Do you use any windowing? If yes, could you please share that code? >>>> If >>>> there is no stateful operation at all, it's strange where the list >>>> state instances are coming from. >>>> >>>> On Tue, Nov 7, 201

Re: Flink memory leak

@apache.org>> wrote: > > Hey Ebru, the memory usage might be increasing as long as a job is > running. > This is expected (also in the case of multiple running jobs). The > screenshots are not helpful in that regard. :-( > > What kind of stateful operations are you using

Re: Flink memory leak

’t explicitly define any state descriptor. We only use map >>> and filters >>> operator. We thought that gc handle clearing the flink’s internal >>> states. >>> So how can we manage the memory if it is always increasing? >>> >>> - Ebru >>

Re: Flink memory leak

state. Best, Ufuk On Tue, Nov 7, 2017 at 12:43 PM, ebru wrote: Begin forwarded message: From: ebru Subject: Re: Flink memory leak Date: 7 November 2017 at 14:09:17 GMT+3 To: Ufuk Celebi Hi Ufuk, There are there snapshots of htop output. 1. snapshot is initial state. 2. snapshot is af

Re: Flink memory leak

.@apache.org>> wrote: >>> >>> Hey Ebru, the memory usage might be increasing as long as a job is running. >>> This is expected (also in the case of multiple running jobs). The >>> screenshots are not helpful in that regard. :-( >>> >>> What kind of stateful

Re: Flink memory leak

. >> This is expected (also in the case of multiple running jobs). The >> screenshots are not helpful in that regard. :-( >> >> What kind of stateful operations are you using? Depending on your use case, >> you have to manually call `clear()` on the state instance in orde

Re: Flink memory leak

gt;>> screenshots are not helpful in that regard. :-( >>>> >>>> What kind of stateful operations are you using? Depending on your use >>>> case, >>>> you have to manually call `clear()` on the state instance in order to >>>> release the m

Re: Flink memory leak

ave to manually call `clear()` on the state instance in order to > release the managed state. > > Best, > > Ufuk > > On Tue, Nov 7, 2017 at 12:43 PM, ebru <mailto:b20926...@cs.hacettepe.edu.tr>> wrote: > > > > Begin forwarded message: > > From: ebr

Re: Flink memory leak

ations are you using? Depending on your use >>> case, >>> you have to manually call `clear()` on the state instance in order to >>> release the managed state. >>> >>> Best, >>> >>> Ufuk >>> >>> On Tue, Nov 7, 2017 at 12

Re: Flink memory leak

managed state. >> >> Best, >> >> Ufuk >> >> On Tue, Nov 7, 2017 at 12:43 PM, ebru wrote: >>> >>> >>> >>> Begin forwarded message: >>> >>> From: ebru >>> Subject: Re: Flink memory leak >>

Re: Flink memory leak

nce in order to > release the managed state. > > Best, > > Ufuk > > On Tue, Nov 7, 2017 at 12:43 PM, ebru wrote: >> >> >> >> Begin forwarded message: >> >> From: ebru >> Subject: Re: Flink memory leak >> Date: 7 November 2017 at 1

Re: Flink memory leak

>> managers in standalone mode. Recently, we've noticed that we have memory >>>> related problems. We use docker container to serve Flink cluster. We have >>>> 300 slots and 20 jobs are running with parallelism of 10. Also the job >>>> count >>>> may be change over time. Taskmanager memory usage always increases. After >>>> job cancelation this memory usage doesn't decrease. We've tried to >>>> investigate the problem and we've got the task manager jvm heap snapshot. >>>> According to the jam heap analysis, possible memory leak was Flink list >>>> state descriptor. But we are not sure that is the cause of our memory >>>> problem. How can we solve the problem? >> > >

Re: Flink memory leak

ed to > investigate the problem and we've got the task manager jvm heap snapshot. > According to the jam heap analysis, possible memory leak was Flink list > state descriptor. But we are not sure that is the cause of our memory > problem. How can we solve the problem?

Flink memory leak

the job count may be change over time. Taskmanager memory usage always increases. After job cancelation this memory usage doesn't decrease. We've tried to investigate the problem and we've got the task manager jvm heap snapshot. According to the jam heap analysis, possible me

Re: Memory Leak - Flink / RocksDB ?

t; writeToCassandra(stats); >>> ``` >>> >>> We recently made a switch to RocksDbStateBackend, for it’s suitability for >>> large states/long windows. However, after making the switch a memory issues >>> has come up, the memory utilisation on TaskManag

Re: Memory Leak - Flink / RocksDB ?

he memory utilisation on TaskManager gradually increases from 50 GB to ~63GB until the container is killed. We are unable to figure out what is causing this behaviour, is there some memory leak on the RocksDB ? How much memory should we allocate to the Flink TaskManager? Since, RocksDB is a nativ

Re: Memory Leak - Flink / RocksDB ?

Hi Kien,Sorry it took me sometime to fetch the logs. I am attaching logs of the machine which died due to lack of free memory. Jun 26 13:00:01 staging-east-dataplatform-02-c01 systemd: Created slice user-0.slice. Jun 26 13:00:01 staging-east-dataplatform-02-c01 systemd: Starting user-0.slice. Jun

Re: Memory Leak - Flink / RocksDB ?

Hi Vinay, @Vinay : I am setting RocksDBStateBackend from the code, not from flink-conf.yaml. I am currently trying out the configurations that you have shared. I’ll let you know how they perform. Thank you so much for your help. However, were you able to figure out what exactly is going wrong

Re: Memory Leak - Flink / RocksDB ?

are unable to figure out what is causing this behaviour, is there some memory leak on the RocksDB ? How much memory should we allocate to the Flink TaskManager? Since, RocksDB is a native application and it does not use the JVM how much of the memory should we allocate/leave for RocksDB (out of

Re: Memory Leak - Flink / RocksDB ?

sues > has come up, the memory utilisation on TaskManager gradually increases from > 50 GB to ~63GB until the container is killed. We are unable to figure out > what is causing this behaviour, is there some memory leak on the RocksDB ? > > How much memory should we allocate to the Fli

Memory Leak - Flink / RocksDB ?

has come up, the memory utilisation on TaskManager gradually increases from 50 GB to ~63GB until the container is killed. We are unable to figure out what is causing this behaviour, is there some memory leak on the RocksDB ? How much memory should we allocate to the Flink TaskManager? Since, Rock

Re: List State in RichWindowFunction leads to RocksDb memory leak

gt; context? > > > > Seth Wiesman > > > > *From: *Seth Wiesman > *Reply-To: *"user@flink.apache.org" > *Date: *Friday, February 24, 2017 at 2:51 PM > > > *To: *"user@flink.apache.org" > *Subject: *Re: List State in RichWindowFunction

Re: List State in RichWindowFunction leads to RocksDb memory leak

tek Reply-To: "user@flink.apache.org" Date: Friday, February 24, 2017 at 12:09 PM To: "user@flink.apache.org" Subject: Re: List State in RichWindowFunction leads to RocksDb memory leak Hi Seth, yes, this is a thorny problem but I actually see one additional possible solution (

Re: List State in RichWindowFunction leads to RocksDb memory leak

;user@flink.apache.org" Date: Friday, February 24, 2017 at 12:09 PM To: "user@flink.apache.org" Subject: Re: List State in RichWindowFunction leads to RocksDb memory leak Hi Seth, yes, this is a thorny problem but I actually see one additional possible solution (that will, however, brea

Re: List State in RichWindowFunction leads to RocksDb memory leak

t; > > for (elem <- current) { > if (!previous.contains(elem)) { > > out.collect(elem) > > } > > > > state.add(elem) //store for the next run > > } > > } > > } > > > > The issue with this is that it c

List State in RichWindowFunction leads to RocksDb memory leak

lem) } state.add(elem) //store for the next run } } } The issue with this is that it causes a memory leak with RocksDb. When the WindowOperator executes clearAllState<https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/win

Re: Possible JVM native memory leak

d what options can tune and limit it? > Internal RocksDB objects are not being disposed of properly, probably during > the aforementioned job restarts, and so we have a growing memory leak. If so, > do you have any idea what can cause this? > > Thank you, > Avihai > >

Re: Possible JVM native memory leak

and limit it? 2. Internal RocksDB objects are not being disposed of properly, probably during the aforementioned job restarts, and so we have a growing memory leak. If so, do you have any idea what can cause this? Thank you, Avihai Attached Native Memory Tracking (jcmd VM.native_memory

Possible JVM native memory leak

restarts, and so we have a growing memory leak. If so, do you have any idea what can cause this? Thank you, Avihai Attached Native Memory Tracking (jcmd VM.native_memory summary): Total: reserved=44399603KB, committed=44180459KB - Java Heap (reserved=4096KB, committed

Re: taskmanager memory leak

skManager? >> - If you are running a job, do you use the RocksDB state backend, of the >> FileSystem state backend? >> - Does it grow infinitely, or simply up a certain point and then goes down >> again? >> >> Greetings, >> Stephan >> >> >

Re: taskmanager memory leak

ttps://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#memory-and-performance-debugging > > It will print which parts of the memory are growing. > > What you can also try is to deactivate checkpointing for one run and see > if that solves it. If yes, then I suspect there

Re: taskmanager memory leak

ng. > > What you can also try is to deactivate checkpointing for one run and see if > that solves it. If yes, then I suspect there is a memory leak in the s3 > library (are you using s3, s3a, or s3n?). > > Can you also check what libraries you are using? We have seen

Re: taskmanager memory leak

and see if that solves it. If yes, then I suspect there is a memory leak in the s3 library (are you using s3, s3a, or s3n?). Can you also check what libraries you are using? We have seen cases of memory leaks in the libraries people used. Greetings, Stephan On Thu, Jul 21, 2016 at 5:13 AM, 김동일

Re: taskmanager memory leak

n again? > > Greetings, > Stephan > > > On Wed, Jul 20, 2016 at 5:58 PM, 김동일 wrote: > >> oh. my flink version is 1.0.3. >> >> >> -- Forwarded message -- >> From: 김동일 >> Date: Thu, Jul 21, 2016 at 12:52 AM >> Subjec

Re: taskmanager memory leak

12:52 AM > Subject: taskmanager memory leak > To: user@flink.apache.org > > > I've set up cluster(stand alone). > Taskmanager consumes memory over the Xmx property and it grows up > continuously. > I saw this link( > http://mail-

taskmanager memory leak

I've set up cluster(stand alone). Taskmanager consumes memory over the Xmx property and it grows up continuously. I saw this link( http://mail-archives.apache.org/mod_mbox/flink-dev/201606.mbox/%3ccak2vtervsw4muboc4swix0mr6y9bijznjuypf6_f9f0g9-_...@mail.gmail.com%3E ). So i set the taskmanager.memo

Fwd: taskmanager memory leak

oh. my flink version is 1.0.3. -- Forwarded message -- From: 김동일 Date: Thu, Jul 21, 2016 at 12:52 AM Subject: taskmanager memory leak To: user@flink.apache.org I've set up cluster(stand alone). Taskmanager consumes memory over the Xmx property and it grows up continuous

Re: Memory Leak using ProcessingTimeTimers?

Hi, by the way, form looking at your email I gather that you want to do some kind of session windowing. Is that correct? I have a pull request that should make it into the next version that adds proper support for session windows. Right now this is only implemented for event-time, since this is the

Re: Memory Leak using ProcessingTimeTimers?

Hi Aljoscha, thanks for looking into it. I have moved the discussion to the issue. Cheers, Konstantin On 27.03.2016 09:35, Aljoscha Krettek wrote: > Hi, > you are right, this is a problem. In an earlier version we were only > setting very few actual timers using the RuntimeContext because a fir

Re: Memory Leak using ProcessingTimeTimers?

Hi, you are right, this is a problem. In an earlier version we were only setting very few actual timers using the RuntimeContext because a firing timer will trigger all the timers with a lower timestamp that we have stored in the trigger queue. We have to change the lower level trigger service (in

Memory Leak using ProcessingTimeTimers?

Hi everyone, we were testing a Flink streaming job (1.0.0), with a GlobalWindow on a KeyedStream with custom Trigger. On each element the trigger function registers a processing time timer and deletes the currently registered processing time timer. So we are registering a lot of timers, but also

RE: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

...@gmail.com] De la part de Stephan Ewen Envoyé : lundi 14 décembre 2015 10:54 À : user@flink.apache.org Objet : Re: Crash in a simple "mapper style" streaming app likely due to a memory leak ? Hi! That is curious. Can you tell us a bit more about your setup? - Did you set Flink to use off-h

Re: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

where it prints the environment) here? Thanks, Stephan On Mon, Dec 14, 2015 at 9:57 AM, LINZ, Arnaud wrote: > Hello, > > > > I did have an off-heap memory leak in my streaming application, due to : > > https://issues.apache.org/jira/browse/HADOOP-12007. > > > >

RE: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

Hello, I did have an off-heap memory leak in my streaming application, due to : https://issues.apache.org/jira/browse/HADOOP-12007. Now that I use the CodecPool to close that leak, I get under load the following error

Re: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

Hi Arnaud! Java direct-memory is tricky to debug. You can turn on the memory logging or check the TaskManager tab in teh web dashboard - both report on direct memory consumption. One thing you can look for is forgetting to close streams. That means the streams consume native resources until the J

Re: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

> On 13 Nov 2015, at 15:49, LINZ, Arnaud wrote: > > Hi Robert, > > Thanks, it works with 50% -- at least way past the previous crash point. > > In my opinion (I lack real metrics), the part that uses the most memory is > the M2 mapper, instantiated once per slot. > The most complex part is

Re: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

alue. > > > > Regards, > > Arnaud > > > > *De :* Robert Metzger [mailto:rmetz...@apache.org] > *Envoyé :* vendredi 13 novembre 2015 15:16 > *À :* user@flink.apache.org > *Objet :* Re: Crash in a simple "mapper style" streaming app likely due > to a memory leak

RE: Crash in a simple "mapper style" streaming app likely due to a memory leak ?

“root-protected” flink-conf.yaml for all the users & flink jobs with that value. Regards, Arnaud De : Robert Metzger [mailto:rmetz...@apache.org] Envoyé : vendredi 13 novembre 2015 15:16 À : user@flink.apache.org Objet : Re: Crash in a simple "mapper style" streaming app likely due to