Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

-- >> *From:* Kostas Kloudas >> *Sent:* 03 February 2020 15:39 >> *To:* Mark Harris >> *Cc:* Piotr Nowojski ; Cliff Resnick < >> cre...@gmail.com>; David Magalhães ; Till >> Rohrmann ; flink-u...@apache.org < >> flink-u...@apache.org>

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

> Sorry, stupid question: How do I set that for a StreamingFileSink? > > Best regards, > > Mark > -- > *From:* Kostas Kloudas > *Sent:* 03 February 2020 14:58 > *To:* Mark Harris > *Cc:* Piotr Nowojski ; Cliff Resnick < > cre...@gmai

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

lkFormat? Best regards, Mark From: Kostas Kloudas Sent: 03 February 2020 15:39 To: Mark Harris Cc: Piotr Nowojski ; Cliff Resnick ; David Magalhães ; Till Rohrmann ; flink-u...@apache.org Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

rk Harris > *Cc:* Piotr Nowojski ; Cliff Resnick < > cre...@gmail.com>; David Magalhães ; Till Rohrmann > ; flink-u...@apache.org > *Subject:* Re: GC overhead limit exceeded, memory full of DeleteOnExit > hooks for S3a files > > Hi Mark, > > Have you tried to se

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi Mark, Have you tried to set your rolling policy to close inactive part files after some time [1]? If the part files in the buckets are inactive and there are no new part files, then the state handle for

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

*To:* Piotr Nowojski > *Cc:* Cliff Resnick ; David Magalhães < > speeddra...@gmail.com>; Till Rohrmann ; > flink-u...@apache.org ; kkloudas < > kklou...@apache.org> > *Subject:* Re: GC overhead limit exceeded, memory full of DeleteOnExit > hooks for S3a files > > Hi, &

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

nick ; David Magalhães > ; Till Rohrmann ; > flink-u...@apache.org ; kkloudas > Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks > for S3a files > > Hi, > > Thanks for your help with this. 🙂 > > The EMR cluster has 3 15GB VMs, and the flink clu

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Sent: 30 January 2020 14:36 To: Piotr Nowojski Cc: Cliff Resnick ; David Magalhães ; Till Rohrmann ; flink-u...@apache.org ; kkloudas Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi, Thanks for your help with this. 🙂 The EMR cluster has 3 15GB VMs

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

ry 2020 13:44 To: Mark Harris Cc: Cliff Resnick ; David Magalhães ; Till Rohrmann ; flink-u...@apache.org ; kkloudas Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi, What is your job setup? Size of the nodes, memory settings of the Flink/JVM? 9 041 060

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

e. > Could this be a factor? > > Best regards, > > Mark > From: Piotr Nowojski > Sent: 27 January 2020 16:16 > To: Cliff Resnick > Cc: David Magalhães ; Mark Harris > ; Till Rohrmann ; > flink-u...@apache.org ; kkloudas > Subject: Re: GC overhead limit exceeded, memo

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

ris mailto:mark.har...@hivehome.com>>; flink-u...@apache.org<mailto:flink-u...@apache.org> mailto:flink-u...@apache.org>>; kkloudas mailto:kklou...@apache.org>> Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi, This is probably a known iss

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

t;> Best regards, >>> >>> Mark >>> ---------- >>> *From:* Piotr Nowojski on behalf of Piotr >>> Nowojski >>> *Sent:* 22 January 2020 13:29 >>> *To:* Till Rohrmann >>> *Cc:* Mark Harris ; flink-u...@apach

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

i > mailto:pi...@ververica.com>> > Sent: 22 January 2020 13:29 > To: Till Rohrmann mailto:trohrm...@apache.org>> > Cc: Mark Harris mailto:mark.har...@hivehome.com>>; > flink-u...@apache.org <mailto:flink-u...@apache.org> <mailto:flink-u...@apache.org>>

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

riting the buffer files, but the taskmanager breaks >> with the same problem. >> >> Best regards, >> >> Mark >> -- >> *From:* Piotr Nowojski on behalf of Piotr >> Nowojski >> *Sent:* 22 January 2020 13:29 >> *To:* Till Rohrmann >> *Cc:* Mark H

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Nowojski on behalf of Piotr > Nowojski > *Sent:* 22 January 2020 13:29 > *To:* Till Rohrmann > *Cc:* Mark Harris ; flink-u...@apache.org < > flink-u...@apache.org>; kkloudas > *Subject:* Re: GC overhead limit exceeded, memory full of DeleteOnExit > hooks for S3a files &

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

rds, Mark From: Piotr Nowojski on behalf of Piotr Nowojski Sent: 22 January 2020 13:29 To: Till Rohrmann Cc: Mark Harris ; flink-u...@apache.org ; kkloudas Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi, This is probably a k

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Hi, This is probably a known issue of Hadoop [1]. Unfortunately it was only fixed in 3.3.0. Piotrek [1] https://issues.apache.org/jira/browse/HADOOP-15658 > On 22 Jan 2020, at 13:56, Till Rohrmann wrote: > > Thanks for reporting this issu

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Thanks for reporting this issue Mark. I'm pulling Klou into this conversation who knows more about the StreamingFileSink. @Klou does the StreamingFileSink relies on DeleteOnExitHooks to clean up files? Cheers, Till On Tue, Jan 21, 2020 at 3:38 PM Mark Harris wrote: > Hi, > > We're using flink 1

GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Hi, We're using flink 1.7.2 on an EMR cluster v emr-5.22.0, which runs hadoop v "Amazon 2.8.5". We've recently noticed that some TaskManagers fail (causing all the jobs running on them to fail) with an "java.lang.OutOfMemoryError: GC overhead limit exceeded”. The taskmanager (and jobs that shou