[
https://issues.apache.org/jira/browse/HUDI-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3774:
----------------------------------
Description:
UPDATE
-------
Addressed in https://github.com/apache/hudi/pull/5181
-------
Our tests occasionally crash w/ OOM in Azure.
I re-run the tests locally under the same conditions trying to reproduce it and
took a few heap dumps.
I see that out of about 1.1Gb about 272Mb are the HoodieLogFileReaders retained
t/h shutdown hooks. This means that those HoodieLogFileReader has not been
properly closed (potentially under conditions that there were some exceptions
thrown)
To make sure that all readers get closed upon reaching their end of life, we'd
use try-resource-statement wherever possible, which will guarantee proper
closing for the reader under any possible control-flow
{code:java}
try (Reader r = new Reader) {
// ...
}
{code}
!Screen Shot 2022-04-01 at 11.58.14 AM.png!
was:
Our tests occasionally crash w/ OOM in Azure.
I re-run the tests locally under the same conditions trying to reproduce it and
took a few heap dumps.
I see that out of about 1.1Gb about 272Mb are the HoodieLogFileReaders retained
t/h shutdown hooks. This means that those HoodieLogFileReader has not been
properly closed (potentially under conditions that there were some exceptions
thrown)
To make sure that all readers get closed upon reaching their end of life, we'd
use try-resource-statement wherever possible, which will guarantee proper
closing for the reader under any possible control-flow
{code:java}
try (Reader r = new Reader) {
// ...
}
{code}
!Screen Shot 2022-04-01 at 11.58.14 AM.png!
> Fix memory leaks through shutdown hooks
> ---------------------------------------
>
> Key: HUDI-3774
> URL: https://issues.apache.org/jira/browse/HUDI-3774
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Alexey Kudinkin
> Assignee: Alexey Kudinkin
> Priority: Blocker
> Fix For: 0.11.0
>
> Attachments: Screen Shot 2022-04-01 at 11.58.14 AM.png, Screen Shot
> 2022-04-01 at 6.24.31 PM.png, image-2022-04-01-18-20-40-163.png
>
>
> UPDATE
> -------
> Addressed in https://github.com/apache/hudi/pull/5181
> -------
>
> Our tests occasionally crash w/ OOM in Azure.
> I re-run the tests locally under the same conditions trying to reproduce it
> and took a few heap dumps.
> I see that out of about 1.1Gb about 272Mb are the HoodieLogFileReaders
> retained t/h shutdown hooks. This means that those HoodieLogFileReader has
> not been properly closed (potentially under conditions that there were some
> exceptions thrown)
>
> To make sure that all readers get closed upon reaching their end of life,
> we'd use try-resource-statement wherever possible, which will guarantee
> proper closing for the reader under any possible control-flow
> {code:java}
> try (Reader r = new Reader) {
> // ...
> }
> {code}
>
> !Screen Shot 2022-04-01 at 11.58.14 AM.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
