Steve Niemitz created BEAM-7684:
-----------------------------------
Summary: WindmillStateCache grossly misunderestimates object size
in its weighting function
Key: BEAM-7684
URL: https://issues.apache.org/jira/browse/BEAM-7684
Project: Beam
Issue Type: Bug
Components: runner-dataflow
Reporter: Steve Niemitz
As a little experiment, we set the WindmillStateCache size to 2 GB. After we
let a job run for a few hours, the workers all died from OOMs. Digging into
the memory usage, over 10 GB of heap was used by the WindmillStateCache. The
total weight the cache had calculated however was only ~800 MB.
It looks like the StateCacheEntry calculation is way off. The weight of an
entry was calculated as only ~200, while the actual heap usage of it (as
reported by YourKit at least) was around 2,400 bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
