I don't think the direct memory is causing any performance bottlenecks. The backpressure is probably caused by something else (high CPU load, slow external system, data skew)
On Wed, Dec 16, 2020 at 7:23 PM Steven Wu <stevenz...@gmail.com> wrote: > if you are running out of direct buffer, you will see > "java.lang.OutOfMemoryError: > Direct buffer memory" > > On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley <r...@remind101.com> wrote: > >> Thanks for the reply. If what I'm understanding is correct there's no >> chance of an OOM, but since direct memory is for I/O, it being completely >> filled may be a sign of backpressure? Currently one of our operators takes >> a tremendous amount of time to align during a checkpoint. Could increasing >> direct memory help checkpointing by improving I/O performance across the >> whole plan (assuming I/O is at least part of the bottleneck)? >> >> On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <rmetz...@apache.org> >> wrote: >> >>> Hey Rex, >>> >>> the direct memory is used for IO. There is no concept of direct memory >>> being "full". The only thing that can happen is that you have something in >>> place (Kubernetes, YARN) that limits / enforces the memory use of a Flink >>> process, and you run out of your memory allowance. The direct memory is >>> allocated outside of the heap's upper limit, thus you could run out of the >>> budget. >>> But Flink is usually properly configuring the memory limits correctly, >>> to avoid running into those situations. >>> >>> tl;dr: you don't need to worry about this. >>> >>> >>> On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <r...@remind101.com> wrote: >>> >>>> Hello, >>>> >>>> Our job consistently shows >>>> Outside JVM >>>> Type >>>> Count >>>> Used >>>> Capacity >>>> *Direct* 32,839 1.03 GB 1.03 GB >>>> for direct memory. >>>> >>>> Is it typical for it to be full? What are the consequences that we may >>>> not be noticing of direct memory being full? >>>> >>>> Thanks! >>>> >>>> -- >>>> >>>> Rex Fenley | Software Engineer - Mobile and Backend >>>> >>>> >>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >>>> <https://www.facebook.com/remindhq> >>>> >>> >> >> -- >> >> Rex Fenley | Software Engineer - Mobile and Backend >> >> >> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >> <https://www.facebook.com/remindhq> >> >
