Ok, thanks. I've said this in another thread but everything seems to go completely idle during checkpoints while waiting on 1 operator, there's no CPU usage, hardly any disk usage. I'll assume it's something else then.
On Wed, Dec 16, 2020 at 10:42 AM Robert Metzger <rmetz...@apache.org> wrote: > I don't think the direct memory is causing any performance bottlenecks. > The backpressure is probably caused by something else (high CPU load, slow > external system, data skew) > > On Wed, Dec 16, 2020 at 7:23 PM Steven Wu <stevenz...@gmail.com> wrote: > >> if you are running out of direct buffer, you will see >> "java.lang.OutOfMemoryError: >> Direct buffer memory" >> >> On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley <r...@remind101.com> wrote: >> >>> Thanks for the reply. If what I'm understanding is correct there's no >>> chance of an OOM, but since direct memory is for I/O, it being completely >>> filled may be a sign of backpressure? Currently one of our operators takes >>> a tremendous amount of time to align during a checkpoint. Could increasing >>> direct memory help checkpointing by improving I/O performance across the >>> whole plan (assuming I/O is at least part of the bottleneck)? >>> >>> On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <rmetz...@apache.org> >>> wrote: >>> >>>> Hey Rex, >>>> >>>> the direct memory is used for IO. There is no concept of direct memory >>>> being "full". The only thing that can happen is that you have something in >>>> place (Kubernetes, YARN) that limits / enforces the memory use of a Flink >>>> process, and you run out of your memory allowance. The direct memory is >>>> allocated outside of the heap's upper limit, thus you could run out of the >>>> budget. >>>> But Flink is usually properly configuring the memory limits correctly, >>>> to avoid running into those situations. >>>> >>>> tl;dr: you don't need to worry about this. >>>> >>>> >>>> On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <r...@remind101.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> Our job consistently shows >>>>> Outside JVM >>>>> Type >>>>> Count >>>>> Used >>>>> Capacity >>>>> *Direct* 32,839 1.03 GB 1.03 GB >>>>> for direct memory. >>>>> >>>>> Is it typical for it to be full? What are the consequences that we may >>>>> not be noticing of direct memory being full? >>>>> >>>>> Thanks! >>>>> >>>>> -- >>>>> >>>>> Rex Fenley | Software Engineer - Mobile and Backend >>>>> >>>>> >>>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >>>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >>>>> <https://www.facebook.com/remindhq> >>>>> >>>> >>> >>> -- >>> >>> Rex Fenley | Software Engineer - Mobile and Backend >>> >>> >>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >>> <https://www.facebook.com/remindhq> >>> >> -- Rex Fenley | Software Engineer - Mobile and Backend Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | FOLLOW US <https://twitter.com/remindhq> | LIKE US <https://www.facebook.com/remindhq>
