No clean solution, but with hacks we've been able to solve the problem for our use case.
I set the heap limit to 8x the limit I actually want. Then, after each call into the isolate, I check if the memory usage has gone over the intended limit. If so, I invoke garbage collection. If it's *still* over the limit after that, then I terminate the isolate at that point. Meanwhile, we also enforce a CPU time limit of 50ms. In practice, a script that allocates tons of memory tends to run out of CPU time before it can hit the 8x heap limit (especially as the GC slows things down when approaching the limit). This together seems to solve the case of scripts that accidentally allocate too much memory. A carefully-crafted malicious script can still cause an abort. In that case, the SIGABRT single handler blacklists the offending script and simply leaves the thread in limbo rather than exit the process. Some time later, if we need that RAM back, we can do a clean restart, in which we start a new process to handle new requests while letting the old process finish up anything in-flight and then exit. One thing I haven't had time to experiment with yet is whether we can longjmp() out of the signal handler and then delete the isolate, rather than leave it in limbo. Obviously this would be super-leaky, but perhaps we could reclaim most of the resources associated with the isolate. (In my first message in this thread I was contemplating whether I could compile with exceptions enabled in order to be able to do a proper stack unwind invoking destructors, which could perhaps make things a little bit less leaky, even though V8 is not exception-safe. But that won't actually work because libunwind can't unwind through JIT'd code.) -Kenton On Fri, Jun 8, 2018 at 3:06 AM, Prakash Bailkeri <prakash.bailk...@gmail.com > wrote: > Hello Kenton, > > Did you find a reliable way to resolve "isolate heap" OOM issue? I had > similar question and proposing to use AddNearHeapLimitCallback? > *https://groups.google.com/forum/?nomobile=true#!topic/v8-users/qLs7-XT2Zvg > <https://groups.google.com/forum/?nomobile=true#!topic/v8-users/qLs7-XT2Zvg>* > > I am trying to solve a issue where hacker/"accidental code"/"bug" asks for > memory more than the heap memory limit on isolate (set_max_old_space_size) > > Any feedback from you? > > Thanks, Prakash > > On Tuesday, October 31, 2017 at 4:10:09 AM UTC+5:30, ken...@cloudflare.com > wrote: >> >> Hi all, >> >> Sorry, I think my first message was confusing. >> >> To be clear, in my case, my C++ process is nowhere near being out of >> memory. But, I'm configuring the V8 isolates I create with a low heap size, >> using v8::ResourceConstraints::set_max_old_space_size(). It's the V8 >> heap that is running out of memory. >> >> I'm well aware that writing C++ code that can recover from malloc() >> failure is basically impossible. >> >> But in my case, V8 is hitting a self-imposed limit on specifically the V8 >> heap. It seems like we should be able to find a hack to work around this. >> >> For example, I could live with a solution like: When V8's heap is >> exhausted, it calls the OOM callback. If that returns without terminating, >> then V8 doubles its own heap limit and continues. >> >> In this case, I would have my OOM callback call TerminateExecution() and >> arrange for the isolate to be discarded as soon as control returns back out >> of V8. It's fine if extra memory is allocated in the meantime as a stopgap >> to avoid aborting. >> >> Is there a way to accomplish something like this? If not, how hard would >> it be to add? >> >> >> For example, what if I compile with C++ exceptions enabled, and have >> my OOM >> >> handler throw an exception, hence unwinding the stack back to where I >> >> entered V8. Then, I promptly destroy the isolate. Would that work? >> > >> > No. It would end very badly. V8 is not exception-safe. >> >> To be clear about this suggestion: Obviously it's not going to be clean, >> but I was hoping to discuss what kind of bad things would actually be >> likely to happen. The isolate would be left in an inconsistent state, but >> if I'm deleting it anyway, how much does that matter? Small memory leaks >> would not be a big deal here, since I can periodically drain the process >> and start a new one to "fix" them. >> >> -Kenton >> >> On Wednesday, October 25, 2017 at 12:13:22 AM UTC-7, Andreas Rossberg >> wrote: >>> >>> To supplement Ben's answer: the reason why this was never a design goal >>> is that it is practically impossible to achieve. It is extremely >>> difficult to recover reliably from OOM in certain situations. AFAIK, no >>> engine out there is able to guarantee that. >>> >>> On 24 October 2017 at 22:44, Ben Noordhuis <in...@bnoordhuis.nl> wrote: >>> >>>> On Tue, Oct 24, 2017 at 10:17 PM, 'Kenton Varda' via v8-users >>>> <v8-u...@googlegroups.com> wrote: >>>> > Hi v8-users, >>>> > >>>> > It appears that in some cases V8 will abort the process when it runs >>>> out of >>>> > heap space rather than throw a JS exception. The behavior can be >>>> overridden >>>> > by registering an OOM callback, but if that callback returns without >>>> > aborting, it seems V8 promptly crashes. >>>> > >>>> > It seems like some code paths are designed to handle OOM gracefully, >>>> but >>>> > others aren't. >>>> > >>>> > For my use case, it's pretty important that a malicious script cannot >>>> cause >>>> > the process to abort, since our processes are multi-tenant. Ideally >>>> OOM >>>> > would throw an exception, but terminating the isolate is also >>>> acceptable, as >>>> > long as other isolates can keep going. >>>> > >>>> > Is there any way to accomplish this? >>>> >>>> No. Graceful handling of OOM conditions is not one of V8's design goals. >>>> >>>> > For example, what if I compile with C++ exceptions enabled, and have >>>> my OOM >>>> > handler throw an exception, hence unwinding the stack back to where I >>>> > entered V8. Then, I promptly destroy the isolate. Would that work? >>>> >>>> No. It would end very badly. V8 is not exception-safe. >>>> >>>> > Or, is there some trick to making V8 less crashy on OOM, aside from >>>> going >>>> > through and fixing all the code paths that crash (which probably isn't >>>> > feasible for me)? >>>> >>>> No tricks, no. The best you can do is monitor memory usage and call >>>> `Isolate::TerminateExecution()` when it gets too high but that won't >>>> be 100% reliable; OOM conditions in C++ code will still be fatal. >>>> >>>> Probably not the answers you were hoping for but there it is. >>>> >>>> -- >>>> -- >>>> v8-users mailing list >>>> v8-u...@googlegroups.com >>>> http://groups.google.com/group/v8-users >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "v8-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to v8-users+u...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- > -- > v8-users mailing list > v8-users@googlegroups.com > http://groups.google.com/group/v8-users > --- > You received this message because you are subscribed to a topic in the > Google Groups "v8-users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/v8-users/vKn1hVs8KNQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > v8-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- -- v8-users mailing list v8-users@googlegroups.com http://groups.google.com/group/v8-users --- You received this message because you are subscribed to the Google Groups "v8-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
