Thanks for the clarification. Prakash
On Friday, June 8, 2018 at 11:32:24 PM UTC+5:30, Kenton Varda wrote: > > No clean solution, but with hacks we've been able to solve the problem for > our use case. > > I set the heap limit to 8x the limit I actually want. Then, after each > call into the isolate, I check if the memory usage has gone over the > intended limit. If so, I invoke garbage collection. If it's *still* over > the limit after that, then I terminate the isolate at that point. > > Meanwhile, we also enforce a CPU time limit of 50ms. In practice, a script > that allocates tons of memory tends to run out of CPU time before it can > hit the 8x heap limit (especially as the GC slows things down when > approaching the limit). > > This together seems to solve the case of scripts that accidentally > allocate too much memory. > > A carefully-crafted malicious script can still cause an abort. In that > case, the SIGABRT single handler blacklists the offending script and simply > leaves the thread in limbo rather than exit the process. Some time later, > if we need that RAM back, we can do a clean restart, in which we start a > new process to handle new requests while letting the old process finish up > anything in-flight and then exit. > > One thing I haven't had time to experiment with yet is whether we can > longjmp() out of the signal handler and then delete the isolate, rather > than leave it in limbo. Obviously this would be super-leaky, but perhaps we > could reclaim most of the resources associated with the isolate. (In my > first message in this thread I was contemplating whether I could compile > with exceptions enabled in order to be able to do a proper stack unwind > invoking destructors, which could perhaps make things a little bit less > leaky, even though V8 is not exception-safe. But that won't actually work > because libunwind can't unwind through JIT'd code.) > > -Kenton > > On Fri, Jun 8, 2018 at 3:06 AM, Prakash Bailkeri <prakash....@gmail.com > <javascript:>> wrote: > >> Hello Kenton, >> >> Did you find a reliable way to resolve "isolate heap" OOM issue? I had >> similar question and proposing to use AddNearHeapLimitCallback? >> *https://groups.google.com/forum/?nomobile=true#!topic/v8-users/qLs7-XT2Zvg >> <https://groups.google.com/forum/?nomobile=true#!topic/v8-users/qLs7-XT2Zvg>* >> >> I am trying to solve a issue where hacker/"accidental code"/"bug" asks >> for memory more than the heap memory limit on isolate >> (set_max_old_space_size) >> >> Any feedback from you? >> >> Thanks, Prakash >> >> On Tuesday, October 31, 2017 at 4:10:09 AM UTC+5:30, >> ken...@cloudflare.com wrote: >>> >>> Hi all, >>> >>> Sorry, I think my first message was confusing. >>> >>> To be clear, in my case, my C++ process is nowhere near being out of >>> memory. But, I'm configuring the V8 isolates I create with a low heap size, >>> using v8::ResourceConstraints::set_max_old_space_size(). It's the V8 heap >>> that is running out of memory. >>> >>> I'm well aware that writing C++ code that can recover from malloc() >>> failure is basically impossible. >>> >>> But in my case, V8 is hitting a self-imposed limit on specifically the >>> V8 heap. It seems like we should be able to find a hack to work around this. >>> >>> For example, I could live with a solution like: When V8's heap is >>> exhausted, it calls the OOM callback. If that returns without terminating, >>> then V8 doubles its own heap limit and continues. >>> >>> In this case, I would have my OOM callback call TerminateExecution() and >>> arrange for the isolate to be discarded as soon as control returns back out >>> of V8. It's fine if extra memory is allocated in the meantime as a stopgap >>> to avoid aborting. >>> >>> Is there a way to accomplish something like this? If not, how hard would >>> it be to add? >>> >>> >> For example, what if I compile with C++ exceptions enabled, and have >>> my OOM >>> >> handler throw an exception, hence unwinding the stack back to where I >>> >> entered V8. Then, I promptly destroy the isolate. Would that work? >>> > >>> > No. It would end very badly. V8 is not exception-safe. >>> >>> To be clear about this suggestion: Obviously it's not going to be clean, >>> but I was hoping to discuss what kind of bad things would actually be >>> likely to happen. The isolate would be left in an inconsistent state, but >>> if I'm deleting it anyway, how much does that matter? Small memory leaks >>> would not be a big deal here, since I can periodically drain the process >>> and start a new one to "fix" them. >>> >>> -Kenton >>> >>> On Wednesday, October 25, 2017 at 12:13:22 AM UTC-7, Andreas Rossberg >>> wrote: >>>> >>>> To supplement Ben's answer: the reason why this was never a design goal >>>> is that it is practically impossible to achieve. It is extremely >>>> difficult to recover reliably from OOM in certain situations. AFAIK, no >>>> engine out there is able to guarantee that. >>>> >>>> On 24 October 2017 at 22:44, Ben Noordhuis <in...@bnoordhuis.nl> wrote: >>>> >>>>> On Tue, Oct 24, 2017 at 10:17 PM, 'Kenton Varda' via v8-users >>>>> <v8-u...@googlegroups.com> wrote: >>>>> > Hi v8-users, >>>>> > >>>>> > It appears that in some cases V8 will abort the process when it runs >>>>> out of >>>>> > heap space rather than throw a JS exception. The behavior can be >>>>> overridden >>>>> > by registering an OOM callback, but if that callback returns without >>>>> > aborting, it seems V8 promptly crashes. >>>>> > >>>>> > It seems like some code paths are designed to handle OOM gracefully, >>>>> but >>>>> > others aren't. >>>>> > >>>>> > For my use case, it's pretty important that a malicious script >>>>> cannot cause >>>>> > the process to abort, since our processes are multi-tenant. Ideally >>>>> OOM >>>>> > would throw an exception, but terminating the isolate is also >>>>> acceptable, as >>>>> > long as other isolates can keep going. >>>>> > >>>>> > Is there any way to accomplish this? >>>>> >>>>> No. Graceful handling of OOM conditions is not one of V8's design >>>>> goals. >>>>> >>>>> > For example, what if I compile with C++ exceptions enabled, and have >>>>> my OOM >>>>> > handler throw an exception, hence unwinding the stack back to where I >>>>> > entered V8. Then, I promptly destroy the isolate. Would that work? >>>>> >>>>> No. It would end very badly. V8 is not exception-safe. >>>>> >>>>> > Or, is there some trick to making V8 less crashy on OOM, aside from >>>>> going >>>>> > through and fixing all the code paths that crash (which probably >>>>> isn't >>>>> > feasible for me)? >>>>> >>>>> No tricks, no. The best you can do is monitor memory usage and call >>>>> `Isolate::TerminateExecution()` when it gets too high but that won't >>>>> be 100% reliable; OOM conditions in C++ code will still be fatal. >>>>> >>>>> Probably not the answers you were hoping for but there it is. >>>>> >>>>> -- >>>>> -- >>>>> v8-users mailing list >>>>> v8-u...@googlegroups.com >>>>> http://groups.google.com/group/v8-users >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "v8-users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to v8-users+u...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >> -- >> v8-users mailing list >> v8-u...@googlegroups.com <javascript:> >> http://groups.google.com/group/v8-users >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "v8-users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/v8-users/vKn1hVs8KNQ/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> v8-users+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- -- v8-users mailing list v8-users@googlegroups.com http://groups.google.com/group/v8-users --- You received this message because you are subscribed to the Google Groups "v8-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
