On Mon, 28 Sep 2026 00:48:39 GMT, Ashay Rane <[email protected]> wrote:
>> Prior to this patch, the `StackYellowPages` and `StackRedPages` >> influenced the total reserved stack memory, but they didn't change the >> minimum stack size specified as required by HotSpot to Windows. This >> patch adds the call to `SetThreadStackGuarantee()` to set the minimum >> stack size. >> >> The key change here is that the argument to `SetThreadStackGuarantee()` >> is set to one less than the total number of yellow pages. The one less >> page is because Windows uses a guard page (access to which tells Windows >> to commit more stack pages); in the terminal case, we want the guard >> page to coincide with the top-most yellow stack page, thus leaving all >> except one yellow page available as the stack size. >> >> Importantly, however, we do not use the red page count as the argument >> to `SetThreadStackGurantee()`, since the red stack pages are unavailable >> for handling recoverable overflows. The red page count impacts the >> _total reserved_ stack size, just not the _minimum_ stack size. Still, >> forcing the red stack page count to be included into the computation of >> the argument to `SetThreadStackGurantee()` causes HotSpot to fail with >> the assertion `assert(!in_vm) failed: Undersized StackShadowPages`, >> since Windows is unable to commit more stack pages due to the minimum >> stack size now being larger than just the yellow page count minus one. >> >> The accompanying test passes zero to `SetThreadStackGuarantee()` to >> probe the current minimum stack size, which we then compare against the >> expected size based on the yellow page count. The same test fails >> without this patch on both Windows/x64 and on Windows/ARM64. >> >> Validated this patch by running through all tier 1, 2, and 3 HotSpot >> jtreg tests on Windows/x64 and Windows/ARM64 in FastDebug config. >> >> --------- >> - [x] I confirm that I make this contribution in accordance with the >> [OpenJDK Interim AI Policy](https://openjdk.org/legal/ai). > > Ashay Rane has updated the pull request incrementally with one additional > commit since the last revision: > > Handle EXCEPTION_STACK_OVERFLOW when yellow zone is unguarded > > The previous code did not handle a corner case that arises when HotSpot > receives `EXCEPTION_STACK_OVERFLOW` (since the guard page cannot be > moved further down) while already processing a recoverable overflow. In > this case, the yellow and reserved zones are writable (as part of the > recoverable overflow), so we retry the same instruction. If the > faulting probe reaches the red zone, we'll receive > `EXCEPTION_ACCESS_VIOLATION`, which will be handled as an unrecoverable > overflow. So here's some more information that might be relevant. Apologies for the long message. It looks like we can avoid EXCEPTION_STACK_OVERFLOW if we pre-commit all pages (in addition to marking the red+yellow pages as PAGE_NOACCESS), but given the potential performance penalty, perhaps we can't afford to go with that approach. If we want Windows to commit stack pages as necessary using PAGE_GUARD, then I'm afraid we have to opt into EXCEPTION_STACK_OVERFLOW. In that case, we don't need to mark the last few pages for red and yellow zones; rather the canonical way is to call `SetThreadStackGuarantee()` with the argument as `red_page_count + yellow_page_count - 1 /* for the guard page */ - 1 /* for the bottom page */` (more on this bottom page later). The most important thing to note here is that the stack guarantee pages are a set of _moving_ pages, so sometime prior to an overflow, the stack may look like this: <high addresses> usable stack (say, N pages) guard page stack guarantee pages reserved / non-committed pages (say, M pages) bottom page <low addresses> After the application accesses an additional stack page, assuming it still hasn't caused a stack overflow, we will have N+1 pages of usable stack and M-1 pages of reserved/non-committed pages. In other words, the stack guarantee pages are shifted down one page at a time as more stack pages get committed. This bottom page, then, becomes part of the trigger condition. Specifically, when the lowest page of the stack guarantee pages hits the bottom page, Windows raises EXCEPTION_STACK_OVERFLOW, in addition to making as many pages writeable as the count of the stack guarantee pages. The guard page disappears as well. <high addresses> usable stack stack guarantee pages -- PAGE_READWRITE bottom page <low addresses> At this point, Windows expects the exception handler to run, after which the application is expected to call `_resetstkoflw()`, which partially unwinds the stack and restores the guard page, so the stack layout goes back to looking like the pre-overflow layout (of course, with a different number of usable stack pages than before, based on the result of the unwinding). Crucially, `_resetstkoflw()` does not de-commit pages that were already committed, and as a result, function prologues cannot use far-access stack banging (like `stack_pointer - shadow_stack_distance`) and instead they need to use a page-by-page banging. This is because far-access banging may land below the page guard, with the accessed page being in a committed state from the previous stack overflow, making HotSpot incorrectly conclude that there's enough stack available: <high addresses> usable stack -------------+ guard page | stack guarantee pages | committed pages <----------+ (far-access stack banging may land here) bottom page <low addresses> The key simplification as a result of this Windows-native approach is that an EXCEPTION_STACK_OVERFLOW in Java state triggers the recoverable handler. For all other cases, we trigger the unrecoverable handler. An EXCEPTION_ACCESS_VIOLATION can occur if HotSpot accesses the bottom page, in which case, we trigger the unrecoverable handler. Once EXCEPTION_STACK_OVERFLOW is delivered and the corresponding handler is running, prior to calling `_resetstkoflw()`, we cannot make the page above the bottom guard PAGE_NOACCESS (in the hopes that accessing that page will cause EXCEPTION_ACCESS_VIOLATION). This is because Windows sees that the stack is still exhausted (since `_resetstkoflw()` isn't called yet), so Windows doesn't deliver EXCEPTION_ACCESS_VIOLATION to the vectored exception handler. Instead, HotSpot crashes with a raw exit code signifying that EXCEPTION_ACCESS_VIOLATION was raised but was not delivered to HotSpot. I have a patch that implements this pure-native stack management, validated using tier 1-3 HotSpot, JDK, and LangTools tests. There are additional details, but I can elaborate on those if needed. To summarize, if we want to go with the pure-HotSpot approach (i.e. fixed red and yellow zones whose accesses raise an exception), we'd need to pre-commit stack pages. If we want to go with the canonical approach to stack management on Windows, the red+yellow zone becomes a floating zone whose protection bits can't be set to PAGE_NOACCESS. I'm happy to organize all of this into a document on cr.openjdk.org if there's interest (so that you don't have to go over the PR comments so far). ------------- PR Comment: https://git.openjdk.org/jdk/pull/32365#issuecomment-5992475874
