Found during code review of [JDK-8292695](https://bugs.openjdk.org/browse/JDK-8292695).
We have two bugs in libjsig when we install hotspot signal handlers. Relevant code in libjsig: int sigaction(int sig, const struct sigaction *act, struct sigaction *oact) { <snip> sigused = sigismember(&jvmsigs, sig); if (jvm_signal_installed && sigused) { /* jvm has installed its signal handler for this signal. */ /* Save the handler. Don't really install it. */ if (oact != NULL) { *oact = sact[sig]; } if (act != NULL) { sact[sig] = *act; } signal_unlock(); return 0; } else if (jvm_signal_installing) { /* jvm is installing its signal handlers. Install the new * handlers and save the old ones. */ res = call_os_sigaction(sig, act, &oldAct); sact[sig] = oldAct; if (oact != NULL) { *oact = oldAct; } /* Record the signals used by jvm. */ sigaddset(&jvmsigs, sig); signal_unlock(); return res; } <snip> } Bug 1: we change state even if the sigaction call failed Bug 2: we change state even if the sigaction call was a non-modifying one (act == NULL) The latter is usually no problem since hotspot always calls `sigaction()` in pairs when installing a signal: first with NULL to get the old handler, then with the real handler. But this is not always true. If `AllowUserSignalHandlers` is set, and we find a custom handler is present, we will not override it: void set_signal_handler(int sig, bool do_check = true) { // Check for overwrite. struct sigaction oldAct; sigaction(sig, (struct sigaction*)NULL, &oldAct); <<<<< first sigaction call, libjsig now remembers signal as set // Query the current signal handler. Needs to be a separate operation // from installing a new handler since we need to honor AllowUserSignalHandlers. void* oldhand = get_signal_handler(&oldAct); if (!HANDLER_IS_IGN_OR_DFL(oldhand) && !HANDLER_IS(oldhand, javaSignalHandler)) { if (AllowUserSignalHandlers) { // Do not overwrite; user takes responsibility to forward to us. return; That means: - we still have the original custom handler in place - but we already called sigaction, albeit with NULL, but libjsig now assumes that hotspot installed a handler itself. The result is that any further attempts to change the signal handler, whether by hotspot or by user code, will be prevented by libjsig. Any further non-modifying sigaction calls will return the original - still installed - custom handler. Admittedly, the error is very exotic. Users would have to set AllowUserSignalHandlers and preload libjsig, and *then* attempt to modify signal handlers after JVM initialization. But it is confusing, and a potential source for other errors. In hotspot, nobody counts on a non-modifying sigaction query changing program state somewhere. This seems to be an old bug, I see it in at least JDK 8. Did not look further into the past --- Tests: Ran the runtime/jsig and the runtime/Thread tests manually. ------------- Commit messages: - JDK-8293466-libjsig-should-ignore-non-modifying-sigaction-calls - JDK-8293466-libjsig-should-ignore-non-modifying-sigaction-calls Changes: https://git.openjdk.org/jdk/pull/10236/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10236&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8293466 Stats: 22 lines in 1 file changed: 14 ins; 3 del; 5 mod Patch: https://git.openjdk.org/jdk/pull/10236.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10236/head:pull/10236 PR: https://git.openjdk.org/jdk/pull/10236