Tomas,
Every time I set the time interval to a value of 1e-5 or smaller (I
think! maybe it was 1e-6 or smaller) , R will crash on my machine.
On 11/10/22 4:53 AM, Tomas Kalibera wrote:
On 11/9/22 00:22, Simon Urbanek wrote:
On Nov 9, 2022, at 10:03 AM, Tomas Kalibera
<tomas.kalib...@gmail.com> wrote:
On 11/7/22 01:58, luke-tier...@uiowa.edu wrote:
On Sun, 6 Nov 2022, Simon Urbanek wrote:
Carl,
first, setting such low interval won't work anyway - the overhead
is bigger than the sampled time, so we should really not allow it
to begin with (on my machine the timer signals arrive before
anything can be done so you have to kill R and you get no output).
That said, it crashes in doprof() which is called on all threads -
the main R one is ok, but one of the other threads crashes in
pthread_self(). At that time R is trying to propagate the signal
from all threads to the main thread which seems odd to me (since
the main thread already got the signal), I'm CCing Luke in the hope
that he has any ideas. This may fall in the category of "don't do
this" and the fix may be to set a lower bound on the interval.
I can't reproduce this on Linux or macOS.
On Linux only one thread receives a signal sent to a process, but the
kernel picks which one if multiple threads have the signal unblocked,
so we make sure the signal gets relayed to the main thread. If macOS
behaves differently then someone who knows how signals and threads
interact there would have to adjust this code.
From my reading this is the same on macOS. The profiling signal is
asynchronous, sent to the process, it will be served by one thread
which is picked by the OS. POSIX doesn't say which thread is preferred.
Yes, I saw the same with extra detail that thread signal blocking
doesn't seem to necessarily work on macOS.
While some OSes prefer the main thread (I read macOS and Linux do,
but from non-authoritative sources), R may also be embedded and not
run on the main thread.
We have to do something to ensure the R thread is not running while
we sample its R stack, anyway. On Windows we suspend the R thread for
that. On Unix we do the relaying. We could in principle suspend the
R thread on macOS as well, but would have to use Mach calls directly.
Disallowing such a low interval is reasonable, but if there is a real
issue on macOS then it would only mask the problem.
Yes. The key question is why pthread_self() crashed.
Yes, that is the main mystery. Looking at the xnu kernel sources it is
equivalent to pthread_getspecific(0) [since it's just the first slot
in TSD] plus a check of a magic content in there. I suspect it's that
check which segfaults for whatever reason. I wanted to see if just
comparing the pointer from pthread_getspecific(0) instead of
pthread_self() would work since we don't care if the pthread_t is
valid as we only compare it to the main thread value (not that I would
propose that as a fix since it's very implementation-specific, just
curious), but I didn't get that far (I cannot really reproduce it -
the closest I get is a mach exception under lldb).
Yes, this is a mystery. The pthread_t validation may probably crash if
pthread_t was corrupted, but, it is not clear why it should be. Then
there is the pointer authentication check which I wonder if does
anything at all on Intel, and the report was from an Intel machine.
What I also find puzzling is that the stack trace doesn't show much
about the crashed thread. The 1st frame on thread 0 is "start" as it is
the main thread. The other threads start with
"thread_start/_pthread_start". But, the crashed thread 6 only with
"_sigtramp" for the handler. No previous frames. Also, the crash has is
due to "no mapping for user data read", a page fault, so probably some
pointer on the stack points to the wrong place. As if the stack was
corrupted or the thread didn't get a chance to be initialized properly
before the signal has arrived (not sure if that is possible).
Carl, is the problem repeatable on your machine? If yes, what are the
steps to repeat it on your machine?
I was trying on M1, but didn't find a way to provoke it.
Best
Tomas
Otherwise, from the stack trace, the behavior looks ok. The main
thread (also R thread) is serving the signal, hence the signal is
blocked, but it is received again, so another thread is picked to
serve it, and it is relaying it to the main thread. One more thread
is picked to serve it, and it crashes while calling pthread_self().
There is also one more thread not involved in the signal handling.
POSIX statest that pthread_self() is async-signal-safe. macOS 12.6
manuals (sigaction) however doesn't include any pthread function in
the list of async-signal-functions.
We could do some work-around (hiding the problem a bit more) like
exit from the handler if the signal is being served by another
thread. We could also report such situation to indicate that the
interval is unreasonable. But it would be good first to know for sure
what caused the problem.
How can you check anything if pthread functions fail? If a simple
pthead_self() crashes then I don't see how you can do anything since
we don't even know what thread we are, cannot call mutexes etc.
Cheers,
Simon
--
Carl Witthoft
personal: c...@witthoft.com
The Witthoft Group, Consulting
https://witthoftgroup.weebly.com/
_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac
