Hi Damien,
On 5/15/24 6:24 PM, Damien Berget via lttng-dev wrote:
Good day,
we have been using LTTng successfully to capture snapshots on user
defined tracepoints and it did provide invaluable to debug our issues.
Thanks to all the contributors of this project!
We'd like to know if it would be possible to trigger on a kernel panic?
I might be dubiously possible as you would still need to have the
file-system working to write the results but I should ask.
For userspace tracing, I think the recommendation is usually to use a
dax/pmem device and have the buffers for the session mapped there. After
a panic, the contents of the buffers can be restored using lttng-crash[1].
Note that dax/pem isn't supported by the kernel space tracer at this time.
If I recall, there are other ways to things in the panic sequence (that
aren't lttng specific), but I'm personally not as familiar with the
details of that stage of linux.
Looking at available kernel syscall, the "reboot" one seems like a good
candidate, however I was not able to capture a snapshot on it. I have
tested the setup below with "--name=chdir" syscall and it works, "cd" to
a directory will create a trace. But no dice with reboot.
The details of how this work will depend on your system. For example, my
installations tend to use systemd as PID 1. The broad strokes seem to
be: `/usr/sbin/reboot` is actually a link to `systemctl`, which I
believe then kicks off the reboot.service, the PID 1 is swapped to
/usr/lib/systemd/systemd-shutdown, sigterm then sigkill are sent to all
processes, unmounts, syncs, calls the reboot system call [2,3].
As both the sigterm and the unmounts are done before the syscall,
lttng-sessiond and the consumers will have already shutdown by the time
it enters.
While this doesn't necessarily help your original question of panics, if
you want to snapshot before shutdown or reboot and are using systemd,
it's possible to leave a script or binary in a known directory so that
it's invoked prior to the rest of the shutdown sequence[4].
[1]: https://lttng.org/docs/v2.13/#doc-persistent-memory-file-systems
[2]:
https://github.com/systemd/systemd/blob/6533c14997700f74e9ea42121303fc1f5c63e62b/src/shutdown/shutdown.c
[3]:
https://github.com/systemd/systemd/blob/main/src/shared/reboot-util.c#L77
[4]: https://www.systutorials.com/docs/linux/man/8-systemd-reboot/
hope this helps,
kienan
Would you have any suggestions?
Thanks for your help,
Cheers
Damien
============================
# Prep output dir
mkdir /application/trace/
rm -rf /application/trace/*
# Create session
sudo lttng destroy snapshot-trace-session
sudo lttng create snapshot-trace-session --snapshot
--output="/application/trace/"
sudo lttng enable-channel --kernel --num-subbuf=8 channelk
sudo lttng enable-channel --userspace --num-subbuf=8 channelu
# Configure session
sudo lttng enable-event --kernel --syscall --all --channel channelk
sudo lttng enable-event --kernel --tracepoint "sched*" --channel channelk
sudo lttng enable-event --userspace --all --channel channelu
sudo lttng add-context -u -t vtid -t procname
sudo lttng remove-trigger trig_reboot
sudo lttng add-trigger --name=trig_reboot \
--condition=event-rule-matches --type=kernel:syscall:entry \
--name=reboot\
--action=snapshot-session snapshot-trace-session \
--rate-policy=once-after:1
# start & list info
sudo lttng start
sudo lttng list snapshot-trace-session
sudo lttng list-triggers
#======== test it...
sudo reboot
#======= reconnect and Nothing :(
$ ls -alu /application/trace/
drwxr-xr-x 2 u u 4096 May 15 2024 .
drwxr-xr-x 10 u u 4096 May 15 2024 ..
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
