Hi Lina,
I have a couple of questions about your tracing setup.
On 1/2/25 5:14 PM, Lina SADI via lttng-dev wrote:
Hello,
We are writing to you regarding an issue we encountered while attempting to
trace Ceph with LTTng-UST. Below are the steps we have followed so far:
1.
*Compiling Ceph with LTTng Support*:
- We modified the debian/rules file to enable LTTng support using the
following flags: -DWITH_LTTNG=ON and
-DWITH_OSD_INSTRUMENT_FUNCTIONS=ON.
- The compilation process took approximately 7 hours on a virtual
machine running Debian 12 with a kernel version 6.1.0-27-amd64, 8 GB of
RAM, and 2 CPU cores.
- The build steps can be found in our GitHub
<https://github.com/KedadsaChakib/ceph-configuration/blob/main/build/README.md>
repository.
2.
*Deploying a Ceph Cluster*:
- We deployed the cluster manually (1MON,1MGR,2OSDs). Please find the
script here
<https://github.com/KedadsaChakib/ceph-configuration/blob/main/cluster/cluster-dep-script.sh>
.
With which user a running lttng-sessiond?
When you run `lttng list -u`, what user are you using?
Is that user a member of the `tracing` group?
If you are running lttng-sessiond with verbose logs and/or verbose
consumers - do you see the traced application(s) connecting?
3.
*Attempting to Trace with LTTng-UST*:
- We tried listing the tracepoints using the following command :
# lttng list -u
- This command did not return any Ceph-related tracepoints:
UST events:
-------------
NONE
4.
*Preloading liblttng-ust-fork.so*:
- We attempted to preload liblttng-ust-fork.so before starting the Ceph
daemons (MON, OSD, MGR) as follows:
# systemctl edit ceph-mon@debian
- We added the following configuration:
[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so
Does adding `LTTNG_UST_DEBUG=1` to the environment variables add
information to the service logs that are relevant?
- After restarting the daemons, we executed lttng list -u, which
displayed tracepoints related to the libraries (lttng_ust_lib:*),
trace logs (lttng_ust_tracelog:*), and state dump events (
lttng_ust_statedump:*), instead of the expected Ceph-related
tracepoints.
*Expected Result*:
UST events:
-------------
PID: 100859 - Name: /path/to/ceph-osd
pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
...
*Actual Result*:
UST events:
-------------
lttng_ust_lib:*
lttng_ust_tracelog:*
lttng_ust_statedump:*
We would greatly appreciate any help you could provide to resolve this problem.
thanks,
kienan
