On 6/21/23 01:39, Yitschak, Yehuda wrote:
On 6/20/23 10:20, Mathieu Desnoyers via lttng-dev wrote:
On 6/20/23 06:27, Mousa, Anas via lttng-dev wrote:
Hello,
Arethereanysuggestionstorootcausethehighlatencyandpotentiallyimproveito
n*platform****1*?
Thanks and best regards,
Anas.
I recommend using "perf" when tracing with the sample program in a
loop to figure out the hot spots. With that information on the "fast"
and "slow" system, we might be able to figure out what differs.
Also, comparing the kernel configurations of the two systems can help.
Also comparing the glibc versions of the two systems would be relevant.
Also make sure you benchmark the lttng "snapshot" mode [1] to make sure
you don't run into a situation where the disk/network I/O throughput cannot
cope with the generated event throughput, thus causing the ring buffer to
discard events. This would therefore "speed up" tracing from the application
perspective because discarding an event is faster than writing it to a ring
buffer.
You mean we should avoid the "discard" loss mode and use "overwrite" loss mode
since discard mode can fake fast performance ?
Yes. In addition to use "overwrite-when-buffer-full" mode, the
"snapshot" session also ensures that no consumer daemon extracts the
trace data (unless an explicit snapshot record is performed), which
allows comparing the ring buffer producer performance with minimal noise.
If you really want to benchmark the discard-when-buffer-full mode and
the the consumer daemon I/O behavior, then you need to take into account
event discarded counts and the actual trace data size that was written
to disk.
Thanks,
Mathieu
Thanks,
Mathieu
[1] https://lttng.org/docs/v2.13/#doc-taking-a-snapshot
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
