On Wed, May 23, 2018 at 5:17 AM, Christoph Berg <m...@debian.org> wrote: > I'm not aware of any guidelines - I guess this stuff is still too new > to make general statements about it. Which other software offers > custom probes?
As the LWN article mentions, user space probes are available for a number of other widely used open source software projects, including MySQL and even glibc. I'm pretty sure that these are not new; they were added for DTrace quite a few years back, and then perhaps forgotten about. I'm pretty sure that these are generally not enabled on standard Linux packages. I bet they're enabled on packages for Solaris/illumos, though, since DTrace is considered an important part of the Solaris ecosystem. In recent years, the Linux kernel gained the ability to expose probes that can be used in much the same way as the --enable-dtrace user-space probes, as well as the ability to run a custom eBPF JIT-compiled program in kernel space, to do the actual instrumentation. BCC is an externally maintained toolkit that is used to write most eBPF-based instrumentation in practice, since it makes the process far easier. BCC isn't a separate technology to eBPF, though. BCC/eBPF make the old DTrace/SystemTap probes much more useful on Linux for a number of reasons. SystemTap had problems with reliability in the past, owing to the fact that it builds kernel modules rather than building eBPF code that runs in the eBPF in-kernel virtual machine (eBPF code is generally considered "safe", which is a huge advantage). Also, eBPF has less overhead for certain types of aggregation which are apparently quite important, and benefits from being JIT compiled (there is an LLVM backend, which is what BCC uses). Finally, eBPF looks like it might emerge as a kind of defacto standard on Linux, which SystemTap clearly never managed to do. The ecosystem questions clearly aren't worked out yet, but there is a lot of investment in and momentum behind eBPF. There is good reason to expect that these teething issues be worked out. Linux should be able to do this stuff without the user having to read a tedious explanation of how the different parts fit together. -- Peter Geoghegan
