Hey folks,
I'm one of the authors and maintainers of Perfetto, also +skyostil@.
I am really sorry for the giant bulk reply. I'll try to do my best to
answer the various open questions about Perfetto but I don't know a
better way than some heavy <snip>-ing given I'm joining the party late.
In short:
- Yep, so far the only distribution we have for the SDK is a C++
amalgamation. I am aware that it isn't great for Linux OSS projects, it
was very optimized for Google projects that have the habit of statically
linking everything.
- There are plans to move beyond that and have a stable C API (docs
linked below). But that will take us quite some time. We should probably
figure out some intermediate solution meanwhile.
- I'd be really keen to learn how Mesa is intending to do tracing. That
can influence a lot our upcoming design. Begin/end markers are IMO the
least interesting thing as they tend to work in whatever form and are
easy to abstract. Richer/structured trace points like
https://github.com/google/perfetto/tree/master/protos/perfetto/trace/gpu
( currently used by Android GPU drivers [1]) are more interesting and
where most of the challenges lie.
- Maybe the discussion here needs to be split into: (1) a shorter-term
plan to iterate, figure out what works, what doesn't, see how the end
result looks like; (2) a longer term plan and on how the API surface
should look like.
I don't have strong opinions on how Mesa should proceed here and you
don't need an extra cook in the kitchen. If I really had to express a
handwavy opinion, my best advice would be to start with something you
can iterate on right now, maybe behind some compile-time flag, and come
up with a plan on how to turn into a production thing later on. We are
interested to hear your feedback and adjust the design of our stable C API.
[1]
https://android-developers.googleblog.com/2020/09/android-gpu-inspector-open-beta.html?m=1
On the tracing library / C++ vendoring / stable C API:
The way the Perfetto SDK is organized today is mainly influenced-by and
designed-for the way Google handles its projects, which boils down to:
(i) statically link everything, to minimize the testing matrix; (ii)
move fast and refactor all dependencies as needed.
It's all about "who pays the maintenance cost and when?". This tends to
work well in a large company which: (i) has a giant repo which allows
~atomic cross-project changes; (ii) has the resources to keep everything
up to date.
I am perfectly aware this is not appealing nor ideal for open source
projects and, more in general, with the way libraries in Linux
distributions work. I hear you when you say "vendoring [...] seems a bad
idea". Yes, it implicitly pushes the burden of up-revving onto the
"depender" [that's bad]
We are committed to maintain ABI stability of our tracing protocol and
socket (see https://perfetto.dev/docs/design-docs/api-and-abi). This is
because Chrome, Android, and now CrOS, and tools like gpuinspector.dev,
which all statically link perfetto in some form, have strongly different
release cycles. [that's good]
We also try to not break the C++ API too much, as robdclark@ found
trying to update through our v3..v12 monthly releases [that's good]. But
that C++ API has a too wide surface and we can't commit to make that a
fully stable API. Nor can we make the current C++ ABI stable across a
.so boundary (the C++ SDK today heavily depends on inlines to allow the
compiler to see through the layers). [that's bad]
For this reason, we recently started making plans to change the way we
do things to meet the needs of open source projects and not just
Google-internal ones. [that's good]
Specifically (Note: to open the docs below you need to join
https://groups.google.com/forum/#!forum/perfetto-dev to inherit the ACLs):
1. https://bit.ly/perfetto-debian has a plan to distribute tracing
services and SDK as standard pkg-config based packages (at least for
Debian. We'll rely on volunteers for other distros)
2. https://bit.ly/perfetto-c has a plan + ongoing discussion for having
a long-term stable C API in a libperfetto.so . The key here for us
(Perfetto) is identifying a subset of the wider C++ API that fits the
bill for projects out there and that we are comfortable maintaining
longer term.
The one thing I also need to be very clear on, though, is that both the
perfetto-debian and perfetto-c discussions are very recent and will take
a while for us to get there. We can't commit to a specific timeline
right now, but if I had to make an educated estimate I'd say more
towards end-of-2021. [that's bad]
I'd also be more keen to commit once there are concrete use-cases,
ideally with iterations/feedback from a project like Mesa.
[obligatory reference at this point: https://youtu.be/Krbl911ZPBA?t=22]
> eero.t.tamminen@
> Just set uprobe for suitable buffer swap function [1], and parse
kernel ftrace events. (paraphrasing for context: "why do we need
instrumentation points? we can't we just use uprobes instead?")
The problem with uprobes is:
1. It's linux specific. Perhaps not a big problem for Mesa here, but the
reason why we didn't go there with Perfetto, at least until now, is that
we need to support all major OSes (Linux, CrOS, Android, Windows, macOS).
2. Even on Linux-based systems, it's really hard to have uprobes enabled
in production (I am not sure what is the situation for CrOS). In Google,
we care a lot about being able to trace from production devices without
reflashing them with dev images, because then we can just tell people
that are experiencing problems "can you just open chrome://tracing,
click on a button and give us debugging data?". Re-flashing reduces by
orders of magnitude the actionable feedback we'd be able to get from
users / testers.
The challenge of ubprobes is that it relies on dynamic rewriting of
.text pages. Whenever I mention that, a platform security team reacts
like the Frau Blucher horses (https://youtu.be/bps5hJ5DQDw?t=10), with
understandable reasons.
> eero.t.tamminen@
> Perfetto appears to be for Android / Chrome(OS?), and not available
from in common Linux distro repos.
That's right. I am aware of the problem. The plan is to address it with
bit.ly/perfetto-debian as a starter.
> mark.a.janes@
> Perfetto seems like an awful lot of infrastructure to capture trace
events. Why not follow the example of GPUVis, and write generic
trace_markers to ftrace?
In my experience ftrace's trace_marker:
1. Works for very simple types of events (e.g.
begin-slice/end-slice/counters) but don't work with richer / structured
event types like the ones linked above, as that gets into
stringification format, mashaling costs and interop.
2. Writing into the marker has some non-trivial cost (IIRC 1-10 us on
Android), it involves a kernel into and back from the kernel;
3. It leads to one global ring buffer, where fast events push out slower
ones, which is particularly problematic if you ever enable sched/*
events. At the userspace level, instead, we can dynamically route
different events to different buffers to mitigate this problem (see
https://perfetto.dev/docs/concepts/config#dynamic-buffer-mapping)
> dylan@
> But maybe we should verify that MSVC is happy with it
FYI I have some work in progress to fully support the tracing protocol
on Windows. With https://r.android.com/1539396, the tracing services and
the tracing SDK build and run with both clang-cl and MSVC 2019. We are
figuring out some final details (e.g., whether to use AF_UNIX and
support only Win10+, or use a TCP socket)
> robdclark@
> AFAIU, perfetto can operate in self-server mode, which seems like it
would be useful for distros which do not have the system daemon.
That's correct. Which is also a reason why the amalgamated source is
that large, as it also contains all the bits to run the service
in-process. (If you link with -Wl,--gc-sections or equivalent, they will
be stripped away though)
See
https://perfetto.dev/docs/instrumentation/tracing-sdk#in-process-vs-system-mode
>mark.a.janes@
>Is it possible to build and run the Perfetto UI locally?
Yes, the UI is fully open source and fully client-only. It can be built
from the same repo by just running `tools/install-build-deps --ui` +
`./ui/build --serve` . See
https://perfetto.dev/docs/contributing/build-instructions#ui-development
> Can it display arbitrary trace events that are written to
/sys/kernel/tracing/trace_marker
Today the only thing that the UI displays, w.r.t trace_marker, are
events that match the syntax that Android's systrace came up with back
then. See
https://cs.android.com/android/platform/superproject/+/master:system/core/libcutils/trace-dev.cpp,
it boils down to a format like "B|$TGID|event_name" to begin a slice,
"E|..." to end it and so on.
Curious to hear about other uses of the trace marker and consider
importing other formats.
> Can it be extended to show i915 and i915-perf-recorder events?
It depends on what you want to extend:
1. If you want to extend the import logic and map custom events to
existing UI concepts, you just need to touch the C++ code in
//src/trace_processor (that code runs in the Web UI on the client via
WebAssembly). Good starting examples are:
i. the code that imports ninja-build logs, which is a completely custom
format, unrelated with perfetto's proto format or ftrace:
https://cs.android.com/android/platform/superproject/+/master:external/perfetto/src/trace_processor/importers/ninja/ninja_log_parser.cc
ii. the code that deals with way android's usage of the trace_marker
(those "B|<tgid>|name" mentioned above):
https://cs.android.com/android/platform/superproject/+/master:external/perfetto/src/trace_processor/importers/systrace/
We are generally happy to accept patches for custom importers, as long
as they have some testing (an input trace and expected output from
queries) so we can tell if we break it while refactoring our internal code.
2. If you want to extend the UI logic with custom widgets
This is the part that is tough today and requires a lot of insider
knowledge. All the code is there, but today is not really architected in
a contributor-friendly way. We have a longer term plan to allow
customization of the UI with some extension API on the JS/TS layer, but
it's still early days for that and getting there will take longer.
>dylan@
> especially if the ui code stops working with our forked version
1. You can always build a blessed version of the UI and host it on any
HTTP server of your choice. Our /ui/build generates a fully static
HTML+Js+Wasm site. It doesn't need any server-side magic. It doesn't
have dependencies on Google infra. Even just `python -m http.server`
will do.
2. We are changing the way our UI deployment works (this month) and will
always leave the old versions around. This means that, while we will
keep up-revving ~monthly the main UI instance @ https://ui.perfetto.dev,
we will allow people to go back to older versions via a link like
https://ui.perfetto.dev/v1.2.3/. You can see this in action on
https://testing-dot-perfetto-ui.wl.r.appspot.com/v12.1.172/ which I just
pushed last week to test this new deployment mechanism.
Happy to talk more about all this, either here on the list or on any
other medium.
If you have some more perfetto-related questions/comment/criticism feel
free to drop by our discord channel https://discord.gg/35ShE3A or ML
(https://groups.google.com/g/perfetto-dev).
_______________________________________________
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
