Hey folks! We're looking into enabling the sidecar to collect async profiles from Cassandra and, digging through the async-profiler code and usage, it seems like there may be a few different ways to do it. I’m curious if other folks have already done this beyond just “run asprof with the pid of the Cassandra process”, as I’m a bit hesitant to depend on executing an external process from the Sidecar to gather the actual profile if we can avoid it.
There seem to be some opportunities to integrate the profiler into another project (see https://github.com/async-profiler/async-profiler/blob/master/docs/IntegratingAsyncProfiler.md#using-java-api) but it seems this would end up having to be part of Cassandra, and somehow callable via the sidecar (JMX? Some virtual table interface where you insert a row to start a profile with the profiler options, and it kicks off the profile, dumping the results into the table when it’s done?). The benefit in putting this functionality into Cassandra would be that other consumers (in-jvm dtests, python dtests, other monitoring systems where Sidecar isn’t available, easy-cass-lab) would be able to leverage the same interface rather than having to re-invent the wheel each time. Drawback is it’s another library, and one with native library dependencies, added to the class path and loaded at runtime. Thoughts? Previous experiences (good or bad)? Thanks, Doug
