> I have seen cases where specific async-profiler/JVM/Cassandra version combos
> (JDK11/4.1-derived source tree) will immediately crash the JVM on profile -
> especially successive profile invocations on the same process
This would be a great candidate for testing to ensure that, at least for
provided profiles, this doesn't happen.
On Fri, Jun 13, 2025, at 10:41 PM, C. Scott Andreas wrote:
> Supportive of inclusion as well. General preference for invoking as a library
> rather than forking processes.
>
> Jon, thanks for the tips on off-CPU profiling - added to my personal cheat
> sheet.
>
> I have seen cases where specific async-profiler/JVM/Cassandra version combos
> (JDK11/4.1-derived source tree) will immediately crash the JVM on profile -
> especially successive profile invocations on the same process - but have not
> observed this on JDK21 or trunk-derived source trees. If we have user reports
> of that happening, we’ll need to figure out how to reproduce and get to the
> bottom of it.
>
> – Scott
>
> > On Jun 13, 2025, at 5:24 PM, Francisco Guerrero <fran...@apache.org> wrote:
> >
> > Thanks for bringing this discussion Doug. I didn't realize that
> > async-profiler allows you to
> > bring it as a dependency. It looks pretty neat from what I could tell. I
> > also think bringing
> > this to Cassandra as a dependency is a reasonable approach. We need to come
> > up with
> > a solid way to expose this via JMX / vtable.
> >
> > Best,
> > - Francisco
> >
> >> On 2025/06/13 21:08:28 Doug Rohrer wrote:
> >> The nice thing from what I can tell about using the Java API per [6] below
> >> is that you can literally just get an instance of the profiler and pass it
> >> some commands in the `execute` method… just need to be careful how much of
> >> that surface area we expose. Jon (and others obviously) I’d love to get
> >> your take on how we could make a useful interface to the async-profiler,
> >> maybe exposed via JMX, that doesn’t require someone to read the entirety
> >> of the async-profiler docs and provides some useful profiles without the
> >> rough edges (things like managing temp files so users don’t have to know
> >> the layout of the filesystem C* is running on, for example, since at least
> >> in the Sidecar we’d be executing this on behalf of a remote user, with all
> >> of the constraints that implies).
> >>
> >> We can always be more protective in the Sidecar than we are server-side as
> >> well, but it seems like helping operators not do bad things is a good
> >> thing.
> >>
> >> Obviously we’d want the ability Cassandra-side to disable this
> >> functionality all together however we implement it.
> >>
> >> Doug
> >>
> >>>> On Jun 13, 2025, at 2:38 PM, Jon Haddad <j...@rustyrazorblade.com> wrote:
> >>>
> >>> I'd be very happy to see async-profiler included with C* I've made
> >>> extensive use of it in my performance evaluations [1][2], and even posted
> >>> a video about it [3] for general Java perf analysis (among others). It's
> >>> part of easy-cass-lab and is easily the most informative tool I've found
> >>> for the getting to the bottom of anything performance related.
> >>>
> >>> There's probably a good case to be made for including it with the C*
> >>> artifact as well as having it be something you can drop in. I lean
> >>> towards including it all the time, but I haven't run it this way myself
> >>> yet, so there might be some downside I'm unaware of.
> >>>
> >>> When you call the asprof executable, it attaches the async-profiler to
> >>> the running jvm using jattach [4]. We could do this as well, if we
> >>> wanted to avoid including it with the release, but I don't know how much
> >>> we really benefit from that. I've run into issues with it when it's
> >>> unable to detatch correctly, then you're unable to reattach it until
> >>> after the server is restarted. On the flip side, I don't know if you're
> >>> able to set up all the same options for arbitrary profiling when it's
> >>> loaded as an agent and turned on/off dynamically. I think we can, based
> >>> on the integration page [6], but I haven't tried it yet. It would be a
> >>> bummer if we only had a single mode of profiling available.
> >>>
> >>> The default mode, CPU profiling, is fantastic, but I've also made
> >>> extensive use of allocation profiling [5] to identify perf issues as well
> >>> so having that available is a must, imo. Wall clock / off cpu profiling
> >>> is great for identifying when IO is the root cause, which isn't clearly
> >>> revealed by on-cpu profiling due to the way threads are scheduled. When
> >>> I look at a system I typically do CPU / Wall / Alloc / Off-CPU to be
> >>> thorough, and the last thing you want to do is have to restart between
> >>> each one. You can also specify specific Java methods, include or exclude
> >>> frames matching specific regex, and a whole slew of other options. The
> >>> latest version even supports continuous profiling with heatmaps although
> >>> I haven't tried it yet.
> >>>
> >>> So hopefully the option we go with allows all of that, otherwise the
> >>> limits would impose more of a headache to me as I'd need to remove it and
> >>> continue to bring my own.
> >>>
> >>> Under the hood, the async-profiler uses Linux perf events + <>
> >>> asynchronous polling of the java stack to match them up and generate it's
> >>> reports. As a result, it requires certain permissions to run and get all
> >>> the details I like. Specifically these kernel parameters:
> >>>
> >>> sudo sysctl kernel.perf_event_paranoid=1
> >>> sudo sysctl kernel.kptr_restrict=0
> >>>
> >>> You also need to enable some capabilities for off-cpu profiliing:
> >>>
> >>> sudo find /usr/lib/jvm/ -type f -name 'java' -exec setcap
> >>> "cap_perfmon,cap_sys_ptrace,cap_syslog=ep" {} \;
> >>>
> >>> Then you can do off-cpu with this wild cryptic version (shout out to
> >>> Andrei Pangin for helping me with this [7]):
> >>>
> >>> asprof -e kprobe:schedule -i 2 --cstack dwarf -X '*Unsafe.park*' "${@:2}"
> >>> $PID
> >>>
> >>> There's also some subtle issues when it's run in a container, since by
> >>> default you don't have access to the perf_event_open syscall. Just
> >>> something to keep in mind. This is one of my main grievances with
> >>> container deployments.
> >>>
> >>> Indeed Patrick, I am very happy to see this discussion! Thanks Doug for
> >>> starting the thread.
> >>>
> >>> Jon
> >>>
> >>> [1] https://issues.apache.org/jira/browse/CASSANDRA-15452
> >>> [2] https://issues.apache.org/jira/browse/CASSANDRA-19477
> >>> [3]
> >>> https://www.youtube.com/watch?v=yNZtnzjyJRI&t=212s&pp=ygUOYXN5bmMgcHJvZmlsZXI%3D
> >>> [4]
> >>> https://github.com/async-profiler/async-profiler/blob/2b556680dc8f5d02c3f26ac119d835dc2381e604/src/jattach/jattach_hotspot.c#L38
> >>> [5] https://issues.apache.org/jira/browse/CASSANDRA-20428
> >>> [6]
> >>> https://github.com/async-profiler/async-profiler/blob/master/docs/IntegratingAsyncProfiler.md
> >>> [7] https://github.com/async-profiler/async-profiler/issues/907
> >>>
> >>>
> >>> On Fri, Jun 13, 2025 at 10:18 AM Patrick McFadin <pmcfa...@gmail.com
> >>> <mailto:pmcfa...@gmail.com>> wrote:
> >>>> The fact o3 used "Bus-factor" as a dimension is just amazing.
> >>>>
> >>>> After reading more about the project, the possibilities are pretty
> >>>> interesting. I suspect we'll see this in a Haddad talk soon.
> >>>>
> >>>> On Fri, Jun 13, 2025 at 1:57 AM Josh McKenzie <jmcken...@apache.org
> >>>> <mailto:jmcken...@apache.org>> wrote:
> >>>>> I was curious if o3 (model from OpenAI) would be able to do a deep dive
> >>>>> health check on a repo to assist in considering taking it as a
> >>>>> dependency. The results can be found here:
> >>>>> https://chatgpt.com/share/684be703-1d4c-8002-b831-f997f829f4b4
> >>>>>
> >>>>> Apparently it can, and can do it quite well. This was a useful time
> >>>>> saver (and honestly did a better job than I usually can in > 10x the
> >>>>> time)
> >>>>>
> >>>>> I'm +1 to taking this as a dependency on the lib in core C*. The rest
> >>>>> of the ecosystem can consume it (more easily if we move to a
> >>>>> cassandra-shared regime shared library build as well), and it opens up
> >>>>> some interesting opportunities for us in both how we test core C*
> >>>>> proper and what we expose in tooling.
> >>>>>
> >>>>> On Thu, Jun 12, 2025, at 7:36 PM, Paulo Motta wrote:
> >>>>>> I'd prefer to avoid calling an external process and use the library if
> >>>>>> possible. Not sure about including it in the project by default, but
> >>>>>> also not against.
> >>>>>>
> >>>>>> If there's contention about including it, I wonder if it would make
> >>>>>> sense to explore java's optional module extension[1] to make this
> >>>>>> available optionally ? I can see this being useful for other
> >>>>>> extensions if we haven't explored that option.
> >>>>>>
> >>>>>> Then we could have another project cassandra-sidecar-extensions (or
> >>>>>> similar) that would be linked by sidecar/advanced operators to enable
> >>>>>> extended featureset in the main process.
> >>>>>>
> >>>>>>
> >>>>>> [1] -
> >>>>>> https://openjdk.org/projects/jigsaw/doc/topics/optional.html
> >>>>>>
> >>>>>> On Thu, 12 Jun 2025 at 17:57 Doug Rohrer <droh...@apple.com
> >>>>>> <mailto:droh...@apple.com>> wrote:
> >>>>>> Hey folks!
> >>>>>>
> >>>>>> We're looking into enabling the sidecar to collect async profiles from
> >>>>>> Cassandra and, digging through the async-profiler code and usage, it
> >>>>>> seems like there may be a few different ways to do it. I’m curious if
> >>>>>> other folks have already done this beyond just “run asprof with the
> >>>>>> pid of the Cassandra process”, as I’m a bit hesitant to depend on
> >>>>>> executing an external process from the Sidecar to gather the actual
> >>>>>> profile if we can avoid it.
> >>>>>>
> >>>>>> There seem to be some opportunities to integrate the profiler into
> >>>>>> another project (see
> >>>>>> https://github.com/async-profiler/async-profiler/blob/master/docs/IntegratingAsyncProfiler.md#using-java-api)
> >>>>>> but it seems this would end up having to be part of Cassandra, and
> >>>>>> somehow callable via the sidecar (JMX? Some virtual table interface
> >>>>>> where you insert a row to start a profile with the profiler options,
> >>>>>> and it kicks off the profile, dumping the results into the table when
> >>>>>> it’s done?).
> >>>>>>
> >>>>>> The benefit in putting this functionality into Cassandra would be that
> >>>>>> other consumers (in-jvm dtests, python dtests, other monitoring
> >>>>>> systems where Sidecar isn’t available, easy-cass-lab) would be able to
> >>>>>> leverage the same interface rather than having to re-invent the wheel
> >>>>>> each time.
> >>>>>>
> >>>>>> Drawback is it’s another library, and one with native library
> >>>>>> dependencies, added to the class path and loaded at runtime.
> >>>>>>
> >>>>>> Thoughts? Previous experiences (good or bad)?
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Doug
> >>>>>
> >>
> >>
>
