Hi Carter,
Your mail made me pick up an old item from my wishlist: to have native
memory tracking information available in JFR recordings. When we, in GC,
do improvements to decrease the native memory overhead of our
algorithms, NMT is a very good tool to track the progress. We have
scripts that sound very similar to what you describe and more than once
I've been thinking about adding this information into JFR. But it has
not been a priority and the greater value has been unclear.
Hearing that others might also benefit from such a change I took a
discussion with the JFR team on how to best proceed with this. I have
created a branch for this and will probably create a PR for it shortly,
but I thought I would drop it here first:
https://github.com/kstefanj/jdk/tree/8157023-jfr-events-for-nmt
The change adds two new JFR events: one for the total usage and one for
the usage of each memory type. These are sent only if Native Memory
Tracking is turned on, and they are enabled in the default JFR profile
with an interval of 1s. This might change during reviewing but it was a
good starting point.
With this you will be able to use JFR streaming to access the events
from within your running process. I hope this will help your use cases
and please let us know if you have any comments or suggestions.
Thanks,
Stefan
On 2022-11-10 16:58, Carter Kozak wrote:
/+serviceability-dev/
Firstly, thank you both for your time and work in this space. Apologies
if this should be a separate thread, but the new title “Extend Native
Memory Tracking over the JDK” aligns directly with some work I’ve been
investigating, and I hope my feedback will be helpful for prioritization
of zlib observability as well as the way users think about native memory
tracking in general.
Observability of native memory in the JVM is critically important, and
becomes even more valuable as the industry shifts to more and smaller
services deployed in right-sized container environments like kubernetes.
Each new JDK release (major and hotfix) offers dramatic improvements,
often based on some form of trade-off. To be clear, I cannot overstate
how impressed I am with quality and velocity of improvement! However,
these trade-offs impact the way that memory is used, and it’s a
difficult balance to ensure containers use the correct amount of memory
without being wasteful (over-provisioned) or oomkilled (under-provisioned).
In production, I have thousands of JVMs running with native memory
tracking summary enabled. Real-time monitoring of the output is painful
and inefficient. Currently the only supported option I’m aware of is
shelling out to create a new jcmd process and parsing the NMT summary
text output periodically. In older releases, it was possible to bypass
the jcmd process by self-attaching, but that was limited in jdk9 by
JDK-8178380 <https://bugs.openjdk.org/browse/JDK-8178380>, and still
required the caller to parse human-readable strings. In fact, attachment
issues in some JDK/environment combinations make automated attachment
/dangerous/ in a way that has crashed the JVM — that may be a story for
another day, but my point is that simple, efficient NMT data collection
would go a very long way. Many modern observability tools, especially
those used in container deployments, operate by reading data from within
the jvm process, and relaying it to a storage system (Prometheus may be
the most ubiquitous example).
For my use-case, I’d love to have a simple API I could invoke from java
code to access structured native-memory-tracking data, similar in a way
to MemoryPoolMXBean for heap pools (although JMX isn’t necessary for me,
it aligns with other observability APIs in the JDK). Additionally, I’d
like to provide JFR events to periodically record native memory tracking
metadata when enabled for better out-of-the-box experience with JMC.
I’ve begun investigating some options for JDK-8182634
<https://bugs.openjdk.org/browse/JDK-8182634>, but would appreciate
feedback before I propose any sort of code change.
Thank you all for beginning this discussion, I’m eager to see the ways
the JDK continues to improve upon observability features, and do what
small part I can to help!
Carter Kozak
