Are there any opinions about whether or not to extend NMT across the JDK? This blocks https://bugs.openjdk.org/browse/JDK-8296360, and I had a PR prepared as https://github.com/openjdk/jdk/pull/10988. Originally I was hoping to get this into JDK 20, but I don't think that is realistic anymore. I am fine with postponing my work in favor of a baseline discussion, but so far there is very little discussion about this topic.
How should I proceed? Thanks, Thomas On Wed, Nov 9, 2022 at 8:12 AM Thomas Stüfe <thomas.stu...@gmail.com> wrote: > Hi Alan, > > (replaced hotspot-runtime-dev with hotspot-dev, since its more of a > general topic) > > thank you for your time! > > I am very happy to talk this through. I think native memory observability > in the JDK (and customer code!) is sorely lacking. Witness the countless > "where did my native memory go" blog articles. At SAP we have been > struggling with this topic for a long time and have come up with a mixture > of solutions. The aforementioned tracker was one, which extended our > version of NMT across the JDK. Our SapMachine MallocTracer, which allows us > to trace uninstrumented customer code, another. We even experimented with > exchanging the allocator (using jemalloc) to gain insights. But that is a > whole different topic with deep logistical implications, I don't want to > touch it here. Exchanging the allocator does not help to observe virtual > memory or the brk segment, of course. > > And to make the picture complete, another insight we currently lack is the > implicit allocator overhead, which can be very significant and is hidden by > the libc. We also have observability for that in the SapMachine, and I miss > it in OpenJDK. > > As you noticed, my original intent was just to instrument Zlib and > possibly improve tracking for DBBs. Although, thinking beyond that, another > attractive instrumentation target would be mapped NIO buffers at least. > > So I think native memory observability is important. Arguably we could > even extend observability to cover other OS resources, e.g. file handles. > If we shift code around, to java/Panama: data that move the java heap does > not need to be tracked, but other memory will always come from one of the > basic system APIs, regardless of who allocates it and where in the stack > allocation happens. Be it native JDK code, Panama, or even customer JNI > code. > > If we agree on the importance of native memory observability, then I > believe NMT is the right tool for it. It is a good tool. The machinery is > already there. It covers both C-heap and virtual memory APIs, as well as > thread stacks, and could easily be extended to cover sbrk if needed. And I > assume that whatever shape OpenJDK takes on in the future, there always > will be a libjvm.so at its core, so we will always have it. But even if > not, NMT could be separated from libjvm.so quite easily, since it has no > deep ties with the JVM. > > About coupling JVM with outside code: We don't have to directly link > against libjvm.so. We can keep things loose if the intent is to be runnable > without a JVM, or be JVM-version-agnostic. That could take the form of a > function-pointer interface like JVMTI. Or outside code could dynamically > dlsym the JVM allocation hooks. In any case gracefully falling back to > system allocation routines when necessary. > > And I agree, polluting the NMT tag space with outside meaning is ugly. I > only did it because I planned to go no further than instrumenting Zlib and > possibly DBBs. But if we take this further, my preferred solution would be > a reserved tag range or -ranges for outside use, whose inner meaning would > be opaque to the JVM. Kind of like SIGRTMIN+SIGRTMAX. Then, outside code > could register tags and their meta information with the JVM, or we find a > different way to convey the tag meaning to NMT (config files, or > callbacks). That could even be opened up for customer use. > > This also touches on another question, that of NMT tag space. NMT tags are > very useful since they allow cheap tracking without capturing call stacks. > However, tags are underused and show growing pains since they are too > one-dimensional and restrictive. We had competing interests in the past > about tag granularity. It is all over the place. We have coarse-grained > tags like "mtThread", and very fine-grained ones like "mtObjectMonitor". > There are several ways we could improve, e.g., by making them combinable > like UL does, or allowing for a hierarchy of them - either a hard-wired > limited one like "domain"+"tag", or an unlimited tree-like one. Technically > interesting since whatever the new encoding is, they still must fit into a > malloc header. I opened https://bugs.openjdk.org/browse/JDK-8281819 to > track ideas like these. > > Instrumenting Panama allocations, including the ability to tag > allocations, would be a very good idea. For instance, if we ever remove the > native Zlib layer and convert it to java using Panama, we can do the same > with Panama I do now natively - use the Zlib zalloc interface to hook in > JVM memory allocation functions. The result could be completely identical, > and the end user looking at the NMT output need never know that anything > changed. > > And that goes for all instrumentation - if today we add it to JNI code, > and that code gets removed tomorrow, we can add it to Panama code too. > Unless data structures move to the heap, in which case there is no need to > track them. > > You mentioned that NMT was more of an in-house support tool. Our > experience is different. Even though it was positioned as a tool for JVM > developers, and we never cared for the backward compatibility or > consistency, it gets used a *lot* by our customers. We have to explain its > output frequently. Also, many blog articles exist documenting its use. So, > maybe it would be okay to elevate it to a user-facing tool since it seems > to occupy that role anyway. We may also open up consumption of NMT results > via java APIs, or expose its results via MXBeans. > > If this is to be a JEP, okay, but I'm afraid it would stall things a bit. > I am interested in getting a simpler and quicker solution for older support > releases at least, possibly based on my PR. I know that would be > unconventional though. > > Thank you, > > Thomas > > > On Sun, Nov 6, 2022 at 9:31 AM Alan Bateman <alan.bate...@oracle.com> > wrote: > >> On 04/11/2022 16:54, Thomas Stüfe wrote: >> > Hi all, >> > >> > I am currently working on https://bugs.openjdk.org/browse/JDK-8296360; >> > I was preparing the final PR [1], but then Alan did ask me to discuss >> > this on core-libs first. >> > >> > Backstory: >> > >> > NMT tracks hotspot native allocations but does not cover the JDK >> > libraries (small exception: Unsafe.AllocateMemory). However, the >> > native memory footprint of JDK libraries can be significant. We have >> > no in-VM tracker for these and need tools like valgrind or our >> > SapMachine MallocTracer [2] to observe them. >> >> Thanks for starting a discussion on this as this is a topic that >> requires agreement from several areas. If this is the start of something >> bigger, where you want to have all allocation sites in the libraries >> using NMT, then I think it needs a write-up, maybe a JEP. >> >> For starters, I think it needs some agreement on using NMT for memory >> allocated outside of libjvm. You mentioned Unsafe as an exception but >> that is implemented in the VM so you get tracking for free, albeit I >> think all allocations are in the "mtOther" category. >> >> A general concern is that it creates more coupling between the VM code >> and the libraries code. As you probably know, we've removed most of the >> dependences on JVM_* functions from non-core areas over many years. So I >> think that needs consideration as I assume we don't want >> memory/allocation.hpp declaring a dozen catagories for allocations done >> in say java.desktop module for example. Maybe your proposal will be >> strictly limited to java.base but even then, do we really want the VM >> even knowing about categories that are specific to zip compression or >> decompression? >> >> There are probably longer term trends that should be part of the >> discussion too. One general trend is that "run time" is becoming more >> and more a hybrid of code in libvm and the Java libraries. Lambdas, >> module system, virtual threads implementations are a few examples in the >> last few release. This comes with many "Java on Java" challenges, >> including serviceability where users of the platform will expect tools >> to just work and won't care where the code is. NMT is probably more for >> support teams and not something that most developers will ever use but I >> think is part of the challenge of having serviceability solutions "just >> work". >> >> In addition to having more of the Java runtime written in Java, there >> will likely be less JNI code in the future. It's very possible that the >> JNI code (including the JNI methods in libzip) will be replaced with >> code that uses Panama memory and linker APIs once they are become >> permanent. The effect of that would to have a lot of the memory >> allocations be tracked in the mtOther category again. Maybe integration >> with memory tracking should be looked at in conjunction with these APIs >> and this migration. I could imagine the proposed "Arena" API >> (MemorySession in Java 19) having some integration with NMT and it might >> be interesting to look into that. >> >> So yes, this topic does need broader discussion and it might be a bit >> premature to start with a PR for libzip without talking about the bigger >> picture first. >> >> -Alan >> >> >> >>
