Hi Jiuming, Could you please check if the Prometheus client can be used to reduce the JVM heap usage? If yes, I think we can consider using the Prometheus client instead of the current implementation together. Otherwise, we'd better focus on the heap memory usage enhancement for this discussion. Using the Prometheus client to refactor the current implementation will be a big project.
Thanks, Penghui On Sun, Feb 27, 2022 at 12:22 AM Jiuming Tao <jm...@streamnative.io.invalid> wrote: > Hi all, > https://github.com/apache/pulsar/pull/14453 < > https://github.com/apache/pulsar/pull/14453> please take a look. > > Thanks, > Tao Jiuming > > > 2022年2月24日 上午1:05，Jiuming Tao <jm...@streamnative.io> 写道： > > > > Hi all, > >> > >> 2. When there are hundreds MB metrics data collected, it causes high > heap memory usage, high CPU usage and GC pressure. In the > `PrometheusMetricsGenerator#generate` method, it uses > `ByteBufAllocator.DEFAULT.heapBuffer()` to allocate memory for writing > metrics data. The default size of `ByteBufAllocator.DEFAULT.heapBuffer()` > is 256 bytes, when the buffer resizes, the new buffer capacity is 512 > bytes(power of 2) and with `mem_copy` operation. > >> If I want to write 100 MB data to the buffer, the current buffer size > is 128 MB, and the total memory usage is close to 256 MB (256bytes + 512 > bytes + 1k + .... + 64MB + 128MB). When the buffer size is greater than > netty buffer chunkSize(16 MB), it will be allocated as UnpooledHeapByteBuf > in the heap. After writing metrics data into the buffer, return it to the > client by jetty, jetty will copy it into jetty's buffer with memory > allocation in the heap, again! > >> In this condition, for the purpose of saving memory, avoid high CPU > usage(too much memory allocations and `mem_copy` operations) and reducing > GC pressure, I want to change `ByteBufAllocator.DEFAULT.heapBuffer()` to > `ByteBufAllocator.DEFAULT.compositeDirectBuffer()`, it wouldn't cause > `mem_copy` operations and huge memory allocations(CompositeDirectByteBuf is > a bit slowly in read/write, but it's worth). After writing data, I will > call the `HttpOutput#write(ByteBuffer)` method and write it to the client, > the method won't cause `mem_copy` (I have to wrap ByteBuf to ByteBuffer, if > ByteBuf wrapped, there will be zero-copy). > > > > The jdk in my local is jdk15, I just noticed that in jdk8, ByteBuffer > cannot be extended and implemented. So, if allowed, I will write metrics > data to temp files and send it to client by jetty’s send_file. It will be > turned out a better performance than `CompositeByteBuf`, and takes lower > CPU usage due to I/O blocking.(The /metrics endpoint will be a bit slowly, > I believe it’s worth). > > If not allowed, it’s no matter and it also has a better performance than > `ByteBufAllocator.DEFAULT.heapBuffer()`(see the first image in original > mail). > > > > Thanks, > > Tao Jiuming > >
