Cool I also observed such problems but I haven't time to work on a proposal.
Looking forward to see your patch Enrico Il Mer 23 Feb 2022, 10:43 Jiuming Tao <jm...@streamnative.io.invalid> ha scritto: > Hi all, > > 1. I have learned that the /metrics endpoint will be requested by more than > one metrics collect system. In the condition, I want to reimplement > `PromethuesMetricsServlet` by sliding window. > PrometheusMetricsGenerator#generate will be invoked once in a period(such > as 1 minute), the result will be cached and returned for every metrics > collect request in the period directly. It could save memory and avoid high > CPU usage. > > 2. When there are hundreds MB metrics data collected, it causes high heap > memory usage, high CPU usage and GC pressure. In the > `PrometheusMetricsGenerator#generate` method, it uses > `ByteBufAllocator.DEFAULT.heapBuffer()` to allocate memory for writing > metrics data. The default size of `ByteBufAllocator.DEFAULT.heapBuffer()` > is 256 bytes, when the buffer resizes, the new buffer capacity is 512 > bytes(power of 2) and with `mem_copy` operation. > If I want to write 100 MB data to the buffer, the current buffer size is > 128 MB, and the total memory usage is close to 256 MB (256bytes + 512 bytes > + 1k + .... + 64MB + 128MB). When the buffer size is greater than netty > buffer chunkSize(16 MB), it will be allocated as UnpooledHeapByteBuf in the > heap. After writing metrics data into the buffer, return it to the client > by jetty, jetty will copy it into jetty's buffer with memory allocation in > the heap, again! > In this condition, for the purpose of saving memory, avoid high CPU > usage(too much memory allocations and `mem_copy` operations) and reducing > GC pressure, I want to change `ByteBufAllocator.DEFAULT.heapBuffer()` to > `ByteBufAllocator.DEFAULT.compositeDirectBuffer()`, it wouldn't cause > `mem_copy` operations and huge memory allocations(CompositeDirectByteBuf is > a bit slowly in read/write, but it's worth). After writing data, I will > call the `HttpOutput#write(ByteBuffer)` method and write it to the client, > the method won't cause `mem_copy` (I have to wrap ByteBuf to ByteBuffer, if > ByteBuf wrapped, there will be zero-copy). > I tested NO.2 in my local, and it turns out performance is better than the > heap buffer(below images). > > > https://drive.google.com/file/d/1-0drrs9s9kZ2NbbVmzQDwPHdgtpE6QyW/view?usp=sharing > (CompositeDirectByteBuf) > > https://drive.google.com/file/d/1-0m15YdsjBudsiweZ4DO7aU3bOFeK17w/view?usp=sharing > (PooledHeapByteBuf) > > Thanks, > Tao Jiuming >
