Re: [DISCUSS] PrometheusMetricsServlet performance improvement

Note that the PrometheusMetricsGenerator will generate the text for Prometheus client metrics registered to the default CollectorRegistry. So it should be possible to progressively convert stats objects to Prometheus Gauge/Counter/etc... Le mer. 9 mars 2022 à 06:57, Michael Marshall a écrit : >

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

Thanks for looking into the prometheus client, Tao Jiuming. I am surprised that it isn't optimized to minimize heap usage. In that case, I think it makes sense to move forward with your solution. Thanks, Michael On Mon, Feb 28, 2022 at 10:23 PM PengHui Li wrote: > > Hi, > > Thanks for the upda

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

Hi, Thanks for the update, I agree to have the JVM heap usage enhancement. +1 Penghui On Mon, Feb 28, 2022 at 5:33 PM Jiuming Tao wrote: > Hi Penghui, > > I had read `io.prometheus.client.exporter.HttpServer` source code, in > `HttpMetricsHandler#handle` method, it uses thread local cached >

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

Hi Penghui, I had read `io.prometheus.client.exporter.HttpServer` source code, in `HttpMetricsHandler#handle` method, it uses thread local cached `ByteArrayOutputStream` , it’s similar with our current implemention(with heap memory resizes and mem_copy). It will spend a plenty of heap memory,

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

Hi Jiuming, Could you please check if the Prometheus client can be used to reduce the JVM heap usage? If yes, I think we can consider using the Prometheus client instead of the current implementation together. Otherwise, we'd better focus on the heap memory usage enhancement for this discussion. U

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

Hi all, https://github.com/apache/pulsar/pull/14453 please take a look. Thanks, Tao Jiuming > 2022年2月24日 上午1:05，Jiuming Tao 写道： > > Hi all, >> >> 2. When there are hundreds MB metrics data collected, it causes high heap >> memory usage, high CPU

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

> Old codes, I think at that time, prometheus is not popular yet I think there is likely more explanation here, since we could have switched any time in the past few years when prometheus was already popular. Before we add caching to our metrics generation, I think we should consider migrating to

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

> I have a historical question. Why do we write and maintain our own > code to generate the metrics response instead of using the prometheus > client library? Old codes, I think at that time, prometheus is not popular yet >> I have learned that the /metrics endpoint will be requested by more tha

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

I have a historical question. Why do we write and maintain our own code to generate the metrics response instead of using the prometheus client library? > I have learned that the /metrics endpoint will be requested by more than > one metrics collect system. In practice, when does this happen? >

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

Hi all, > > 2. When there are hundreds MB metrics data collected, it causes high heap > memory usage, high CPU usage and GC pressure. In the > `PrometheusMetricsGenerator#generate` method, it uses > `ByteBufAllocator.DEFAULT.heapBuffer()` to allocate memory for writing > metrics data. The defa

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

+1 Great work. Thanks, Penghui On Wed, Feb 23, 2022 at 8:56 PM Enrico Olivelli wrote: > Cool > > I also observed such problems but I haven't time to work on a proposal. > > Looking forward to see your patch > > > Enrico > > Il Mer 23 Feb 2022, 10:43 Jiuming Tao ha > scritto: > > > Hi all, > >

Re: [DISCUSS] PrometheusMetricsServlet performance improvement

Cool I also observed such problems but I haven't time to work on a proposal. Looking forward to see your patch Enrico Il Mer 23 Feb 2022, 10:43 Jiuming Tao ha scritto: > Hi all, > > 1. I have learned that the /metrics endpoint will be requested by more than > one metrics collect system. In t

[DISCUSS] PrometheusMetricsServlet performance improvement

Hi all, 1. I have learned that the /metrics endpoint will be requested by more than one metrics collect system. In the condition, I want to reimplement `PromethuesMetricsServlet` by sliding window. PrometheusMetricsGenerator#generate will be invoked once in a period(such as 1 minute), the result w