DaanHoogland commented on issue #13586: URL: https://github.com/apache/cloudstack/issues/13586#issuecomment-5021189382
thanks, guys, an AI generated analysis: **Root cause (code inspection, 4.22.0.0 tag):** `PrometheusExporterImpl.updateMetrics()` calls `alertManager.recalculateCapacity()` once per datacenter inside its scrape loop: https://github.com/apache/cloudstack/blob/4.22.0.0/plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java#L486-L504 `AlertManagerImpl.recalculateCapacity()` already loops over **every** zone, host, and storage pool internally: https://github.com/apache/cloudstack/blob/4.22.0.0/server/src/main/java/com/cloud/alert/AlertManagerImpl.java#L373-L376 So with Z zones, a single `/metrics` scrape triggers Z redundant full-environment recalculations (O(Z²) work). Each call also spins up a brand-new `Executors.newFixedThreadPool(...)` per host and per storage pool (`recalculateHostCapacities()` / `recalculateStorageCapacities()`, lines ~296-368), then tears it down — so the thread-pool churn scales with scrape frequency, not with the intended once-per-`capacity.check.period` schedule from `AlertManagerImpl`'s own `CapacityChecker` timer (`start()`, line ~264). On top of that, `PrometheusExporterServerImpl.start()` creates the JDK `HttpServer` without a custom executor (no `setExecutor(...)` call), so scrape requests are served through the server's single default dispatch path — a slow scrape blocks/queues the next one rather than running concurrently. Put together: every Prometheus scrape (typically every 15-60s) forces a full, uncoordinated, thread-pool-churning capacity recalculation across the whole environment, running concurrently with — and uncoordinated against — the existing timer-driven recalculation. That matches "duration climbs slowly over 7-10 days, restart clears it" well: no single scrape is catastrophic, but the accumulated thread churn and request queuing degrade the JVM over time until a restart resets it. I checked open issues/PRs for `recalculateCapacity`/`prometheus` and didn't find this tracked elsewhere, so filing this as new. **Proposed fix** (PR incoming): stop calling `recalculateCapacity()` from the scrape path — the exporter should read the capacity data the existing timer already maintains, not force a synchronous recompute per request — and have `AlertManagerImpl` reuse one long-lived executor for capacity calculation instead of creating/tearing one down per call. **Can you confirm on your end?** If you're able to test a build with `alertManager.recalculateCapacity()` commented out of `updateMetrics()`, that would confirm this is the whole story before we merge. In the meantime, reducing your Prometheus scrape interval (and/or the periodic restart you're already doing) is the only workaround. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
