[
https://issues.apache.org/jira/browse/CAMEL-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121958#comment-18121958
]
Federico Mariani commented on CAMEL-25267:
------------------------------------------
Root cause confirmed, and there was a second quadratic cost besides the one
described:
# {{ManagedRoute}} found its processors by reading the route id of every
processor MBean in the context. It did this in {{processorIds()}},
{{dumpRouteStatsAsXml/JSon(includeProcessors=true)}}, {{dumpStepStatsAsXml}},
{{dumpRouteSourceLocationsAsXml}} and {{reset(true)}}. The last one used a
{{QueryExp}}, which still reads the attribute of every MBean.
# The callers then call {{ManagedCamelContext.getManagedProcessor(id)}} for
each id, and that walked every route twice ({{CamelContext.getProcessor(id)}} +
{{Model.getProcessorDefinition(id)}}).
Fix: [PR #27280|https://github.com/apache/camel/pull/27280].
{{DefaultManagementAgent}} indexes the processor and step MBeans it registers,
by route and by id, and these lookups use the index. A custom agent, or an id
that is not indexed, still uses the previous lookup. No API, MBean name or
result changes.
Measured with 300 routes x 12 processors, JMX enabled (harness attached:
ProcessorIdsTimingTest.java, copy it into camel-console tests):
|| || before || after ||
| processorIds() for all routes | ~12 s | ~45 ms |
| + getManagedProcessor(id) for each id | ~13 s | ~65 ms |
| route console, processors=true | ~27 s | ~0.7 s |
The PR has a minimal test. A longer reproducer covering each ManagedRoute
method is attached for reference
(ManagedRouteProcessorsOfOtherRoutesFullTest.java). Without the fix, all 6 of
its tests fail; with the fix, they pass.
Seen along the way but not addressed here (not verified): the route-group
console seems to process each group once per route in it, possibly because its
{{.distinct()}} doesn't de-duplicate the JMX proxies.
_Claude Code on behalf of Croway_
> camel-management - ManagedRoute.processorIds is quadratic, the route dev
> console with processors takes seconds on large integrations
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25267
> URL: https://issues.apache.org/jira/browse/CAMEL-25267
> Project: Camel
> Issue Type: Improvement
> Components: camel-core, camel-management
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
> Attachments: ManagedRouteProcessorsOfOtherRoutesFullTest.java,
> ProcessorIdsTimingTest.java
>
>
> {{ManagedRoute.processorIds()}} queries the JMX MBeans of *all* processors of
> the CamelContext, creates a proxy for each one and keeps those of its own
> route. The route dev console with {{processors=true}} calls it for every
> route ({{RouteDevConsole.includeProcessorsJson}}), so it creates routes ×
> processors proxies: quadratic in the size of the integration.
> With 316 YAML routes (about 4000 processors) one call of the route console
> with {{processors=true}} took about 30 s (thread dumps show the time in
> {{DefaultManagementAgent.newProxyClient}} / {{isRegistered}} under
> {{ManagedRoute.processorIds}}).
> Who is affected:
> * camel-cli-connector collects this as part of the status snapshot: every
> poll of the file transport (so {{camel get}} / {{camel cmd}} are delayed by
> the same amount, the connector thread is busy almost all the time) and every
> snapshot of the WebSocket transport (CAMEL-25197, where it starved the
> heartbeat).
> * anything else calling {{processorIds()}} per route (e.g. the {{route}} dev
> console over HTTP, JMX clients).
> Possible fixes:
> * collect the processor ids of a route from the route itself (its processors
> / model) instead of a JMX query of the whole context, or query only once and
> group by route id;
> * avoid one {{newProxyClient}} per processor when only the id and route id
> are needed.
> Reproducer: a Camel Main application with a few hundred routes of ~10
> processors each, JMX enabled, then call the route dev console with
> {{processors=true}} (or run it with camel-cli-connector and look at the
> status).
> _Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
