[
https://issues.apache.org/jira/browse/CAMEL-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Snijder updated CAMEL-24248:
---------------------------------
Description:
Somewhat related to CAMEL-23840
When pollEnrich processes concurrent exchanges using different dynamic endpoint
URIs and its consumer cache is smaller than the concurrency level, a polling
consumer’s pool can be evicted while the consumer is still in use. When that
exchange finishes, the consumer is released after its pool has already been
removed, so the consumer is not stopped and its resources remain allocated.
Repeated requests cause these unclosed consumers to accumulate, resulting in
steadily increasing memory usage and potentially leaking component-specific
resources such as connections, clients, threads, or buffers. In the attached
reproducer, four concurrent SEDA consumers use dynamic polling URIs with
cacheSize(1). After 400 requests, 304 polling consumers remain active and
retain approximately 76 MB, whereas only the single cached consumer should
remain.
We found out about this issue on production with SQS component with the setting
*concurrentConsumers=4* and after that a *pollEnrich* with *cacheSize* set to
*1* (with sftp component). And we saw our application go OOM after about 4 days
consistently.
This turned out to be the issue.
Reproducer:
[https://github.com/msnijder30/reproduce-camel-pollenrich-memory-issue/tree/main]
was:
Somewhat related to CAMEL-23840
When pollEnrich processes concurrent exchanges using different dynamic endpoint
URIs and its consumer cache is smaller than the concurrency level, a polling
consumer’s pool can be evicted while the consumer is still in use. When that
exchange finishes, the consumer is released after its pool has already been
removed, so the consumer is not stopped and its resources remain allocated.
Repeated requests cause these unclosed consumers to accumulate, resulting in
steadily increasing memory usage and potentially leaking component-specific
resources such as connections, clients, threads, or buffers. In the attached
reproducer, four concurrent SEDA consumers use dynamic polling URIs with
cacheSize(1). After 400 requests, 304 polling consumers remain active and
retain approximately 76 MB, whereas only the single cached consumer should
remain.
We found out about this issue on production with SQS component with the setting
*concurrentConsumers=4* and after that a *pollEnrich* with *cacheSize* set to
one (with sftp component). And we saw our application go OOM after about 4 days
consistently.
This turned out to be the issue.
Reproducer:
https://github.com/msnijder30/reproduce-camel-pollenrich-memory-issue/tree/main
> Concurrent pollEnrich can fail to stop in-use consumers after cache eviction
> ----------------------------------------------------------------------------
>
> Key: CAMEL-24248
> URL: https://issues.apache.org/jira/browse/CAMEL-24248
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 4.14.6, 4.18.0, 4.22.0
> Reporter: Mark Snijder
> Priority: Minor
> Attachments: reproduce-camel-pollenrich-memory-issue-main.zip
>
>
> Somewhat related to CAMEL-23840
>
> When pollEnrich processes concurrent exchanges using different dynamic
> endpoint URIs and its consumer cache is smaller than the concurrency level, a
> polling consumer’s pool can be evicted while the consumer is still in use.
> When that exchange finishes, the consumer is released after its pool has
> already been removed, so the consumer is not stopped and its resources remain
> allocated.
> Repeated requests cause these unclosed consumers to accumulate, resulting in
> steadily increasing memory usage and potentially leaking component-specific
> resources such as connections, clients, threads, or buffers. In the attached
> reproducer, four concurrent SEDA consumers use dynamic polling URIs with
> cacheSize(1). After 400 requests, 304 polling consumers remain active and
> retain approximately 76 MB, whereas only the single cached consumer should
> remain.
>
> We found out about this issue on production with SQS component with the
> setting *concurrentConsumers=4* and after that a *pollEnrich* with
> *cacheSize* set to *1* (with sftp component). And we saw our application go
> OOM after about 4 days consistently.
>
> This turned out to be the issue.
>
> Reproducer:
> [https://github.com/msnijder30/reproduce-camel-pollenrich-memory-issue/tree/main]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
