shashank created CAMEL-26743:
--------------------------------
Summary: camel-sql - JdbcOrphanLockAwareIdempotentRepository: the
keep-alive deadlocks with add and remove on a small connection pool, and leases
expire
Key: CAMEL-26743
URL: https://issues.apache.org/jira/browse/CAMEL-26743
Project: Camel
Issue Type: Bug
Components: camel-sql
Reporter: shashank
Assignee: shashank
{{JdbcOrphanLockAwareIdempotentRepository}} guards its set of held keys with a
{{StampedLock}}, and holds the lock while SQL runs:
* {{keepAlive()}} takes the read lock and then starts its transaction, which
borrows a connection from the pool.
* {{add()}} and {{remove()}} borrow a connection first (the transaction of
{{AbstractJdbcMessageIdRepository}}), and then take the write lock in
{{insert}} / {{delete}}, around the COUNT, INSERT, UPDATE or DELETE.
With more threads using the repository than the pool has connections, the two
orders form a cycle: the keep-alive holds the read lock and waits for a
connection, while every connection is held by an add or remove that waits for
the write lock. With a pool that waits as long as it takes (the defaults of
plain Agroal and DBCP2) the node deadlocks: no add or remove completes again.
With a pool timeout (HikariCP: 30 s) the keep-alive round fails after the
timeout, only logged at ERROR, and the add/remove calls stall until then; under
steady load the next round can be caught the same way. With an infinite wait no
lease is refreshed again; with a timeout, when rounds keep failing for
{{lockMaxAgeMillis}}, the leases expire. Then another node takes over keys this
node is still processing, so a message is processed twice at the same time.
The same lock also makes the keep-alive wait for any slow statement of an add
or remove: an INSERT that waits for a row lock of another node (for example a
transacted route on that node holding an uncommitted insert of the same key)
blocks the keep-alive of the whole node for as long as it waits. PostgreSQL,
Oracle and SQL Server have no lock timeout by default.
The lock only protects the in-memory set; it does not make the COUNT and the
INSERT atomic, as the transaction of add or remove commits after the lock is
released.
Expected: the keep-alive and add/remove never wait for each other while one of
them holds a connection, and the keep-alive refreshes the held keys while an
add or remove runs.
Suggested fix: keep the held keys in a concurrent set and hold no lock across
SQL; the keep-alive takes its timestamp and the keys once it has its connection.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
