andygrove opened a new issue, #2673: URL: https://github.com/apache/datafusion-ballista/issues/2673
## Describe the issue `DefaultBallistaClientPool::shared` (`ballista/executor/src/client_pool.rs`, added in #2670) keeps one `SharedSlot` per `(host, port, config)` key in a `DashMap`. Slots are never removed: - discarding a connection clears the slot's connection but leaves the slot in the map; - `evict_idle` only drains the checked-out pool's idle deque and ignores shared slots. With executor churn, slots for `(host, port, config)` keys that no longer exist accumulate for the life of the process. Each is small (two mutexes and an `Option`), so this is a slow leak rather than a fast one. A shared connection to an executor that is still running but no longer read from is also never closed, since shared connections have no idle timeout. ## Proposed change - Remove the slot on discard when the discarded connection is the slot's current one. - In the eviction pass, drop slots whose connection has no holder outside the slot (`Arc::strong_count == 1`) for longer than the idle timeout, closing the connection. - Note the race to test: a caller can be inside `shared()` holding the slot `Arc` while the eviction pass removes the slot from the map. It then inserts a connection into a detached slot, which works but is not reused by the next caller. ## Context Raised in the review of #2670. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
