akshaychitneni opened a new pull request, #2889:
URL: https://github.com/apache/datafusion-ballista/pull/2889
Part of #1829.
# Rationale for this change
`DataFrame::cache()` data is never freed: removing a session leaves its
caches
pinned, and a dropped cache's files are deleted a fixed 300 s later even
if a
query is still reading them.
# What changes are included in this PR?
- **Removing a session drops its caches.** gRPC `RemoveSession` and Flight
SQL
session expiry now go through `SchedulerState::remove_session`:
- built caches leave the registry at once;
- builds still in flight are cancelled, and what waits on them fails with
"its session was closed";
- using the session again rebuilds its caches.
- **Dropped cache data is deleted once no running query reads it.** Every
drop
path (session removal, validate-on-read, recovery, the cleanup after a
cache-lost failure) now orphans the build while queries read it, then
deletes
it after the usual delay. A healthy cache dropped by another query's
failure
is no longer deleted under a query still reading it.
**Deferred:**
- **A trigger clients use.** No client library sends `RemoveSession` today
(Rust, Python, CLI), and Flight SQL sessions hold no caches. Next:
idle-session
expiry on the scheduler, and an explicit close API for clients.
- **Dropping only the lost caches.** After a cache-lost failure, every
cache the
job read is still dropped; this PR only stops their data being deleted
under
other readers. Narrowing the drop needs structured lost-input reporting.
# Are there any user-facing changes?
Yes: removing a session frees its cached data, and queries held on that
session's in-flight builds fail with "its session was closed". There's no
public API, proto or configuration change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]