akshaychitneni opened a new pull request, #2889:
URL: https://github.com/apache/datafusion-ballista/pull/2889

    Part of #1829.
   
     # Rationale for this change
   
     `DataFrame::cache()` data is never freed: removing a session leaves its 
caches
     pinned, and a dropped cache's files are deleted a fixed 300 s later even 
if a
     query is still reading them.
   
     # What changes are included in this PR?
   
     - **Removing a session drops its caches.** gRPC `RemoveSession` and Flight 
SQL
       session expiry now go through `SchedulerState::remove_session`:
       - built caches leave the registry at once;
       - builds still in flight are cancelled, and what waits on them fails with
         "its session was closed";
       - using the session again rebuilds its caches.
     - **Dropped cache data is deleted once no running query reads it.** Every 
drop
       path (session removal, validate-on-read, recovery, the cleanup after a
       cache-lost failure) now orphans the build while queries read it, then 
deletes
       it after the usual delay. A healthy cache dropped by another query's 
failure
       is no longer deleted under a query still reading it.
   
     **Deferred:**
     - **A trigger clients use.** No client library sends `RemoveSession` today
       (Rust, Python, CLI), and Flight SQL sessions hold no caches. Next: 
idle-session
       expiry on the scheduler, and an explicit close API for clients.
     - **Dropping only the lost caches.** After a cache-lost failure, every 
cache the
       job read is still dropped; this PR only stops their data being deleted 
under
       other readers. Narrowing the drop needs structured lost-input reporting.
   
     # Are there any user-facing changes?
   
     Yes: removing a session frees its cached data, and queries held on that
     session's in-flight builds fail with "its session was closed". There's no
     public API, proto or configuration change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to