milenkovicm commented on issue #1316: URL: https://github.com/apache/datafusion-ballista/issues/1316#issuecomment-3279895972
I think you could make an epic in this area There are few things I would like to propose regarding to files: 1. shuffle files are absolute paths, i would like to make them relative to flight store directory (improves security) 2. I would like to change how files are organised, instead of having `job_id/shuffle...` i would like to prefix files with `session_id/job_id/shuffle...` or `session_id/cache/` or `session_id/job_id/operator_state` ... i would like to be able to delete any of them, but also delete all files for given session_id when client disconnects 3. should we make arrow flight server owner of files, and expose actions on them to delete files ? 4. #558 wdyt @KR-bluejay ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
