Kimahriman commented on PR #50612: URL: https://github.com/apache/spark/pull/50612#issuecomment-2813777136
> @Kimahriman - yea and with 4.0, the only additional cost for `doMaintenance` would be around uploading the snapshot and deleting old versions. Both likely won't be expensive on a per batch basis (mostly a no-op for most batches) but you could see some spikes in latency when both these operations are performed. (I guess thats acceptable in your case rather than keeping the resources running) Yeah I thought about trying to use the background maintenance to clean up the state but that just seemed hacky and race condition prone, and I'm already saying I care less about latency in this mode -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
