Hi All,
We've been fighting checkpoint corruption on stateful Structured Streaming with S3 checkpoints for a few months now (Spark 3.5.x). The error looks like this: Error reading delta file .../state/0/506/1.delta ... does not exist 1.delta, on a query sitting at version xxx . The provider can't find a snapshot, falls back to replaying from version zero, and asks for the first delta, which expired weeks ago by S3 lifecyle policy. Snapshots only get written by the background maintenance task. If that stops keeping up, you're now relying on every single delta since version 1 still being there. Then the S3 bucket's lifecycle rule expires some of those deltas, and you encounter the error above upon the next restart. As for why maintenance falls behind: the trigger finishes too fast for a small streaming profile. The maintenance task is low priority, it doesn't hold the app up, so it just gets killed before it finishes. Anyway, the actual question. We're planning to drop S3 expiry from the checkpoint prefixes entirely and handle size with a cleanup that checks whether a file is still referenced instead of checking how old it is. Is that what everyone else does? Or is anyone running bounded retention on checkpoint prefixes without getting bitten, and if so what makes it safe? Thanks, Dibyendu
