> Or was it the querying of the checkpoints you were advising against?
Yes, I meant the approach, not file removal itself. Mainly because how
exactly FileSource stores its state is an implementation detail and there
are no external guarantees for its consistency between even the minor
versions.
On
> This is not a robust solution, I would advise against it.
Oh no? Am curious as to why not. It seems not dissimilar to how Kafka
topic retention works: the messages are removed after some time period
(hopefully after they are processed), so why would it be bad to remove
files that are already pr
