Hi,
We noticed it first when running out of space on the S3 PV. We worked out
the subdirectory is the job Id, so I don't know how that is meant to be
cleaned up when the job is recreated with a new job Id . It might be we
need to work something out outside of flink.
But we also noticed a huge num
FW to user ML.
Hi Jean-Marc,
Could you elaborate more about how you noticed an increasing number of
checkpoints are left behind? Is the number of subdirectories under
s3://flink-application/checkpoints increasing? And have you set the state
TTL?
Best,
Zakelly
On Thu, Jan 2, 2025 at 12:19 PM Za
As far as I know, it's not possible to do without using the Catalog.
Oh... I forgot that the FileSystemCatalog is currently only used for
testing[1]. One approach is to implement and package a FileSystemCatalog by
refering this testing catalog.
[1]
https://github.com/apache/flink/blob/mast
