Re: Why calling ListBucket for each file in a checkpoint

2024-01-21 Thread Zakelly Lan
Are you accessing the s3 API with presto implementation? If so, you may read the code of `com.facebook.presto.hive.s3.PrestoS3FileSystem#create` and find it check the existence of the target path first, in which the `getFileStatus` and `listPrefix` are called. There is no option for this. Best, Z

Why calling ListBucket for each file in a checkpoint

2024-01-18 Thread Evgeniy Lyutikov
Hi all! I'm trying to understand the logic of saving checkpoint files and from the exchange dump with ceph I see the following requests HEAD /checkpoints/example-job//shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9 HTTP/1.1 HTTP/1.1 404 Not Found HEAD /checkpoints/e