Why calling ListBucket for each file in a checkpoint

[Evgeniy Lyutikov]

Hi all!
I'm trying to understand the logic of saving checkpoint files and from the 
exchange dump with ceph I see the following requests

HEAD 
/checkpoints/example-job/00000000000000000000000000000000/shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9
 HTTP/1.1
HTTP/1.1 404 Not Found
HEAD 
/checkpoints/example-job/00000000000000000000000000000000/shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9/
 HTTP/1.1
HTTP/1.1 404 Not Found
GET 
/checkpoints/?prefix=example-job%2F00000000000000000000000000000000%2Fshared%2F9701fae2-0de3-4d6c-b08b-0a92fb7285c9%2F&delimiter=%2F&max-keys=1&encoding-type=url
 HTTP/1.1
HTTP/XML 486    HTTP/1.1 200 OK
<?xml version="1.0" encoding="UTF-8" ?>
    <ListBucketResult
        xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>
        <EncodingType>
        <Name>
            checkpoints
        </Name>
        <Prefix>
            
example-job/00000000000000000000000000000000/shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9/
        </Prefix>
        <MaxKeys>
            1
        </MaxKeys>
        <Delimiter>
            /
        </Delimiter>
        <IsTruncated>
            false
        </IsTruncated>
        <Marker>
        </Marker>
    </ListBucketResult>
PUT 
/checkpoints/example-job/00000000000000000000000000000000/shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9
 HTTP/1.1

It’s not entirely clear why, after checking for the presence of an object and a 
directory by HEAD methods additionally call ListBucket get request.
Can this be disabled using options or does the code require modification?
We have our own Ceph cluster and frequent ListBucket requests heavily load the 
index pool.
Flink 1.17.1, presto backend for checkpoint.



________________________________
“This message contains confidential information/commercial secret. If you are 
not the intended addressee of this message you may not copy, save, print or 
forward it to any third party and you are kindly requested to destroy this 
message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся 
коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного 
сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его 
каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом 
отправителя электронным письмом.”