Are you accessing the s3 API with presto implementation? If so, you may read the code of `com.facebook.presto.hive.s3.PrestoS3FileSystem#create` and find it check the existence of the target path first, in which the `getFileStatus` and `listPrefix` are called. There is no option for this.
Best, Zakelly On Fri, Jan 19, 2024 at 9:32 PM Evgeniy Lyutikov <eblyuti...@avito.ru> wrote: > Hi all! > I'm trying to understand the logic of saving checkpoint files and from the > exchange dump with ceph I see the following requests > > HEAD > /checkpoints/example-job/00000000000000000000000000000000/shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9 > HTTP/1.1 > HTTP/1.1 404 Not Found > HEAD > /checkpoints/example-job/00000000000000000000000000000000/shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9/ > HTTP/1.1 > HTTP/1.1 404 Not Found > GET > /checkpoints/?prefix=example-job%2F00000000000000000000000000000000%2Fshared%2F9701fae2-0de3-4d6c-b08b-0a92fb7285c9%2F&delimiter=%2F&max-keys=1&encoding-type=url > HTTP/1.1 > HTTP/XML 486 HTTP/1.1 200 OK > <?xml version="1.0" encoding="UTF-8" ?> > <ListBucketResult > xmlns="http://s3.amazonaws.com/doc/2006-03-01/";> > <EncodingType> > <Name> > checkpoints > </Name> > <Prefix> > > example-job/00000000000000000000000000000000/shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9/ > </Prefix> > <MaxKeys> > 1 > </MaxKeys> > <Delimiter> > / > </Delimiter> > <IsTruncated> > false > </IsTruncated> > <Marker> > </Marker> > </ListBucketResult> > PUT > /checkpoints/example-job/00000000000000000000000000000000/shared/9701fae2-0de3-4d6c-b08b-0a92fb7285c9 > HTTP/1.1 > > It’s not entirely clear why, after checking for the presence of an object > and a directory by HEAD methods additionally call ListBucket get request. > Can this be disabled using options or does the code require modification? > We have our own Ceph cluster and frequent ListBucket requests heavily load > the index pool. > Flink 1.17.1, presto backend for checkpoint. > > > > * ------------------------------ *“This message contains confidential > information/commercial secret. If you are not the intended addressee of > this message you may not copy, save, print or forward it to any third party > and you are kindly requested to destroy this message and notify the sender > thereof by email. > Данное сообщение содержит конфиденциальную информацию/информацию, > являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом > данного сообщения, Вы не вправе копировать, сохранять, печатать или > пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и > уведомить об этом отправителя электронным письмом.” >
