I had a call with some developers from S3 and asked and they said this change should resolve the "negative caching" issue.
Atomic renames are on their radar but they said this will take a lot of work on their part. On Fri, 4 Dec 2020 at 21:57, Ryan Blue <rb...@netflix.com.invalid> wrote: > It isn't clear whether this S3 consistency change also fixes the negative > caching (HEAD when file doesn't exist causes later HEAD to not see the > file), but I think that it does not fix it because there was a PR opened to > add consistency using LIST before a HEAD operation. > > I think it is still a good idea to use the new S3FileIO for S3 tables. > > On Wed, Dec 2, 2020 at 2:11 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> What about S3FileIO implementation? I see some issue filed that even with >> Hive catalog working with S3 brings unexpected issues, and S3FileIO >> supposed to fix the issue (according to Ryan). Is it safe without S3FileIO >> to use Hive catalog + Hadoop API for S3 now? >> >> 2020년 12월 2일 (수) 오후 6:54, Vivekanand Vellanki <vi...@dremio.com>님이 작성: >> >>> Iceberg tables backed by HadoopTables and HadoopCatalog require an >>> atomic rename. This is not yet supported with S3. >>> >>> On Wed, Dec 2, 2020 at 3:20 PM Mass Dosage <massdos...@gmail.com> wrote: >>> >>>> Hello all, >>>> >>>> Yesterday AWS announced that S3 now has strong read-after-write >>>> consistency: >>>> >>>> >>>> https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency >>>> >>>> https://aws.amazon.com/s3/consistency/ >>>> >>>> Does this mean that Iceberg tables backed by HadoopTables and >>>> HadoopCatalog can now be used on S3 in addition to HDFS? >>>> >>>> Thanks, >>>> >>>> Adrian >>>> >>> > > -- > Ryan Blue > Software Engineer > Netflix >
