Secondary Index Support Draft

Hi, As we discussed in the sync up meeting, I come up with a draft for secondary index support. Draft. It is WIP. Detailed API design is not included in this version. Thanks! Miao

Re: Iceberg/Hive properties handling

When I was working on the CREATE TABLE patch I found the following TBLPROPERTIES on newly created tables: external.table.purge EXTERNAL bucketing_version numRows rawDataSize totalSize numFiles numFileErasureCoded I am afraid that we can not change the name of most of these properties, and might

Re: Iceberg/Hive properties handling

Sorry, I accidentally didn’t copy the dev list on this reply. Resending: Also I expect that we want to add Hive write specific configs to table level when the general engine independent configuration is not ideal for Hive, but every Hive query for a given table should use some specific config. Hi

Re: S3 strong read-after-write consistency

This feature will definitely help cases where we saw a file not found exception after creating the new file using s3a (spark use to retry task in that case). On Wed, Dec 2, 2020 at 2:11 AM Jungtaek Lim wrote: > What about S3FileIO implementation? I see some issue filed that even with > Hive cata

Re: S3 strong read-after-write consistency

What about S3FileIO implementation? I see some issue filed that even with Hive catalog working with S3 brings unexpected issues, and S3FileIO supposed to fix the issue (according to Ryan). Is it safe without S3FileIO to use Hive catalog + Hadoop API for S3 now? 2020년 12월 2일 (수) 오후 6:54, Vivekanand

Re: S3 strong read-after-write consistency

Iceberg tables backed by HadoopTables and HadoopCatalog require an atomic rename. This is not yet supported with S3. On Wed, Dec 2, 2020 at 3:20 PM Mass Dosage wrote: > Hello all, > > Yesterday AWS announced that S3 now has strong read-after-write > consistency: > > > https://aws.amazon.com/blog

S3 strong read-after-write consistency

Hello all, Yesterday AWS announced that S3 now has strong read-after-write consistency: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency https://aws.amazon.com/s3/consistency/ Does this mean that Iceberg tables backed by HadoopTables and HadoopCatalog can no