Re: Iceberg disaster recovery and relative path sync-up

2021-08-23 Thread Yufei Gu
@Ryan, how do these properties work with multiple table locations? 1. write.metadata.path 2. write.folder-storage.path 3. write.object-storage.path The current logic with single table location is to honor these properties on top of table location. In case of multiple roots(ta

Re: Iceberg disaster recovery and relative path sync-up

2021-08-23 Thread Anjali Norwood
Hi Ryan, All, *"The more I think about this, the more I like the solution to add multiple table roots to metadata, rather than removing table roots. Adding a way to plug in a root selector makes a lot of sense to me and it ensures that the metadata is complete (table location is set in metadata) a

Re:How to store the data of a table two months ago to oss

2021-08-23 Thread 1
Additional supplement: In hive, I can mv data(pt=today-60) from hdfs to oss, and then set the partition to point to the oss directory in hms But in iceberg, What should I do ? | | liubo07199 | | liubo07...@hellobike.com | On 08/23/2021 19:27,1 wrote: Hi: I have a table partitioned by

How to store the data of a table two months ago to oss

2021-08-23 Thread 1
Hi: I have a table partitioned by day, the historical data of this table needs to be stored on oss, That is to say: Table A, partitioned by day, store on hdfs. transfer data(partition=today-60) to oss everyday. data(partiton>today-60) is store on hdfs, and data(partiton<=today