Re: Iceberg disaster recovery and relative path sync-up

2021-09-02 Thread Ryan Blue
Are we planning to make sure that the tables with relative paths will always contain every data/metadata file in single root folder? This depends on the use case. For table mirroring, there would be a back-end service making copies of all metadata, but not for other use cases. For example, the ren

Re: [CWS] Re: Iceberg disaster recovery and relative path sync-up

2021-09-02 Thread Ryan Blue
Anjali, my thoughts are inline below: On Mon, Aug 23, 2021 at 1:14 PM Anjali Norwood wrote: > *"The more I think about this, the more I like the solution to add > multiple table roots to metadata, rather than removing table roots. Adding > a way to plug in a root selector makes a lot of sense to

Re: Iceberg disaster recovery and relative path sync-up

2021-09-02 Thread Ryan Blue
Jack, I agree with just about everything you've said. On Sun, Aug 29, 2021 at 2:13 PM Jack Ye wrote: > trying to catch up with the conversation here, just typing some of my > thought process: > > Based on my understanding, there are in general 2 use cases: > > 1. multiple data copies in differen

Re: Iceberg disaster recovery and relative path sync-up

2021-09-02 Thread Ryan Blue
Yufei, answers inline: On Mon, Aug 23, 2021 at 4:06 PM Yufei Gu wrote: > @Ryan, how do these properties work with multiple table locations? > >1. > >write.metadata.path >2. > >write.folder-storage.path >3. > >write.object-storage.path > > The current logic with single tab