Perfect, thank you Yufei. Regards Anjali
On Thu, Aug 12, 2021 at 9:58 PM Yufei Gu <flyrain...@gmail.com> wrote: > Hi Anjali, > > Inline... > On Thu, Aug 12, 2021 at 5:31 PM Anjali Norwood > <anorw...@netflix.com.invalid> wrote: > >> Thanks for the summary Yufei. >> Sorry, if this was already discussed, I missed the meeting yesterday. >> Is there anything in the design that would prevent multiple roots from >> being in different aws regions? >> > No. DR is the major use case of relative paths, if not the only one. So, > it will support roots in different regions. > > For disaster recovery in the case of an entire aws region down or slow, is >> metastore still a point of failure or can metastore be stood up in a >> different region and could select a different root? >> > Normally, DR also requires a backup metastore, besides the storage(s3 > bucket). In that case, the backup metastore will be in a different region > along with the table files. For example, the primary table is located in > region A as well as its metastore, the backup table is located in region B > as well as its metastore. The primary table root points to a path in region > A, while backup table root points to a path in region B. > > >> regards, >> Anjali. >> >> On Thu, Aug 12, 2021 at 11:35 AM Yufei Gu <flyrain...@gmail.com> wrote: >> >>> Here is a summary of yesterday's community sync-up. >>> >>> >>> Yufei gave a brief update on disaster recovery requirements and the >>> current progress of relative path approach. >>> >>> >>> Ryan: We all agreed that relative path is the way for disaster recovery. >>> >>> >>> *Multiple roots for the relative path* >>> >>> Ryan proposed an idea to enable multiple roots for a table, basically, >>> we can add a list of roots in table metadata, and use a selector to choose >>> different roots when we move the table from one place to another. The >>> selector reads a property to decide which root to use. The property could >>> be either from catalog or the table metadata, which is yet to be decided. >>> >>> >>> Here is an example I’d image: >>> >>> 1. Root1: hdfs://nn:8020/path/to/the/table >>> 2. Root2: s3://bucket1/path/to/the/table >>> 3. Root3: s3://bucket2/path/to/the/table >>> >>> *Relative path use case* >>> >>> We brainstormed use cases for relative paths. Please let us know if >>> there are any other use cases. >>> >>> 1. Disaster Recovery >>> 2. Jack: AWS s3 bucket alias >>> 3. Ryan: fall-back use case. In case that the root1 doesn’t work, >>> the table falls back to root2, then root3. As Russell mentioned, it is >>> challenging to do snapshot expiration and other table maintenance >>> actions. >>> >>> >>> *Timeline* >>> >>> In terms of timeline, relative path could be a feature in Spec V3, since >>> Spec V1 and V2 assume absolute path in metadata. >>> >>> >>> *Misc* >>> >>> 1. Miao: How is the relative path compatible with the absolute path? >>> 2. How do we migrate an existing table? Build a tool for that. >>> >>> Please let us know if you have any ideas, questions, or concerns. >>> >>> >>> >>> Yufei >>> >>
