Perfect, thank you Yufei.

Regards
Anjali

On Thu, Aug 12, 2021 at 9:58 PM Yufei Gu <flyrain...@gmail.com> wrote:

> Hi Anjali,
>
> Inline...
> On Thu, Aug 12, 2021 at 5:31 PM Anjali Norwood
> <anorw...@netflix.com.invalid> wrote:
>
>> Thanks for the summary Yufei.
>> Sorry, if this was already discussed, I missed the meeting yesterday.
>> Is there anything in the design that would prevent multiple roots from
>> being in different aws regions?
>>
> No. DR is the major use case of relative paths, if not the only one. So,
> it will support roots in different regions.
>
> For disaster recovery in the case of an entire aws region down or slow, is
>> metastore still a point of failure or can metastore be stood up in a
>> different region and could select a different root?
>>
> Normally, DR also requires a backup metastore, besides the storage(s3
> bucket). In that case, the backup metastore will be in a different region
> along with the table files. For example, the primary table is located in
> region A as well as its metastore, the backup table is located in region B
> as well as its metastore. The primary table root points to a path in region
> A, while backup table root points to a path in region B.
>
>
>> regards,
>> Anjali.
>>
>> On Thu, Aug 12, 2021 at 11:35 AM Yufei Gu <flyrain...@gmail.com> wrote:
>>
>>> Here is a summary of yesterday's community sync-up.
>>>
>>>
>>> Yufei gave a brief update on disaster recovery requirements and the
>>> current progress of relative path approach.
>>>
>>>
>>> Ryan: We all agreed that relative path is the way for disaster recovery.
>>>
>>>
>>> *Multiple roots for the relative path*
>>>
>>> Ryan proposed an idea to enable multiple roots for a table, basically,
>>> we can add a list of roots in table metadata, and use a selector to choose
>>> different roots when we move the table from one place to another. The
>>> selector reads a property to decide which root to use. The property could
>>> be either from catalog or the table metadata, which is yet to be decided.
>>>
>>>
>>> Here is an example I’d image:
>>>
>>>    1. Root1: hdfs://nn:8020/path/to/the/table
>>>    2. Root2: s3://bucket1/path/to/the/table
>>>    3. Root3: s3://bucket2/path/to/the/table
>>>
>>> *Relative path use case*
>>>
>>> We brainstormed use cases for relative paths. Please let us know if
>>> there are any other use cases.
>>>
>>>    1. Disaster Recovery
>>>    2. Jack: AWS s3 bucket alias
>>>    3. Ryan: fall-back use case. In case that the root1 doesn’t work,
>>>    the table falls back to root2, then root3. As Russell mentioned, it is
>>>    challenging to do snapshot expiration and other table maintenance 
>>> actions.
>>>
>>>
>>> *Timeline*
>>>
>>> In terms of timeline, relative path could be a feature in Spec V3, since
>>> Spec V1 and V2 assume absolute path in metadata.
>>>
>>>
>>> *Misc*
>>>
>>>    1. Miao: How is the relative path compatible with the absolute path?
>>>    2. How do we migrate an existing table? Build a tool for that.
>>>
>>> Please let us know if you have any ideas, questions, or concerns.
>>>
>>>
>>>
>>> Yufei
>>>
>>

Reply via email to