Jack, I agree with just about everything you've said.

On Sun, Aug 29, 2021 at 2:13 PM Jack Ye <yezhao...@gmail.com> wrote:

> trying to catch up with the conversation here, just typing some of my
> thought process:
>
> Based on my understanding, there are in general 2 use cases:
>
> 1. multiple data copies in different physical storages, which includes:
>
> 1.1 disaster recovery: if 1 storage is completely down, all access needs
> to be redirected another storage
> 1.2 multi-tier data access: a table is stored in multiple tiers, such as a
> fast single-availability-zone (AZ) layer, a common multi-AZ layer, and a
> slow but cheap archival layer
> 1.3 table migration: user can manually migrate a table to point to a
> completely different location with completely different set of data files
>
> 2. different access points referring to the same storage, which includes:
>
> 2.1 access point: different names are pointed to the same physical storage
> with different access policies.
>
>
> I think we mostly reached the consensus that multiple root locations are
> needed.
> Now we need to ask the question of "Can a user access the storage from
> catalog to at least get the table metadata and read all the different root
> locations".
> I think the answer is Yes for 1.2 and 1.3, but No for 1.1 and 2.1.
> Therefore, we need to deal with those 2 situations separately.
>
> For 1.1 and 2.1, the metadata cannot be accessed by the user unless an
> alternative root bucket/storage endpoint is provided at catalog level. So
> this information should be put as a part of the catalog API.
> For 1.2 and 1.3, we need some more details about the characteristics of
> the root location, specifically:
> 1. if the location is read only or it can also be written to. Typically we
> should only allow 1 single "master" write location to avoid complicated
> situations of data in different locations out of sync, unless certain
> storage supports a bi-directional sync.
> 2. how should a reader/writer choose which root to access. To achieve
> that, I think we need some root location resolver interface, with some
> implementations like RegionBasedRootLocationResolver,
> AccessPolicyBasedRootLocationResolver, etc. (this is just my brainstorming)
>
> -Jack Ye
>
>
>
>
>
>
> On Mon, Aug 23, 2021 at 4:06 PM Yufei Gu <flyrain...@gmail.com> wrote:
>
>> @Ryan, how do these properties work with multiple table locations?
>>
>>    1.
>>
>>    write.metadata.path
>>    2.
>>
>>    write.folder-storage.path
>>    3.
>>
>>    write.object-storage.path
>>
>> The current logic with single table location is to honor these properties
>> on top of table location. In case of multiple roots(table locations), we
>> cannot mix them. For example, write.object-storage.path and table
>> location should be in the same region for the DR use case.
>>
>>
>> One of solutions to keep the similar logic is that we support all
>> properties for each root, like this
>>
>>
>> 1. Root1: (table location1, metadata path1, folder-storage path1,
>> object-storage path1)
>>
>> 2. Root2: (table location2, metadata path2, folder-storage path2,
>> object-storage path2)
>>
>> 3. Root3: (table location3, null, null, null)
>> What do you think?
>>
>> @Anjali, not sure supporting federation by using multiple roots is a good
>> idea. Can we create a partition with a specific location to distinguish
>> data between regions?
>>
>> On Mon, Aug 23, 2021 at 1:14 PM Anjali Norwood
>> <anorw...@netflix.com.invalid> wrote:
>>
>>> Hi Ryan, All,
>>>
>>> *"The more I think about this, the more I like the solution to add
>>> multiple table roots to metadata, rather than removing table roots. Adding
>>> a way to plug in a root selector makes a lot of sense to me and it ensures
>>> that the metadata is complete (table location is set in metadata) and that
>>> multiple locations can be used. Are there any objections or arguments
>>> against doing it that way?"*
>>>
>>> In the context of your comment on multiple locations above, I am
>>> thinking about the following scenarios:
>>> 1) Disaster recovery or low-latency use case where clients connect to
>>> the region geographically closest to them: In this case, multiple table
>>> roots represent a copy of the table, the copies may or may not be in sync.
>>> (Most likely active-active replication would be set up in this case and the
>>> copies are near-identical). A root level selector works/makes sense.
>>> 2) Data residency requirement: data must not leave a country/region. In
>>> this case, federation of data from multiple roots constitutes the entirety
>>> of the table.
>>> 3) One can also imagine combinations of 1 and 2 above where some
>>> locations need to be federated and some locations have data replicated from
>>> other locations.
>>>
>>> Curious how the 2nd and 3rd scenarios would be supported with this
>>> design.
>>>
>>> regards,
>>> Anjali.
>>>
>>>
>>>
>>> On Sun, Aug 22, 2021 at 11:22 PM Peter Vary <pv...@cloudera.com.invalid>
>>> wrote:
>>>
>>>>
>>>> @Ryan: If I understand correctly, currently there is a possibility to
>>>> change the root location of the table, and it will not change/move the old
>>>> data/metadata files created before the change, only the new data/metadata
>>>> files will be created in the new location.
>>>>
>>>> Are we planning to make sure that the tables with relative paths will
>>>> always contain every data/metadata file in single root folder?
>>>>
>>>> Here is a few scenarios which I am thinking about:
>>>> 1. Table T1 is created with relative path in HadoopCatalog/HiveCatalog
>>>> where the root location is L1, and this location is generated from the
>>>> TableIdentifier (at least in creation time)
>>>> 2. Data inserted to the table, so data files are created under L1, and
>>>> the metadata files contain R1 relative path to the current L1.
>>>> 3a. Table location is updated to L2 (Hive: ALTER TABLE T1 SET LOCATION
>>>> L2)
>>>> 3b. Table renamed to T2 (Hive: ALTER TABLE T1 RENAME TO T2) - Table
>>>> location is updated for Hive tables if the old location was the default
>>>> location.
>>>> 4. When we try to read this table we should read the old data/metadata
>>>> files as well.
>>>>
>>>> So in both cases we have to move the old data/metadata files around
>>>> like Hive does for the native tables, and for the tables with relative
>>>> paths we do not have to change the metadata other than the root path? Will
>>>> we do the same thing with other engines as well?
>>>>
>>>> Thanks,
>>>> Peter
>>>>
>>>> On Mon, 23 Aug 2021, 06:38 Yufei Gu, <flyrain...@gmail.com> wrote:
>>>>
>>>>> Steven, here is my understanding. It depends on whether you want to
>>>>> move the data. In the DR case, we do move the data, we expect data to be
>>>>> identical from time to time, but not always be. In the case of S3 aliases,
>>>>> different roots actually point to the same location, there is no data 
>>>>> move,
>>>>> and data is identical for sure.
>>>>>
>>>>> On Sun, Aug 22, 2021 at 8:19 PM Steven Wu <stevenz...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> For the multiple table roots, do we expect or ensure that the data
>>>>>> are identical across the different roots? or this is best-effort 
>>>>>> background
>>>>>> synchronization across the different roots?
>>>>>>
>>>>>> On Sun, Aug 22, 2021 at 11:53 AM Ryan Blue <b...@tabular.io> wrote:
>>>>>>
>>>>>>> Peter, I think that this feature would be useful when moving tables
>>>>>>> between root locations or when you want to maintain multiple root
>>>>>>> locations. Renames are orthogonal because a rename doesn't change the 
>>>>>>> table
>>>>>>> location. You may want to move the table after a rename, and this would
>>>>>>> help in that case. But actually moving data is optional. That's why we 
>>>>>>> put
>>>>>>> the table location in metadata.
>>>>>>>
>>>>>>> Anjali, DR is a big use case, but we also talked about directing
>>>>>>> accesses through other URLs, like S3 access points, table migration 
>>>>>>> (like
>>>>>>> the rename case), and background data migration (e.g. lifting files 
>>>>>>> between
>>>>>>> S3 regions). There are a few uses for it.
>>>>>>>
>>>>>>> The more I think about this, the more I like the solution to add
>>>>>>> multiple table roots to metadata, rather than removing table roots. 
>>>>>>> Adding
>>>>>>> a way to plug in a root selector makes a lot of sense to me and it 
>>>>>>> ensures
>>>>>>> that the metadata is complete (table location is set in metadata) and 
>>>>>>> that
>>>>>>> multiple locations can be used. Are there any objections or arguments
>>>>>>> against doing it that way?
>>>>>>>
>>>>>>> On Fri, Aug 20, 2021 at 9:00 AM Anjali Norwood
>>>>>>> <anorw...@netflix.com.invalid> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> This thread is about disaster recovery and relative paths, but I
>>>>>>>> wanted to ask an orthogonal but related question.
>>>>>>>> Do we see disaster recovery as the only (or main) use case for
>>>>>>>> multi-region?
>>>>>>>> Is data residency requirement a use case for anybody? Is it
>>>>>>>> possible to shard an iceberg table across regions? How is the location
>>>>>>>> managed in that case?
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Anjali.
>>>>>>>>
>>>>>>>> On Fri, Aug 20, 2021 at 12:20 AM Peter Vary
>>>>>>>> <pv...@cloudera.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> Sadly, I have missed the meeting :(
>>>>>>>>>
>>>>>>>>> Quick question:
>>>>>>>>> Was table rename / location change discussed for tables with
>>>>>>>>> relative paths?
>>>>>>>>>
>>>>>>>>> AFAIK when a table rename happens then we do not move old data /
>>>>>>>>> metadata files, we just change the root location of the new data / 
>>>>>>>>> metadata
>>>>>>>>> files. If I am correct about this then we might need to handle this
>>>>>>>>> differently for tables with relative paths.
>>>>>>>>>
>>>>>>>>> Thanks, Peter
>>>>>>>>>
>>>>>>>>> On Fri, 13 Aug 2021, 15:12 Anjali Norwood,
>>>>>>>>> <anorw...@netflix.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> Perfect, thank you Yufei.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Anjali
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 12, 2021 at 9:58 PM Yufei Gu <flyrain...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Anjali,
>>>>>>>>>>>
>>>>>>>>>>> Inline...
>>>>>>>>>>> On Thu, Aug 12, 2021 at 5:31 PM Anjali Norwood
>>>>>>>>>>> <anorw...@netflix.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the summary Yufei.
>>>>>>>>>>>> Sorry, if this was already discussed, I missed the meeting
>>>>>>>>>>>> yesterday.
>>>>>>>>>>>> Is there anything in the design that would prevent multiple
>>>>>>>>>>>> roots from being in different aws regions?
>>>>>>>>>>>>
>>>>>>>>>>> No. DR is the major use case of relative paths, if not the only
>>>>>>>>>>> one. So, it will support roots in different regions.
>>>>>>>>>>>
>>>>>>>>>>> For disaster recovery in the case of an entire aws region down
>>>>>>>>>>>> or slow, is metastore still a point of failure or can metastore be 
>>>>>>>>>>>> stood up
>>>>>>>>>>>> in a different region and could select a different root?
>>>>>>>>>>>>
>>>>>>>>>>> Normally, DR also requires a backup metastore, besides the
>>>>>>>>>>> storage(s3 bucket). In that case, the backup metastore will be in a
>>>>>>>>>>> different region along with the table files. For example, the 
>>>>>>>>>>> primary table
>>>>>>>>>>> is located in region A as well as its metastore, the backup table is
>>>>>>>>>>> located in region B as well as its metastore. The primary table 
>>>>>>>>>>> root points
>>>>>>>>>>> to a path in region A, while backup table root points to a path in 
>>>>>>>>>>> region B.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> regards,
>>>>>>>>>>>> Anjali.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 12, 2021 at 11:35 AM Yufei Gu <flyrain...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Here is a summary of yesterday's community sync-up.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei gave a brief update on disaster recovery requirements
>>>>>>>>>>>>> and the current progress of relative path approach.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ryan: We all agreed that relative path is the way for disaster
>>>>>>>>>>>>> recovery.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Multiple roots for the relative path*
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ryan proposed an idea to enable multiple roots for a table,
>>>>>>>>>>>>> basically, we can add a list of roots in table metadata, and use 
>>>>>>>>>>>>> a selector
>>>>>>>>>>>>> to choose different roots when we move the table from one place 
>>>>>>>>>>>>> to another.
>>>>>>>>>>>>> The selector reads a property to decide which root to use. The 
>>>>>>>>>>>>> property
>>>>>>>>>>>>> could be either from catalog or the table metadata, which is yet 
>>>>>>>>>>>>> to be
>>>>>>>>>>>>> decided.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here is an example I’d image:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. Root1: hdfs://nn:8020/path/to/the/table
>>>>>>>>>>>>>    2. Root2: s3://bucket1/path/to/the/table
>>>>>>>>>>>>>    3. Root3: s3://bucket2/path/to/the/table
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Relative path use case*
>>>>>>>>>>>>>
>>>>>>>>>>>>> We brainstormed use cases for relative paths. Please let us
>>>>>>>>>>>>> know if there are any other use cases.
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. Disaster Recovery
>>>>>>>>>>>>>    2. Jack: AWS s3 bucket alias
>>>>>>>>>>>>>    3. Ryan: fall-back use case. In case that the root1
>>>>>>>>>>>>>    doesn’t work, the table falls back to root2, then root3. As 
>>>>>>>>>>>>> Russell
>>>>>>>>>>>>>    mentioned, it is challenging to do snapshot expiration and 
>>>>>>>>>>>>> other table
>>>>>>>>>>>>>    maintenance actions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Timeline*
>>>>>>>>>>>>>
>>>>>>>>>>>>> In terms of timeline, relative path could be a feature in Spec
>>>>>>>>>>>>> V3, since Spec V1 and V2 assume absolute path in metadata.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Misc*
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. Miao: How is the relative path compatible with the
>>>>>>>>>>>>>    absolute path?
>>>>>>>>>>>>>    2. How do we migrate an existing table? Build a tool for
>>>>>>>>>>>>>    that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please let us know if you have any ideas, questions, or
>>>>>>>>>>>>> concerns.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Tabular
>>>>>>>
>>>>>>

-- 
Ryan Blue
Tabular

Reply via email to