Dan,
Thanks for the clarifications.
Looking forward to the sync.
- Wing Yew


On Fri, Aug 1, 2025 at 8:43 AM Daniel Weeks <dwe...@apache.org> wrote:

> Hey Wing Yu
>
> I see that you have been updating the Google doc containing the proposal.
>
>
> That's correct, I've been working with Talat to update the doc based on
> feedback from the comments and first round of discussion we had on this
> topic.
>
> Looking through it now, as far as I can tell, the basic idea (from the
>> original proposal) of inferring the table location from the path to the
>> current metadata.json has not changed. Is my reading correct?
>
>
> So far, nothing has changed about table location inference, but we will
> probably be revisiting this with respect to other updates/clarifications.
> There are still a couple open comments related to this point, but it is one
> of the main goals.
>
> You have added clarification around how the path to the metadata is
>> constructed from table location (from which the table location is thus
>> reverse engineered) and around path relativization, but the original idea
>> does not appear to have changed. In that case, the use case of having a
>> single copy of metadata but more than one copy of data (two or more
>> locations) is not supported by the proposal. This was the sticking point in
>> the last sync to discuss the proposal.
>
>
> I don't believe this was the sticking point from the original discussion.
> Having multiple copies/locations of the same data files under a single
> table's management is explicitly a non-goal.  It was discussed in the
> comments of the doc for caching/fallback use cases, but I think that's
> better handled by specific engine/fileio implementations.
>
> The main sticking points were confusion around the complexity of how paths
> are constructed/persisted and the interplay between table/metadata/data
> locations depending on how those values are set in the table metadata.
> Based on that feedback, we're suggesting some changes, which is primarily
> consist of: 1) defining path construction, resolution, and relativization
> separately, 2) making all paths relative to the table location (which
> simplifies resolution/relativization, 3) address confusing/complex issues
> like path separators and expectations around separators.
>
> We're still in the process of updating the document, but we will schedule
> another sync to discuss these updates in detail and address a few points
> that are still outstanding.
>
> Thanks,
> Dan
>
> On Thu, Jul 31, 2025 at 5:47 PM Wing Yew Poon <wyp...@cloudera.com.invalid>
> wrote:
>
>> Hi Daniel Weeks,
>> I see that you have been updating the Google doc containing the proposal.
>> Looking through it now, as far as I can tell, the basic idea (from the
>> original proposal) of inferring the table location from the path to the
>> current metadata.json has not changed. Is my reading correct?
>> You have added clarification around how the path to the metadata is
>> constructed from table location (from which the table location is thus
>> reverse engineered) and around path relativization, but the original idea
>> does not appear to have changed. In that case, the use case of having a
>> single copy of metadata but more than one copy of data (two or more
>> locations) is not supported by the proposal. This was the sticking point in
>> the last sync to discuss the proposal.
>> Do you intend to have another sync to continue the discussion?
>> Thanks,
>> Wing Yew
>>
>>
>> On Thu, Jul 10, 2025 at 4:41 PM Anurag Mantripragada
>> <amantriprag...@apple.com.invalid> wrote:
>>
>>> Thanks Kevin, yes, I see the recording link too but don’t have access. I
>>> have requested access.
>>>
>>>
>>> ~ Anurag Mantripragada
>>>
>>>
>>> On Jul 10, 2025, at 2:43 PM, Kevin Liu <kevinjq...@apache.org> wrote:
>>>
>>> Yes it was recorded. Dan or Talat should have the recording. I see
>>> there's already a link for the recording associated with the gcal event but
>>> I dont have access to it.
>>>
>>> Best,
>>> Kevin Liu
>>>
>>> On Thu, Jul 10, 2025 at 12:37 PM Anurag Mantripragada
>>> <amantriprag...@apple.com.invalid> wrote:
>>>
>>>> Hey folks, was the sync recorded? I missed it due to calendar sync
>>>> issues :(
>>>>
>>>>
>>>> ~ Anurag Mantripragada
>>>>
>>>> On Jul 7, 2025, at 6:27 PM, ally heev <allyh...@gmail.com> wrote:
>>>>
>>>> Thanks. I can see it now
>>>>
>>>> On Tue, Jul 8, 2025 at 12:37 AM Kevin Liu <kevinjq...@apache.org>
>>>> wrote:
>>>>
>>>>>
>>>>> I can see the new event on the dev calendar.
>>>>> [image: Screenshot 2025-07-07 at 12.04.08 PM.png]
>>>>>
>>>>> Subscribe to the "Iceberg Dev Events" calendar here:
>>>>> https://iceberg.apache.org/community/#iceberg-community-events
>>>>>
>>>>> Best,
>>>>> Kevin Liu
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 7, 2025 at 11:38 AM Daniel Weeks <dwe...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hey Ally (and everyone else).
>>>>>>
>>>>>> We hadn't scheduled the discussion for relative paths, but I just
>>>>>> added an event to the dev calendar for Thursday at 9am (PT).
>>>>>>
>>>>>> Let me know if you still don't see it on the calendar.
>>>>>>
>>>>>> -Dan
>>>>>>
>>>>>> On Sat, Jul 5, 2025 at 9:37 PM Jean-Baptiste Onofré <j...@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Talat
>>>>>>>
>>>>>>> Thanks for the update. I will do a new pass on the doc.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On Wed, May 28, 2025 at 12:13 AM Talat Uyarer
>>>>>>> <tal...@google.com.invalid> wrote:
>>>>>>> >
>>>>>>> > Hi, Iceberg Community,
>>>>>>> >
>>>>>>> > As mentioned at the last sync, Dan and I have been working on a
>>>>>>> proposal to add support for relative paths, which has been a long 
>>>>>>> requested
>>>>>>> feature. There have been a number of discussions/proposals over the 
>>>>>>> years,
>>>>>>> but we'd like to scope down and refocus effort to make some meaningful
>>>>>>> progress on this issue.
>>>>>>> >
>>>>>>> > Please take a look at the linked doc and provide feedback. We'd
>>>>>>> love to open up discussion on this topic at the next community sync and 
>>>>>>> we
>>>>>>> can hold one-off syncs on the topic if there's a lot of interest.
>>>>>>> >
>>>>>>> > You can access Iceberg's First V4 Spec change from here :)
>>>>>>> >
>>>>>>> > Proposal Issue: https://github.com/apache/iceberg/issues/13141
>>>>>>> > Doc: https://s.apache.org/iceberg-spec-relative-path
>>>>>>> >
>>>>>>> > Talat
>>>>>>>
>>>>>>
>>>>
>>>

Reply via email to