Thanks, Kevin. If you need to customize the metadata path within the view location, then it makes more sense to me to use `write.metadata.path`.
On Tue, Jan 27, 2026 at 4:39 PM Kevin Liu <[email protected]> wrote: > Thanks for starting this thread, and for all the great discussions. > > I can chime in on use cases from the Microsoft OneLake perspective. We > have a requirement to customize the metadata path of the view metadata json > file. We want the file path to have a specific naming convention that can > be an arbitrary subdirectory name. > > Right now, using the `location` parameter, `metadata/` is added > automatically as the subdirectory for the json file [1], ie > {location}/metadata/view1.metadata.json > We want to customize the name of the subdirectory with > `write.metadata.path` [2], so the view will be stored in > {location}/{write.metadata.path}/view1.metadata.json > > In summary, I prefer option 2 to give writers more control over the exact > path of the metadata json. And this also follows the convention for > creating an Iceberg table with the write.metadata.path property. > > Best, > Kevin Liu > > > [1] > https://github.com/apache/iceberg/blob/1bf44d9fed8b99dd73115ec31659266248cbb363/core/src/main/java/org/apache/iceberg/view/BaseViewOperations.java#L173 > [2] > https://github.com/apache/iceberg/blob/1bf44d9fed8b99dd73115ec31659266248cbb363/core/src/main/java/org/apache/iceberg/view/BaseViewOperations.java#L166-L169 > > > On Tue, Jan 27, 2026 at 2:43 PM Ryan Blue <[email protected]> wrote: > >> I think it is okay to go with option 1, but I don't see a justification >> for why a change here is needed. Why break with existing convention when it >> should not matter to users what location is used for metadata? >> >> I'd prefer option 2, to let people configure this if they choose. But I >> think the best solution is to do nothing and encourage people not to care >> about where metadata files end up. Looks like the justification is "we have >> use cases". Could you share what those are? >> >> On Mon, Jan 26, 2026 at 1:27 PM Yufei Gu <[email protected]> wrote: >> >>> Given that the Iceberg community is moving toward a catalog based >>> approach, I think option 1 should work reasonably well. From a reader >>> perspective, there should be no behavior change, since the metadata path is >>> always resolved from the catalog, regardless of whether the underlying path >>> includes a metadata directory or not. For writers, as long as we converge >>> on the convention of not implicitly adding a metadata subdirectory, this >>> also seems acceptable. This is already how tables behave today. We have not >>> enforced a strict rule for table metadata paths in the spec, and different >>> impls have been able to interoperate without issues. >>> >>> Yufei >>> >>> >>> On Fri, Jan 23, 2026 at 11:35 AM Gábor Kaszab <[email protected]> >>> wrote: >>> >>>> Thank you All for your responses! >>>> Let me summarize what we have so far to see what potential next steps >>>> we can make. >>>> >>>> 1) Change 'location' property to not create a 'metadata/' subfolder >>>> - Doesn't seem to be a great issue that already written tables >>>> have 'metadata/' subfolder, while newer ones won't >>>> - Easy to implement in Iceberg code >>>> - Doing this will let us drop 'write.metadata.path' property >>>> My concern: >>>> - We only have control on Iceberg code, that means non-REST >>>> catalogs, but different REST catalog implementations could still create >>>> paths the old way. >>>> - Is there a way we can enforce this? Adding to spec seems >>>> overkill. >>>> >>>> 2) De-deprecate 'write.metadata.path' >>>> - Using this will let us to avoid having 'metadata/' folder in the >>>> path >>>> - Doc <https://iceberg.apache.org/docs/latest/view-configuration/> >>>> says "Base location for metadata files". Seems ambiguous if this means that >>>> we can have subfolders or not >>>> - Seems common sense to not add 'metadata/' folder, but there >>>> seems to be no strict guarantee that all REST catalog implementations >>>> follow this >>>> >>>> WDYT about next steps? Maybe go with 2) and be more explicit in the >>>> docs? >>>> >>>> Best Regards, >>>> Gabor Kaszab >>>> >>>> Ryan Blue <[email protected]> ezt írta (időpont: 2026. jan. 16., P, >>>> 1:03): >>>> >>>>> I think that the "non-catalog option" means that it works when you're >>>>> pointed at a particular metadata.json file. Commits always have to go >>>>> through some catalog-based mechanism (now that Hadoop/FS commits are >>>>> deprecated in the spec). >>>>> >>>>> On Thu, Jan 15, 2026 at 1:23 AM Péter Váry < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Yufei, >>>>>> >>>>>> I think keeping the non-catalog option available makes sense, and >>>>>> adding an extra check for this non-standard approach doesn’t seem like a >>>>>> big issue to me. That said, the community consensus appears to be >>>>>> gradually >>>>>> shifting away from the non-catalog approach. If you look at the REST >>>>>> planning mode discussion [1], we’re even considering Iceberg tables >>>>>> without >>>>>> any metadata stored on the underlying storage. It might be worth >>>>>> defining a >>>>>> consistent, general approach for decisions like these. >>>>>> >>>>>> Thanks, >>>>>> Peter >>>>>> >>>>>> [1] - [DISCUSS] REST: Scan Planning mode - >>>>>> https://lists.apache.org/thread/z1g4y8b4ogdrn0jjtjlgg7yjgxdbzpvg >>>>>> >>>>>> Yufei Gu <[email protected]> ezt írta (időpont: 2026. jan. 15., >>>>>> Cs, 0:17): >>>>>> >>>>>>> Peter, >>>>>>> Based on the discussion(sorry, I don't have a link to share) so far, >>>>>>> the community conclusion is that we should always keep the non catalog >>>>>>> option available. In other words, even catalogs are the recommended way >>>>>>> to >>>>>>> locate view metadata, non catalog based access remains a supported >>>>>>> option. >>>>>>> This gives us flexibility to support different deployment models. >>>>>>> >>>>>>> Yufei >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 14, 2026 at 2:20 AM Péter Váry < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> > For example, when locating the view metadata.json, clients would >>>>>>>> need to check two possible locations, "/location" and >>>>>>>> "/location/metadata". >>>>>>>> >>>>>>>> Do I understand correctly that users are expected to use the >>>>>>>> Catalog to locate metadata.json? In my opinion they should not rely on >>>>>>>> filenames to locate views.” >>>>>>>> >>>>>>>> Yufei Gu <[email protected]> ezt írta (időpont: 2026. jan. 14., >>>>>>>> Sze, 3:49): >>>>>>>> >>>>>>>>> Thanks Gabor for raising this. The alternative approach is >>>>>>>>> appealing, as it avoids supporting the same behavior through multiple >>>>>>>>> properties. My main concern is potential impact on existing use >>>>>>>>> cases. For >>>>>>>>> example, when locating the view metadata.json, clients would need to >>>>>>>>> check >>>>>>>>> two possible locations, "/location" and "/location/metadata". This >>>>>>>>> may not >>>>>>>>> be a significant issue, since the spec does not mandate the metadata >>>>>>>>> suffix, but it does introduce incompatibility with the existing >>>>>>>>> convention. >>>>>>>>> It would be helpful for others to chime in and share view use cases >>>>>>>>> that >>>>>>>>> rely on the "metadata" suffix today, and POV on whether it's OK to >>>>>>>>> adapt >>>>>>>>> the change. >>>>>>>>> >>>>>>>>> Yufei >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jan 13, 2026 at 5:23 AM Gábor Kaszab < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hey Iceberg Community, >>>>>>>>>> >>>>>>>>>> *TLDR* >>>>>>>>>> I'd like to de-deprecate the 'write.metadata.path' view property >>>>>>>>>> <https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/view/ViewProperties.java> >>>>>>>>>> and would like to see if there are any objections. >>>>>>>>>> >>>>>>>>>> *Details* >>>>>>>>>> We have use-cases where it's necessary to configure tables and >>>>>>>>>> views to have custom paths for metadata, in particular not to enforce >>>>>>>>>> adding the metadata/ folder under the hood. For both tables and >>>>>>>>>> views there >>>>>>>>>> is 'write.metadata.path' property that could achieve this. However, >>>>>>>>>> for >>>>>>>>>> views this is deprecated now. >>>>>>>>>> >>>>>>>>>> After talking to Yufei, the reason for this property being >>>>>>>>>> deprecated is that for views this is somewhat duplicated >>>>>>>>>> functionality with >>>>>>>>>> the 'location' property seemingly serving the same purpose. There >>>>>>>>>> are some >>>>>>>>>> differences though: >>>>>>>>>> - 'location': This gives the root folder of the view and >>>>>>>>>> metadata would be written under 'location'/metadata folder. >>>>>>>>>> - 'write.metadata.path': This has higher precedence over >>>>>>>>>> 'location' and metadata would be written directly into >>>>>>>>>> 'write.metadata.path' without the metadata/ folder. >>>>>>>>>> >>>>>>>>>> An *alternative approach* could be to drop 'write.metadata.path' >>>>>>>>>> and change the behaviour of 'location' to not create a metadata/ >>>>>>>>>> folder >>>>>>>>>> under the hood. The view spec doesn't mention explicitly that >>>>>>>>>> metadata/ >>>>>>>>>> folder is a requirement, however, it gives an example >>>>>>>>>> <https://iceberg.apache.org/view-spec/#appendix-a-an-example> >>>>>>>>>> where it is present and also mentions that it is there to follow the >>>>>>>>>> structure of the tables. So changing this would be confusing at >>>>>>>>>> least, but >>>>>>>>>> could be considered a breaking change too. I wouldn't go this way. >>>>>>>>>> >>>>>>>>>> So long story short, I'd like to keep 'write.metadata.path' for >>>>>>>>>> views despite the similar purpose as 'location', to avoid generating >>>>>>>>>> the >>>>>>>>>> metadata/ folder. >>>>>>>>>> >>>>>>>>>> Any feedback is appreciated! >>>>>>>>>> Gabor >>>>>>>>>> >>>>>>>>>
