> This is a stricter requirement than we have for Iceberg tables. Are we really going to enforce this? How will we do it efficiently? If not, let's not put it in the spec.
The efficiency is a good point, if we are supporting arbitrary nested namespaces, the efficiency is definitely a concern. Maybe we can restrict that for generic tables, but I think it would be good for us to stay consistent with Iceberg tables on this, since we share the namespace concept. We can exclude this from the spec. However, I do think that is the right restriction to put for both Iceberg and generic tables for better security guarantee, maybe we can do a separate discussion on this topic. >It would be trivial to add update support for generic entities. Why canonicalize this restriction in the spec? We don't, for example, currently detail a restriction around the fact that you can't change a generic table's format. Sure, we don't have to mention this in the Spec. > generic tables are a catch-all type not specific to any format (including Iceberg) Generic Table APis today have a clear separation with Iceberg table APIs. I don't think we want to close the door for that, and that is also why I think "generic" is a good name. However, if want to move on to include certain semantics for iceberg tables, for example, showing iceberg tables in list tables, there will be a repurpose of the API endpoints, and I think it would be more proper to move on for V2 spec, because people will have to use those APIs differently. > GenericTableEntity is the type I'm most likely to look to for the conversion service, which means it will indeed be used to represent Iceberg tables. For conversion, if we are converting a table to an iceberg table, and the table only has one root location, the target iceberg table will also have one root location, so I don't see a problem with this. If we are converting from an iceberg table to a target format that only supports one location, I don't see a problem also. Even with Iceberg table spec today, I believe the locations it has are : top level location, metadata.path, and data.path. I don't think that can be achieved with an array of locations also, Because it can not tell which path is for metadata, which path is for data, I don't think relying on the size and position of an array is a good idea, and that extra path information can continue be represented with generic tables using properties and top level location. Even with all those location configurations, I don't think Iceberg spec is capturing all locations a table can have, because every snapshot can potentially write into a different location, and those are not tracked anywhere by anyone today. Furthermore it might require information more than just a location, for example, it might need to be associated with the snapshot. I know Dennis was discussing a multi-location spec for Iceberg, but the information needed seems more complicated than just a list of locations. Table with multiple location support seems a bigger topic that requires much more thought to me, again I am not saying we shouldn't support it in the future, but I think we should put more thought into how tables with multiple locations work before we start supporting those. > The multi-location support in Polaris seems not very well also, the overlap check and credential vending seems all done with one location Sorry, i think i misread the caller of the code for the overlap check. Dennis mentioned that we only use one location for credential, but it might be for something else. Best Regards, Yun On Thu, May 22, 2025 at 3:08 PM Eric Maynard <eric.w.mayn...@gmail.com> wrote: > > i meant no two tables under the same catalog can have the same location > > This is a stricter requirement than we have for Iceberg tables. Are we > really going to enforce this? How will we do it efficiently? If not, let's > not put it in the spec. > > > we do not have any update support > > It would be trivial to add update support for generic entities. Why > canonicalize this restriction in the spec? We don't, for example, currently > detail a restriction around the fact that you can't change a generic > table's format. > > > generic tables are designed for non-Iceberg tables today, > > I don't actually think this is true. There's nothing about generic tables > that make them more useful for Delta tables than Iceberg tables, for > example. On the contrary, I initially proposed the name "generic" in part > to capture that generic tables are a catch-all type not specific to any > format (including Iceberg). More practically, GenericTableEntity is the > type I'm most likely to look to for the conversion service, which means it > will indeed be used to represent Iceberg tables. > > > The multi-location support in Polaris seems not very well also, the > overlap check and credential vending seems all done with one location > > This is not true. >
