We've run into this same issue in a number of cases previously and I think where we want to ideally go (in most cases) is language agnostic properties/values. For example, the property for `catalog-impl` <https://github.com/apache/iceberg/blob/be6e9daf8901cdee63197e77fcf95624bb694f39/core/src/main/java/org/apache/iceberg/CatalogProperties.java#L27> we prefer using the aliases (e.g. rest, hive, jdbc) as opposed to the fully qualified java class name. For something like `write.location-provider.impl` <https://github.com/apache/iceberg/blob/be6e9daf8901cdee63197e77fcf95624bb694f39/core/src/main/java/org/apache/iceberg/TableProperties.java#L257>, you could extend the same logic and use: 'partitioned' (hive-like) or 'object-store'. Similarly for `io-impl` <https://github.com/apache/iceberg/blob/be6e9daf8901cdee63197e77fcf95624bb694f39/core/src/main/java/org/apache/iceberg/io/FileIOParser.java#L32>, we could use aliases (e.g. S3, GCP, ADFSS, HDFS). This avoids the need to use language specific implementations.
I feel like there will always be cases that are language specific, but we should be on the lookout for being able to generalize the configurations and steer away from dynamic loading (though there are specific cases where it may still be necessary/beneficial). -Dan On Fri, Jan 24, 2025 at 10:47 PM Fokko Driesprong <fo...@apache.org> wrote: > We formalized some of the configurations in the REST spec > <https://github.com/apache/iceberg/blob/8ce1b32ecc2d9ed3e5000ae6a5eb663037eabfe8/open-api/rest-catalog-open-api.yaml#L3195>. > Still, I would be reluctant to try to create an exhaustive list of > configuration options since it will be tough to make this complete, and per > language, we rely on what is to be supported based on the implementations. > > I agree with Matt. If a language doesn't support it, I don't think we > should force it. The whole idea behind having different implementations is > to be language agnostic. > > Kind regards, > Fokko > > Op vr 24 jan 2025 om 07:35 schreef Matt Topol <zotthewiz...@gmail.com>: > >> My initial comment on this is that neither Go nor Rust, to my knowledge, >> support dynamic loading in this way. So formalization of rs-, cpp- or go- >> prefixes like this wouldn't be useful in any meaningful way as they would >> need to be linked in ahead of time for it to work. >> >> There are probably very hacky ways to support it, but I don't think it >> would be a good idea to support this as a universal idea for multiple >> languages at the spec level. >> >> --Matt >> >> On Fri, Jan 24, 2025, 1:02 AM Gang Wu <ust...@gmail.com> wrote: >> >>> Generally it makes sense to define separate language-specific >>> configurations. >>> >>> I think we need to think about the following items: >>> 1. Is it python-specific to add the prefix? Should Rust/Go be -rs/-go as >>> the convention? >>> 2. Which part of the spec is the best place to describe this? It seems >>> that we need to include all affected configurations and add guidelines for >>> future ones. >>> >>> Best, >>> Gang >>> >>> On Thu, Jan 23, 2025 at 3:58 AM Kevin Liu <kevinjq...@apache.org> wrote: >>> >>>> Hi everyone, >>>> >>>> I’d like to open a discussion on the naming scheme for configuration >>>> parameters across different languages. >>>> >>>> While working on the LocationProvider implementation in PyIceberg ( >>>> #1452 <https://github.com/apache/iceberg-python/pull/1452>), we >>>> encountered a challenge with naming configuration parameters for custom >>>> implementations. In the Java library, the parameter >>>> `write.location-provider.impl` >>>> <https://github.com/apache/iceberg/blob/be6e9daf8901cdee63197e77fcf95624bb694f39/core/src/main/java/org/apache/iceberg/TableProperties.java#L257> >>>> is used to configure a custom LocationProvider implementation. However, for >>>> cross-language compatibility, we couldn’t reuse this parameter directly. >>>> Configuration parameters defined in one language are not automatically >>>> compatible with others. >>>> >>>> To address this in PyIceberg, we chose to use >>>> `write.py-location-provider.impl` as the parameter name. This approach >>>> aligns with PyIceberg’s existing naming convention for custom >>>> implementations, such as `py-catalog-impl` (vs. `catalog-impl` >>>> <https://github.com/apache/iceberg/blob/be6e9daf8901cdee63197e77fcf95624bb694f39/core/src/main/java/org/apache/iceberg/CatalogProperties.java#L27>) >>>> for custom catalog implementations and `py-io-impl` (vs. `io-impl` >>>> <https://github.com/apache/iceberg/blob/be6e9daf8901cdee63197e77fcf95624bb694f39/core/src/main/java/org/apache/iceberg/io/FileIOParser.java#L32>) >>>> for custom FileIO implementations. >>>> >>>> I’d like to propose formalizing the use of the `py-` prefix for >>>> Python-specific configurations and extending this naming scheme to other >>>> languages, such as Rust, Go, and C++. This would create a consistent, >>>> predictable pattern for language-specific configuration parameters across >>>> the project. >>>> >>>> Looking forward to hearing your thoughts! >>>> >>>> Best, >>>> Kevin Liu >>>> >>>>
