Re: [DISCUSS] Formalized File IO Properties

Hi everyone, I tried to consolidate S3 properties in this PR: https://github.com/apache/iceberg/pull/11321 Hopefully we can start building a single source of truth from it. Thanks for your review, -Hsiang > On Aug 7, 2024, at 3:44 AM, Kevin Liu wrote: > > +1 on standardizing, and possibly e

Re: [DISCUSS] Formalized File IO Properties

+1 on standardizing, and possibly extending this to include catalog properties. On the PyIceberg side, a recent development is the ability to separate S3 FileIO configurations from the Glue Catalog configurations, with an optional configuration to use the same for both if specified. See Unified AW

Re: [DISCUSS] Formalized File IO Properties

Thanks you all. I'm going to prepare a proposal PR for this. On Fri, Jul 12, 2024, at 10:06, Honah J. wrote: > Hello everyone, > > Thank you all for the valuable insights. I am also +1 on having standardized > names for File IO properties. Creating a dedicated section to summarize > property n

Re: [DISCUSS] Formalized File IO Properties

Hello everyone, Thank you all for the valuable insights. I am also +1 on having standardized names for File IO properties. Creating a dedicated section to summarize property names in the Java implementation is a good starting point. Since pyiceberg, icebergRust, and IcebergGolang will support only

Re: [DISCUSS] Formalized File IO Properties

I don't know what the recommended way to start standardizing is. We can start a proposal for each context or have one proposal to handle all. Suggested contexts to start with: - Rest Catalog - FileIO I believe that most of the other cases are supported by the configuration topic in the Table

Re: [DISCUSS] Formalized File IO Properties

Hi, Also +1 on standardizing properties. I'm looking forward to this discussion topic. In particular, REST catalog properties imho should be standardized with the "rest." prefix, and REST auth properties should imho have a prefix like "rest.auth..", e.g. "rest.auth.oauth2.issuer-url". Thanks, Ale

Re: [DISCUSS] Formalized File IO Properties

Sounds reasonable to me On Wed, Jul 10, 2024 at 9:28 AM Renjie Liu wrote: > Hi: > > +1 for standardizing iceberg properties. This will help to align different > language implementations. > > On Wed, Jul 10, 2024 at 9:44 PM wrote: > >> Hello Everyone, >> >> I was considering discussing the stand

Re: [DISCUSS] Formalized File IO Properties

Hi: +1 for standardizing iceberg properties. This will help to align different language implementations. On Wed, Jul 10, 2024 at 9:44 PM wrote: > Hello Everyone, > > I was considering discussing the standardization of Iceberg properties, > and I believe this thread could be a great place to sta

Re: [DISCUSS] Formalized File IO Properties

Hello Everyone, I was considering discussing the standardization of Iceberg properties, and I believe this thread could be a great place to start. I'm writing an Iceberg client in Elixir and using the Java, Python, and Rust implementations as references. However, I've had some difficulty determ

Re: [DISCUSS] Formalized File IO Properties

Hey Xuanwo, Thanks for raising this. - The S3 properties are largely covered under the S3FileIO page: https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks like some important ones are missing indeed. I've raised an issue here

[DISCUSS] Formalized File IO Properties

Hello everyone I've been working on the iceberg-rust FileIO recently and have found it challenging to identify all the necessary IO properties we need to support. For instance, consider AWS S3. There are no documents specifying which properties are supported by S3. The only relevant documentat