Re: [DISCUSS] Apache Iceberg (java) 1.8.0 release

Thanks for the clarification, Amogh! Much appreciated! Would you mind adding these PRs to the 1.9 milestone, please? https://github.com/apache/iceberg/pull/11992 https://github.com/apache/iceberg/pull/11995 Thanks, Dmitri. On Fri, Jan 24, 2025 at 7:07 PM Amogh Jahagirdar <2am...@gmail.com> wro

Re: [DISCUSS] Apache Iceberg (java) 1.8.0 release

I think it would be nice to include the Auth Manager improvements into 1.8.0. Here's the most recent PR on that subject, but I think there will be a couple more (per previous discussions): https://github.com/apache/iceberg/pull/11844 Thanks, Dmitri. On Thu, Jan 9, 2025 at 1:52 AM Jean-Baptiste

Re: [Discuss] Proposal to Adjust Catalog Sync Schedule & Cancel Next Wednesday’s Meeting

Thanks for keeping track of this, Honah! +1 to keep the Wednesday 9 AM Pacific Time meeting every 3 weeks I'm ok to pause the 8 PM PST meeting - this time does not work for me personally. As for two time slots recurring every 6 weeks each, IMHO, if people in those meetings end up being distinct

Re: [DISCUSS] REST: Way to query if metadata pointer is the latest

Hi Gabor, I'm going to propose something that does not quitte align with your idea, so please bear with me. Both options in you email (A and B) assume that the engine is going to make freshness decisions based on the location of metadata. I see some conceptual rough edges here. A change in metad

Re: [DISCUSS] REST: OAuth2 Authentication Guide

eberg.apache.org > *Subject: *Re: [DISCUSS] REST: OAuth2 Authentication Guide > > Thanks Christian. Nice write-up! Authentication is essential to a > production env. It's great to document it well given a lot of people don't > necessarily have enough OAthen2 knowled

Re: [DISCUSS] Partial Metadata Loading

Hello All, This is an interesting discussion and I'd like to offer my perspective. When a REST Catalog is involved, the metadata is loaded and modified via the catalog API. So control over the metadata is delegated to the catalog. I'd argue that in this situation, catalogs should have the flexib

Re: [VOTE] Endpoint for refreshing vended credentials

e a differentiation between what specific credentials a client wants to > receive from the server. > > Thanks, > Eduard > > On Mon, Oct 21, 2024 at 6:36 PM Dmitri Bourlatchkov > wrote: > >> -0 (non-binding) >> >> If multiple credentials are vended for

Re: [VOTE] Endpoint for refreshing vended credentials

-0 (non-binding) If multiple credentials are vended for a table (which is allowed) the current API requires all credentials to be refreshed, when any of the previous credentials expires. I think this is suboptimal (but can probably be made to work in most practical cases). Cheers, Dmitri. On Mon

Re: [VOTE] Standardize vended credentials in OpenAPI spec

+1 (non-binding) Cheers, Dmitri. On Tue, Oct 15, 2024 at 1:15 PM Eduard Tudenhöfner wrote: > Hey everyone, > > I'd like to vote on #10722 , > which has been open for quite a while now. > I believe we're in agreement on how we want to standardize cre

Re: [DISCUSS] REST: Standardize vended credentials in Spec

itri for reviewing the PR and the doc. >>> >>> wrt ISO 8601 timestamps: I'd like to keep things consistent with other >>> places in the spec, which are typically defined as millisecond values. >>> >>> Thanks >>> Eduard >>> >>&g

Re: [DISCUSS] REST: Standardize vended credentials in Spec

> Eduard > > On Fri, Sep 20, 2024 at 4:46 PM Dmitri Bourlatchkov > wrote: > >> Thanks for proposing this improvement, Eduard! >> >> Overall it seems pretty reasonable to me. I added a few comments in GH >> and in the doc. >> >> One higher level p

Re: [DISCUSS] REST: Standardize vended credentials in Spec

Thanks for proposing this improvement, Eduard! Overall it seems pretty reasonable to me. I added a few comments in GH and in the doc. One higher level point I'd like to discuss is using ISO 8601 to format expiry timestamps (as opposed to numeric milliseconds values). This should hopefully make th

Re: [DISCUSS] REST: OAuth2 Authentication Guide

Hi Christian, Very nice proposal. Thanks for putting it together! I added some comments to the doc. I think it is related to PR #10753 [4], which proposes some foundational refactoring to the java REST client to enable further enhancements in OAuth2 flows. Cheers, Dmitri. [4] https://github.com

Re: [DISCUSS] REST: AuthManager API #10753

ou see between these two PRs? > The second PR aims at standardizing credentials used for different storage > providers in the OpenAPI spec. > > On Thu, Sep 12, 2024 at 3:09 PM Dmitri Bourlatchkov > wrote: > >> Apologies if I missed earlier discussions about this PR. >> &

[DISCUSS] REST: AuthManager API #10753

Apologies if I missed earlier discussions about this PR. Would it be possible to revive the reviews / exchange of ideas about the proposed AuthManager [1]? I think it is quite relevant in light of the recent changes to the credentials configuration [2], even though it deals with the authenticatio

Re: [VOTE] Merge guidelines for committing PRs

+1 (nb) Cheers, Dmitri. On Wed, Aug 28, 2024 at 12:29 PM Micah Kornfield wrote: > I propose to merge https://github.com/apache/iceberg/pull/10780 as a > starting place for describing community norms around merging/discussing PRs. > > We've discussed this [1] and gone through a bunch of revision

Re: [VOTE] Make namespace separator configurable in REST Spec

+1 (nb) to the spec change. Cheers, Dmitri. On Fri, Aug 16, 2024 at 4:31 AM Eduard Tudenhöfner wrote: > Hey everyone, > > as I mentioned on the DISCUSS thread, this is providing a simple path > forward for users of the V1 APIs (make the namespace separator > *configurable* instead of *hardcoded

Re: [DISCUSS] REST Endpoint discovery

proposal. In terms of the format, the current solution is >>>>>> simple enough. But I propose to use a trimmed openAPI's format directly. >>>>>> It >>>>>> won't add much cost as we can just take the minimum fields we want. But >>&g

Re: [DISCUSS] REST Endpoint discovery

;> On Wed, Aug 14, 2024 at 9:20 AM Eduard Tudenhöfner < >> etudenhoef...@apache.org> wrote: >> >>> Hey Dmitri, >>> >>> this proposal is the result of the community feedback from the >>> Capabilities proposal. Ultimately the capabilities turned out t

Re: [DISCUSS] REST Endpoint discovery

Would you mind adding a "related proposals" section to the doc to clarify this? Thanks, Dmitri. On Wed, Aug 14, 2024 at 11:15 AM Dmitri Bourlatchkov < dmitri.bourlatch...@dremio.com> wrote: > Hi Eduard, > > How is this proposal related to the Server Capabilities discuss

Re: [DISCUSS] REST Endpoint discovery

Hi Eduard, How is this proposal related to the Server Capabilities discussion? Thanks, Dmitri. On Wed, Aug 14, 2024 at 5:14 AM Eduard Tudenhöfner wrote: > Hey everyone, > > I'd like to propose a way for REST servers to communicate to clients what > endpoints it supports via a new *endpoints* f

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congrats, everyone :) On Tue, Aug 13, 2024 at 4:25 PM Russell Spitzer wrote: > Hi Y'all, > > It is my pleasure to let everyone know that the Iceberg PMC has voted to > have several talented individuals join us. > > So without further ado, please welcome Péter Váry, Amogh Jahagirdar and > Eduard

Re: [DISCUSS] Guidelines for committing PRs

I agree that a threshold on the mere number of changed files (20 in the PR) would be nice to avoid. On the other hand, I think leaving this to be a reviewer's call is fine from my perspective. Any committer could flag a change as large enough to be discussed on the dev list of go through the impro

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

t; On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov > wrote: > >> I do not think the spec is meant to allow only SQL representations, >> although it is certainly faviouring SQL in examples... It would be nice to >> add a non-SQL example, indeed. >> >> Cheers,

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

I do not think the spec is meant to allow only SQL representations, although it is certainly faviouring SQL in examples... It would be nice to add a non-SQL example, indeed. Cheers, Dmitri. On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong wrote: > Coming from PyIceberg, I have concerns as this p

Re: [DISCUSS] Changing namespace separator in REST spec

spec change I'll add the required impl > <https://github.com/apache/iceberg/pull/10905> (which just goes on top of > #10877 <https://github.com/apache/iceberg/pull/10877>) to send the *delim* > query > param. > > Eduard > > On Thu, Aug 8, 2024 at 12

Re: [DISCUSS] Changing namespace separator in REST spec

The idea of client-chosen separator char (?delim=.) sounds pretty reasonable to me. Nonetheless, I do not think this covers all the issues in putting namespaces in URI paths for servers running under the new Servlet spec. In particular, there are other chars that are considered "suspicious" by the

Re: [DISCUSS] Clarify in REST spec expected implementation behavior for unknown updates or requirements

I'd suggest using error code 422 (Unprocessable Content) [1] instead. 400 generally means the request was not well-formed (e.g. syntax error, or using invalid chars, etc.). However, here we have a situation when a client actually sends a valid request, but the server is not able to properly proces

Re: Meeting time for catalog community sync

The proposed schedule (Wednesdays) looks good to me. Cheers, Dmitri. On Sun, Jul 28, 2024 at 10:07 PM Jack Ye wrote: > Hi everyone, > > Looks like we have some general preference to do it on Wednesday when > there is no community sync, and also rotate the time time to accommodate > participatio

Re: [DISCUSS] Describing REST Server capabilities

g time on this topic, but this >>>> is so fundamental that I feel we should think through the alternatives. >>>> Would it be possible to at least document in the design proposal why the >>>> alternatives are not desirable, what are the pros and cons? >>&g

Re: [VOTE] Drop Java 8 support in Iceberg 1.7.0

I mean +1 to _drop_ java 8 in 1.7.0 :) On Fri, Jul 26, 2024 at 1:04 PM Dmitri Bourlatchkov < dmitri.bourlatch...@dremio.com> wrote: > +1 (nb) to Java 8 support in Iceberg 1.7.0. > > Cheers, > Dmitri. > > On Fri, Jul 26, 2024 at 7:29 AM Piotr Findeisen > wrote: >

Re: [VOTE] Drop Java 8 support in Iceberg 1.7.0

+1 (nb) to Java 8 support in Iceberg 1.7.0. Cheers, Dmitri. On Fri, Jul 26, 2024 at 7:29 AM Piotr Findeisen wrote: > Hi, > > Dropping support for building and running on Java 8 was discussed > previously on "Dropping JDK 8 support" and "Building with JDK 21" mail > threads. > > As JB kindly poi

Re: Meeting time for catalog community sync

As for me both Mondays (current) and Wednesdays (proposed) work fine. Cheers, Dmitri. On Wed, Jul 24, 2024 at 4:00 PM Jack Ye wrote: > Hi everyone, > > First of all, thanks everyone that has participated in the sync so far. > > As one of the action items for the catalog community sync meeting,

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats everybody! :) On Tue, Jul 23, 2024 at 10:23 AM Gang Wu wrote: > Congrats! > > On Tue, Jul 23, 2024 at 10:17 PM Russell Spitzer < > russell.spit...@gmail.com> wrote: > >> "so many" :) >> >> On Tue, Jul 23, 2024 at 9:14 AM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> This

Re: [VOTE] Release Apache Iceberg 1.6.0 RC1

+1 (nb) Verified the REST Client warning about OAuth2 URL (again, just in case). Cheers, Dmitri. On Thu, Jul 18, 2024 at 2:39 AM Jean-Baptiste Onofré wrote: > Hi everyone, > > I propose that we release the following RC as the official Apache > Iceberg 1.6.0 release. > > The commit ID is 229d8f

Re: [RESULT][VOTE] Merge table spec clarifications on time travel and equality deletes

I don't think the current spec > references the REST catalog, but I think the same issue occurs for table > specification. > > Cheers, > Micah > > On Fri, Jul 19, 2024 at 10:37 AM Dmitri Bourlatchkov > wrote: > >> Sorry for the late reply. The vote closed, so I&

Re: [RESULT][VOTE] Merge table spec clarifications on time travel and equality deletes

Sorry for the late reply. The vote closed, so I'll just post my comments without voting here. My reading of the spec change in PR #8982 [1] is that it is not normative. More specifically, REST catalog implementations that do not expose the full snapshot history in metadata JSON will not violate th

Re: [DISCUSS] Describing REST Server capabilities

or example table-spec v1 and just drop the manifest-file list in a table >>> snapshot leading to data loss? >>> >>> IMO capabilities must contain the table/view/... spec versions supported >>> by the server. >>> >>> There's also the concern a

Re: [VOTE] Release Apache Iceberg 1.6.0 RC0

+1 (nb) I verified OAuth2 in the REST Catalog with Spark / Keycloak (client secret) / Nessie. The token URI warning is prominently displayed, when `oauth2-server-uri` is not configured. When the token URI is configured, the client secret flow works fine with Keycloak. Cheers, Dmitri. On Fri, J

Re: [DISCUSS] Describing REST Server capabilities

-Jack > > > > > > > > On Thu, Jul 11, 2024 at 9:51 AM Jack Ye wrote: > >> Sorry I will take a look at the new comments later today. >> >> -Jack >> >> On Wed, Jul 10, 2024, 11:42 PM Eduard Tudenhöfner < >> etudenhoef...@apache.org> wrote

Re: [DISCUSS] Describing REST Server capabilities

er cases. Multi-table commits, for example, >>> are a building block for multi-statement transactions. If a service doesn't >>> support multi-table commits then we ideally want clients to know that ahead >>> of time so that they don't run a big transaction

Re: [DISCUSS] Describing REST Server capabilities

rror. In that case, why do we > need all these other capabilities like tables, remote-signing, etc. in the > first place? > > Maybe it is too extreme of a thought, but could anyone help describe how > the other capabilities could be used beyond potentially returning an error > earlier?

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard, > I've also added the 501 error to the response of the respective endpoints but worth mentioning that *HEAD* / *GET *requests must not return a 501 (this implies that the server impl would e.g. return a *404* in such a case)

Re: [Vote] Deprecate oauth tokens endpoint

+1 (non-binding) I previously posted comments in the GH PR (already addressed). Cheers, Dmitri. On Mon, Jul 8, 2024 at 12:15 PM Robert Stupp wrote: > Hi Everyone, > > I propose that we merge PR to "Deprecate oauth/tokens endpoint". > > The background and overall plan is discussed on this maili

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard, The capabilities PR looks good to me overall. I have a concern with the "oauth2" tag name though. I also commented [1] in GH but the comment appears to be closed by default :) I believe the term "oauth2" is confusing in this context with respect to RFC 6749 [2] as discussed in depth o

Re: [PROPOSAL] New REST Catalog Spec

Re: TCK and ref impl in the "spec" repo. I believe the reference implementation there does not have to be complete in the sense of actually managing Iceberg metadata. It only needs to be complete enough to allow TCK tests to pass. TCKs will be the normative checks for "real" catalog implementations

Re: [DISCUSS] June board report

And "soon" is realized by this PR :) https://github.com/projectnessie/nessie/pull/7043 On Fri, Jun 14, 2024 at 12:46 PM Dmitri Bourlatchkov < dmitri.bourlatch...@dremio.com> wrote: > Not sure if there were any other announcements but the Project Nessie site > has this >

Re: [DISCUSS] June board report

Not sure if there were any other announcements but the Project Nessie site has this https://projectnessie.org/blog/2024/05/13/nessie-integration-of-iceberg-rest/ Cheers, Dmitri. On Fri, Jun 14, 2024 at 12:43 PM Ryan Blue wrote: > JB, do you have a link to the announcement? I heard that it was p

Re: Addressing security questions in the Iceberg REST specification

I think Jack makes a good point with AWS SigV4 Authentication. I suppose, in REST Catalog implementations that support that auth method, the /v1/oauth/token Catalog REST endpoint is redundant. Cheers, Dmitri. On Thu, May 23, 2024 at 9:20 AM Jack Ye wrote: > I do not know enough details about OA

Re: Exploring Nessie Integration for Iceberg Tables: Insights Needed

Hi Pani, > My understanding is that Nessie primarily integrates with Apache Iceberg, offering versioned metadata management specifically for Iceberg tables ATM - yes. > if there is a need to maintain non-Iceberg tables alongside Iceberg tables, the Hive Metastore would still be necessary Using

Re: Inconsistency between REST spec and table/view spec

I'd like to make a slightly different point regarding metadata files. Currently the table spec does require that metadata be stored in a "file", however there is no way to discover that file outside of a Catalog. In other words, a client that is operating purely at the file level has no way of det

Re: Welcome new committers and PMC!

Congrats, Eduard , Amogh, and Szehon! On Wed, May 3, 2023 at 3:07 PM Ryan Blue wrote: > Hi everyone, > > I want to congratulate Amogh and Eduard, who were just added as Ierberg > committers and Szehon, who was just added to the PMC. Thanks for all your > contributions! > > Ryan > > -- > Ryan Blu

Re: Welcome new PMC members!

Congratulations Fokko, Steven, and Yufei! On Tue, Apr 11, 2023 at 5:22 PM Ryan Blue wrote: > Hi everyone! > > I want to congratulate 3 new PMC members, Fokko Driesprong, Steven Wu, and > Yufei Gu. Thanks for all your contributions! > > I was going to wait a little longer to announce, but since t

Re: Tableau Connectivity with Iceberg tables

Hi Sanket, I do not have the full picture of tools that support both Iceberg and Tableau, but Dremio, for example, does offer this connectivity. https://docs.dremio.com/software/client-applications/tableau/ https://docs.dremio.com/software/data-formats/apache-iceberg/ Cheers, Dmitri. On Thu, N

Re: [DISCUSS] Content of Iceberg 1.1.0 release

issues/5927> in MOR vectorized read. >> We will need it in the new release. >> >> Best, >> >> Yufei >> >> `This is not a contribution` >> >> >> On Fri, Oct 21, 2022 at 9:26 AM Dmitri Bourlatchkov < >> dmitri.bourlatch...@dremio.com&

Re: [DISCUSS] Content of Iceberg 1.1.0 release

nto 1.1.0, > then there will be a 1.2.0 release soonish as well. > > On Fri, Oct 21, 2022 at 4:55 PM Dmitri Bourlatchkov < > dmitri.bourlatch...@dremio.com> wrote: > >> I'd like to add https://github.com/apache/iceberg/issues/6031 to the 1.1 >> milestone (I believe p

Re: [DISCUSS] Content of Iceberg 1.1.0 release

I'd like to add https://github.com/apache/iceberg/issues/6031 to the 1.1 milestone (I believe permissions are indeed required for that). Most of the work will happen on the Nessie side. When the new API is ready, the expectation is that Iceberg will simply need to compile with a new version of Nes

Re: [DISCUSS] Automatic Code Formatting / Code Style / Enforcing Code Style

d >> >> On Mon, Jul 11, 2022 at 10:54 PM Ryan Blue wrote: >> >>> Okay, it sounds like there's mostly agreement for going with spotless. >>> Let's try that out. We'll work on some changes to add spotless so that >>> `spotlessApply` w

Re: [DISCUSS] Automatic Code Formatting / Code Style / Enforcing Code Style

My experience with the Google Code Style + Spotless was positive too. I'd be fine with another code style as long as it is "deterministic" (e.g. does not make changes on repeated execution) and works in IntelliJ IDEA / Eclipse / etc. Regarding cherry-picking into older branches, I think Robert's

Re: Welcome new PMC members!

Congrats Jack and Russell On Wed, Nov 17, 2021 at 7:12 PM Ryan Blue wrote: > Hi everyone, I want to welcome Jack Ye and Russell Spitzer to the Iceberg > PMC. They've both been amazing at reviewing and helping people in the > community and the PMC has decided to invite them to join. Congratulatio