Re: [DISCUSS] Apache Iceberg 1.7.0 Release Cutoff

2024-10-22 Thread Prashant Singh
Hi Russell, I recently stumbled across an issue in Iceberg Kafka-Connect which shows up when we are using UUID / FixedType, which leads to failure when we are actually writing the files, so makes it kinda unusable when we are using these data-types. I have a fix out for it already : https://github

Re: [DISCUSS] Discrepancy Between Iceberg Spec and Java Implementation for Snapshot summary's 'operation' key

2024-10-22 Thread rdb...@gmail.com
> For example, the `Snapshot` `summary` field is optional in V1 but required in V2. Therefore, the REST spec definition should mark the `summary` field as optional to support both versions. Yeah, this is technically true. But as I said in my first email, unless you have tables that are 5 years old

Re: Overwrite old properties on table replace with REST catalog

2024-10-22 Thread rdb...@gmail.com
Thanks Vladimir! Would you like to open a PR to make that change? It sounds like another good item to put into the "Implementation notes" section. On Sun, Oct 20, 2024 at 11:41 PM Vladimir Ozerov wrote: > Hi Jean-Baptiste, > > Agreed. REST spec looks good. I am talking about the general spec, wh

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Prashant Singh
+1 (non-binding) Regards, Prashant On Tue, Oct 22, 2024 at 10:50 AM John Zhuge wrote: > +1 (non-binding) > > John Zhuge > > > On Tue, Oct 22, 2024 at 9:45 AM Jack Ye wrote: > >> +1 (binding) >> >> Best, >> Jack Ye >> >> On Tue, Oct 22, 2024 at 9:32 AM Dmitri Bourlatchkov >> wrote: >> >>> Than

Re: Spec changes for deletion vectors

2024-10-22 Thread rdb...@gmail.com
Thanks for the careful consideration here, everyone. I completely agree that we do not want this to be confused as setting a precedent about delegating design decisions. I don't think the Delta community would do that either! We have to make decisions that are right for Iceberg. It sounds like we

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread rdb...@gmail.com
+1 (binding) Thanks for your work on this! On Tue, Oct 22, 2024 at 2:47 PM Prashant Singh wrote: > +1 (non-binding) > > Regards, > Prashant > > On Tue, Oct 22, 2024 at 10:50 AM John Zhuge wrote: > >> +1 (non-binding) >> >> John Zhuge >> >> >> On Tue, Oct 22, 2024 at 9:45 AM Jack Ye wrote: >>

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Amogh Jahagirdar
+1 (binding) On Tue, Oct 22, 2024 at 5:20 PM rdb...@gmail.com wrote: > +1 (binding) > > Thanks for your work on this! > > On Tue, Oct 22, 2024 at 2:47 PM Prashant Singh > wrote: > >> +1 (non-binding) >> >> Regards, >> Prashant >> >> On Tue, Oct 22, 2024 at 10:50 AM John Zhuge wrote: >> >>> +1

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Christian Thiel
+1 (non-binding). Great feature, thanks! Von: Amogh Jahagirdar <2am...@gmail.com> Gesendet: Tuesday, October 22, 2024 8:00:00 PM An: dev@iceberg.apache.org Betreff: Re: [VOTE] Endpoint for refreshing vended credentials +1 (binding) On Tue, Oct 22, 2024 at 5:20 PM

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Jean-Baptiste Onofré
Hi +1 (non binding) @Dmitri I don't think it's an issue: the endpoint can return multiple credentials, the client will take one. The credential refresh is not so costly though (and required to verify valid credential at retrieval time). Regards JB On Tue, Oct 22, 2024 at 5:09 PM Eduard Tudenhöf

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Jack Ye
+1 (binding) Best, Jack Ye On Tue, Oct 22, 2024 at 9:32 AM Dmitri Bourlatchkov wrote: > Thanks for the reply Eduard! > > I think it is fine to defer fine-tuning credential refreshes to a later PR. > > I'm upgrading my vote to +1 (non-binding). > > Cheers, > Dmitri. > > On Tue, Oct 22, 2024 at 1

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Dmitri Bourlatchkov
Thanks for the reply Eduard! I think it is fine to defer fine-tuning credential refreshes to a later PR. I'm upgrading my vote to +1 (non-binding). Cheers, Dmitri. On Tue, Oct 22, 2024 at 11:11 AM Eduard Tudenhöfner < etudenhoef...@apache.org> wrote: > Hey Dmitri, > > the idea behind the endpo

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread John Zhuge
+1 (non-binding) John Zhuge On Tue, Oct 22, 2024 at 9:45 AM Jack Ye wrote: > +1 (binding) > > Best, > Jack Ye > > On Tue, Oct 22, 2024 at 9:32 AM Dmitri Bourlatchkov > wrote: > >> Thanks for the reply Eduard! >> >> I think it is fine to defer fine-tuning credential refreshes to a later >> PR.

Re: [Discuss] Iceberg View Interoperability

2024-10-22 Thread Will Raschkowski
It’s all nascent, but we have a relatively solid experience going from Spark to Substrait [1], Substrait to Calcite [2], and Calcite to one of its supported SQL dialects [3]. All these translations are Java-based. [1]: https://github.com/substrait-io/substrait-java/pull/271 [2]: https://github.c

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Fanng
+1 (non-binding) Christian Thiel 于2024年10月23日周三 08:22写道: > +1 (non-binding). Great feature, thanks! > -- > *Von:* Amogh Jahagirdar <2am...@gmail.com> > *Gesendet:* Tuesday, October 22, 2024 8:00:00 PM > *An:* dev@iceberg.apache.org > *Betreff:* Re: [VOTE] Endpoint fo

Re: [DISCUSS] Apache Iceberg 1.7.0 Release Cutoff

2024-10-22 Thread Jean-Baptiste Onofré
Hi Prashant Thanks for the heads up. We still have time to review and include in the release. I will do a pass on your PR. Thanks ! Regards JB On Tue, Oct 22, 2024 at 9:29 PM Prashant Singh wrote: > > Hi Russell, > > I recently stumbled across an issue in Iceberg Kafka-Connect which shows up

Re: Overwrite old properties on table replace with REST catalog

2024-10-22 Thread Jean-Baptiste Onofré
I second Ryan here, it would be great to clarify in the "implementation notes" section. Thanks ! Regards JB On Wed, Oct 23, 2024 at 1:10 AM rdb...@gmail.com wrote: > > Thanks Vladimir! Would you like to open a PR to make that change? It sounds > like another good item to put into the "Implement

Re: [Discuss] Iceberg View Interoperability

2024-10-22 Thread Ajantha Bhat
Thanks Dan for the reply. but I'm not convinced using a procedure is a good idea or really moves > things forward in that direction. > I just feel like the procedure approach has a number of drawbacks > including, relying on the user to do the translation, being dependent on > Spark for defining t

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Eduard Tudenhöfner
Hey Dmitri, the idea behind the endpoint itself is really just to provide *valid* credentials for a given table when a client asks for them. If the server returned you two S3 credentials, the client will use the one with the longest prefix and if that credential expires, it will ask the server aga