Re: Pagination for List APIs in the REST spec

2023-12-20 Thread Renjie Liu
> > I think if servers provide a meaningful error message on expiration > hopefully, this would be a good first step in debugging. I think saying > tokens should generally support O(Minutes) at least should cover most > use-cases? > Sounds reasonable to me. Clients just need to be aware that the

Re: Pagination for List APIs in the REST spec

2023-12-20 Thread Micah Kornfield
> > Overall, I don't think it's a good idea to add parallel listing for things > like tables and namespaces as it just adds complexity for an incredibly > narrow (and possibly poorly designed) use case. +1 I think that there are likely a few ways parallelization of table and namespace listing can

Re: Pagination for List APIs in the REST spec

2023-12-20 Thread Daniel Weeks
Overall, I don't think it's a good idea to add parallel listing for things like tables and namespaces as it just adds complexity for an incredibly narrow (and possibly poorly designed) use case. I feel we should leave it up to the server to define whether it will provide consistency across paginat

Re: Proposal for REST APIs for Iceberg table scans

2023-12-20 Thread Micah Kornfield
> > Also +1 for having a more strict definition of the shard. Having arbitrary > JSON was basically what we experimented with a string shard ID, and we > ended up with something very similar to the manifest plan task you describe > in the serialized ID string. IIUC the proposal correctly, I'd act

Re: Pagination for List APIs in the REST spec

2023-12-20 Thread Micah Kornfield
> > I agree that this is not quite useful for clients at this moment. But I'm > thinking that maybe exposing this will help debugging or diagnosing, user > just need to be aware of this potential expiration. I think if servers provide a meaningful error message on expiration hopefully, this would

Re: CDC Merge Failing - https://github.com/apache/iceberg/issues/8333

2023-12-20 Thread Ryan Blue
Great, thanks for letting us know it must have been a Spark bug. I think that tracks with what Amogh's testing showed. He wasn't able to reproduce it in 3.4.x, 3.5.x, or the latest in the 3.3.x line, only in 3.3.0. On Wed, Dec 20, 2023 at 9:04 AM Harshith Bolar wrote: > There was an issue in my

Re: [DISCUSS] Apache Iceberg 1.4.3

2023-12-20 Thread Jean-Baptiste Onofré
Hi all, FYI, I'm working with Eduard to include #9230 in 1.4.3 as well. We have updated the KEYS file with my sign/key, I'm doing a preparation step (build, verify, etc). Regards JB On Tue, Dec 19, 2023 at 8:38 AM Eduard Tudenhoefner wrote: > > Hey everyone, > > I'd like to start a discussion