You are right, thanks Jack.

On Mon, May 20, 2024 at 8:06 AM Jack Ye <yezhao...@gmail.com> wrote:

> I believe this is already merged?
> https://github.com/apache/iceberg/pull/9782
>
> Best,
> Jack Ye
>
> On Sat, May 18, 2024 at 4:06 PM Pucheng Yang <py...@pinterest.com.invalid>
> wrote:
>
>> Hi all, is there an ETA for this? thanks
>>
>> On Wed, Dec 20, 2023 at 6:03 PM Renjie Liu <liurenjie2...@gmail.com>
>> wrote:
>>
>>> I think if servers provide a meaningful error message on expiration
>>>> hopefully, this would be a good first step in debugging.  I think saying
>>>> tokens should generally support O(Minutes) at least should cover most
>>>> use-cases?
>>>>
>>>
>>> Sounds reasonable to me. Clients just need to be aware that the token is
>>> for transient usage and should not store it for too long.
>>>
>>> On Thu, Dec 21, 2023 at 8:43 AM Micah Kornfield <emkornfi...@gmail.com>
>>> wrote:
>>>
>>>> Overall, I don't think it's a good idea to add parallel listing for
>>>>> things like tables and namespaces as it just adds complexity for an
>>>>> incredibly narrow (and possibly poorly designed) use case.
>>>>
>>>>
>>>> +1 I think that there are likely a few ways parallelization of table
>>>> and namespace listing can be incorporated in the future into the API if
>>>> necessary.
>>>>
>>>> I think the one place where parallelization is important immediately is
>>>> for Planning,  but that is already a separate thread.  Apologies if I
>>>> forked the conversation too far from that.
>>>>
>>>> On Wed, Dec 20, 2023 at 4:06 PM Daniel Weeks <dwe...@apache.org> wrote:
>>>>
>>>>> Overall, I don't think it's a good idea to add parallel listing for
>>>>> things like tables and namespaces as it just adds complexity for an
>>>>> incredibly narrow (and possibly poorly designed) use case.
>>>>>
>>>>> I feel we should leave it up to the server to define whether it will
>>>>> provide consistency across paginated listing and avoid
>>>>> bleeding time-travel like concepts (like 'asOf') into the API.  I really
>>>>> just don't see what practical value it provides as there are no explicit 
>>>>> or
>>>>> consistently held guarantees around these operations.
>>>>>
>>>>> I'd agree with Micah's argument that if the server does provide
>>>>> stronger guarantees, it should manage those via the opaque token and
>>>>> respond with meaningful errors if it cannot satisfy the internal
>>>>> constraints it imposes (like timeouts).
>>>>>
>>>>> It would help to have articulable use cases to really invest in more
>>>>> complexity in this area and I feel like we're drifting a little into the
>>>>> speculative at this point.
>>>>>
>>>>> -Dan
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 20, 2023 at 3:27 PM Micah Kornfield <emkornfi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I agree that this is not quite useful for clients at this moment. But
>>>>>>> I'm thinking that maybe exposing this will help debugging or diagnosing,
>>>>>>> user just need to be aware of this potential expiration.
>>>>>>
>>>>>>
>>>>>> I think if servers provide a meaningful error message on expiration
>>>>>> hopefully, this would be a good first step in debugging.  I think saying
>>>>>> tokens should generally support O(Minutes) at least should cover most
>>>>>> use-cases?
>>>>>>
>>>>>> On Tue, Dec 19, 2023 at 9:18 PM Renjie Liu <liurenjie2...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> If we choose to manage state on the server side, I recommend not
>>>>>>>> revealing the expiration time to the client, at least not for now. We 
>>>>>>>> can
>>>>>>>> introduce it when there's a practical need. It wouldn't constitute a
>>>>>>>> breaking change, would it?
>>>>>>>
>>>>>>>
>>>>>>> I agree that this is not quite useful for clients at this moment.
>>>>>>> But I'm thinking that maybe exposing this will help debugging or
>>>>>>> diagnosing, user just need to be aware of this potential expiration.
>>>>>>>
>>>>>>> On Wed, Dec 20, 2023 at 11:09 AM Xuanwo <xua...@apache.org> wrote:
>>>>>>>
>>>>>>>> > For the continuation token, I think one missing part is about the
>>>>>>>> expiration time of this token, since this may affect the state cleaning
>>>>>>>> process of the server.
>>>>>>>>
>>>>>>>> Some storage services use a continuation token as a binary
>>>>>>>> representation of internal states. For example, they serialize a 
>>>>>>>> structure
>>>>>>>> into binary and then perform base64 encoding. Services don't need to
>>>>>>>> maintain state, eliminating the need for state cleaning.
>>>>>>>>
>>>>>>>> > Do servers need to expose the expiration time to clients?
>>>>>>>>
>>>>>>>> If we choose to manage state on the server side, I recommend not
>>>>>>>> revealing the expiration time to the client, at least not for now. We 
>>>>>>>> can
>>>>>>>> introduce it when there's a practical need. It wouldn't constitute a
>>>>>>>> breaking change, would it?
>>>>>>>>
>>>>>>>> On Wed, Dec 20, 2023, at 10:57, Renjie Liu wrote:
>>>>>>>>
>>>>>>>> For the continuation token, I think one missing part is about the
>>>>>>>> expiration time of this token, since this may affect the state
>>>>>>>> cleaning process of the server. There are several things to discuss:
>>>>>>>>
>>>>>>>> 1. Should we leave it to the server to decide it or allow the
>>>>>>>> client to config in api?
>>>>>>>>
>>>>>>>> Personally I think it would be enough for the server to determine
>>>>>>>> it for now, since I don't see any usage to allow clients to set the
>>>>>>>> expiration time in api.
>>>>>>>>
>>>>>>>> 2. Do servers need to expose the expiration time to clients?
>>>>>>>>
>>>>>>>> Personally I think it would be enough to expose this through the
>>>>>>>> getConfig api to let users know this. For now there is no requirement 
>>>>>>>> for
>>>>>>>> per request expiration time.
>>>>>>>>
>>>>>>>> On Wed, Dec 20, 2023 at 2:49 AM Micah Kornfield <
>>>>>>>> emkornfi...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> IMO, parallelization needs to be a first class entity in the end
>>>>>>>> point/service design to allow for flexibility (I scanned through the
>>>>>>>> original proposal for the scan planning and it looked like it was on 
>>>>>>>> the
>>>>>>>> right track).  Using offsets for parallelization is problematic from 
>>>>>>>> both a
>>>>>>>> consistency and scalability perspective if you want to allow for
>>>>>>>> flexibility in implementation.
>>>>>>>>
>>>>>>>> In particular, I think the server needs an APIs like:
>>>>>>>>
>>>>>>>> DoScan - returns a list of partitions (represented by an opaque
>>>>>>>> entity).  The list of partitions should support pagination (in an ideal
>>>>>>>> world, it would be streaming).
>>>>>>>> GetTasksForPartition - Returns scan tasks for a partition (should
>>>>>>>> also be paginated/streaming, but this is up for debate).  I think it 
>>>>>>>> is an
>>>>>>>> important consideration to allow for empty partitions.
>>>>>>>>
>>>>>>>> With this implementation you don't necessarily require separate
>>>>>>>> server side state (objects in GCS should be sufficient), I think as 
>>>>>>>> Ryan
>>>>>>>> suggested, one implementation could be to have each partition 
>>>>>>>> correspond to
>>>>>>>> a byte-range in a manifest file for returning the tasks.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Micah
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023 at 9:55 AM Walaa Eldin Moustafa <
>>>>>>>> wa.moust...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Not necessarily. That is more of a general statement. The
>>>>>>>> pagination discussion forked from server side scan planning.
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023 at 9:52 AM Ryan Blue <b...@tabular.io> wrote:
>>>>>>>>
>>>>>>>> > With start/limit each client can query for own's chunk without
>>>>>>>> coordination.
>>>>>>>>
>>>>>>>> Okay, I understand now. Would you need to parallelize the client
>>>>>>>> for listing namespaces or tables? That seems odd to me.
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023 at 9:48 AM Walaa Eldin Moustafa <
>>>>>>>> wa.moust...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> > You can parallelize with opaque tokens by sending a starting
>>>>>>>> point for the next request.
>>>>>>>>
>>>>>>>> I meant we would have to wait for the server to return this
>>>>>>>> starting point from the past request? With start/limit each client can
>>>>>>>> query for own's chunk without coordination.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023 at 9:44 AM Ryan Blue <b...@tabular.io> wrote:
>>>>>>>>
>>>>>>>> > I think start and offset has the advantage of being
>>>>>>>> parallelizable (as compared to continuation tokens).
>>>>>>>>
>>>>>>>> You can parallelize with opaque tokens by sending a starting point
>>>>>>>> for the next request.
>>>>>>>>
>>>>>>>> > On the other hand, using "asOf" can be complex to  implement and
>>>>>>>> may be too powerful for the pagination use case
>>>>>>>>
>>>>>>>> I don't think that we want to add `asOf`. If the service chooses to
>>>>>>>> do this, it would send a continuation token that has the
>>>>>>>> information embedded.
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023 at 9:42 AM Walaa Eldin Moustafa <
>>>>>>>> wa.moust...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Can we assume it is the responsibility of the server to ensure
>>>>>>>> determinism (e.g., by caching the results along with query ID)? I think
>>>>>>>> start and offset has the advantage of being parallelizable (as 
>>>>>>>> compared to
>>>>>>>> continuation tokens). On the other hand, using "asOf" can be complex to
>>>>>>>>  implement and may be too powerful for the pagination use case 
>>>>>>>> (because it
>>>>>>>> allows to query the warehouse as of any point of time, not just now).
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Walaa.
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023 at 9:40 AM Ryan Blue <b...@tabular.io> wrote:
>>>>>>>>
>>>>>>>> I think you can solve the atomicity problem with a
>>>>>>>> continuation token and server-side state. In general, I don't think 
>>>>>>>> this is
>>>>>>>> a problem we should worry about a lot since pagination commonly has 
>>>>>>>> this
>>>>>>>> problem. But since we can build a system that allows you to solve it 
>>>>>>>> if you
>>>>>>>> choose to, we should go with that design.
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023 at 9:13 AM Micah Kornfield <
>>>>>>>> emkornfi...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Jack,
>>>>>>>> Some answers inline.
>>>>>>>>
>>>>>>>>
>>>>>>>> In addition to the start index approach, another potential simple
>>>>>>>> way to implement the continuation token is to use the last item name, 
>>>>>>>> when
>>>>>>>> the listing is guaranteed to be in lexicographic order.
>>>>>>>>
>>>>>>>>
>>>>>>>> I think this is one viable implementation, but the reason that the
>>>>>>>> token should be opaque is that it allows several different 
>>>>>>>> implementations
>>>>>>>> without client side changes.
>>>>>>>>
>>>>>>>> For example, if an element is added before the continuation token,
>>>>>>>> then all future listing calls with the token would always skip that
>>>>>>>> element.
>>>>>>>>
>>>>>>>>
>>>>>>>> IMO, I think this is fine, for some of the REST APIs it is likely
>>>>>>>> important to put constraints on atomicity requirements, for others 
>>>>>>>> (e.g.
>>>>>>>> list namespaces) I think it is OK to have looser requirements.
>>>>>>>>
>>>>>>>> If we want to enforce that level of atomicity, we probably want to
>>>>>>>> introduce another time travel query parameter (e.g. 
>>>>>>>> asOf=1703003028000) to
>>>>>>>> ensure that we are listing results at a specific point of time of the
>>>>>>>> warehouse, so the complete result list is fixed.
>>>>>>>>
>>>>>>>>
>>>>>>>> Time travel might be useful in some cases but I think it is
>>>>>>>> orthogonal to services wishing to have guarantees around
>>>>>>>> atomicity/consistency of results.  If a server wants to ensure that 
>>>>>>>> results
>>>>>>>> are atomic/consistent as of the start of the listing, it can embed the
>>>>>>>> necessary timestamp in the token it returns and parse it out when 
>>>>>>>> fetching
>>>>>>>> the next result.
>>>>>>>>
>>>>>>>> I think this does raise a more general point around service
>>>>>>>> definition evolution in general.  I think there likely need to be 
>>>>>>>> metadata
>>>>>>>> endpoints that expose either:
>>>>>>>> 1.  A version of the REST API supported.
>>>>>>>> 2.  Features the API supports (e.g. which query parameters are
>>>>>>>> honored for a specific endpoint).
>>>>>>>>
>>>>>>>> There are pros and cons to both approaches (apologies if I missed
>>>>>>>> this in the spec or if it has already been discussed).
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Micah
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023 at 8:25 AM Jack Ye <yezhao...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Yes I agree that it is better to not enforce the implementation to
>>>>>>>> favor any direction, and continuation token is probably better than
>>>>>>>> enforcing a numeric start index.
>>>>>>>>
>>>>>>>> In addition to the start index approach, another potential simple
>>>>>>>> way to implement the continuation token is to use the last item name, 
>>>>>>>> when
>>>>>>>> the listing is guaranteed to be in lexicographic order. Compared to the
>>>>>>>> start index approach, it does not need to worry about the change of 
>>>>>>>> start
>>>>>>>> index when something in the list is added or removed.
>>>>>>>>
>>>>>>>> However, the issue of concurrent modification could still exist
>>>>>>>> even with a continuation token. For example, if an element is added 
>>>>>>>> before
>>>>>>>> the continuation token, then all future listing calls with the token 
>>>>>>>> would
>>>>>>>> always skip that element. If we want to enforce that level of 
>>>>>>>> atomicity, we
>>>>>>>> probably want to introduce another time travel query parameter (e.g.
>>>>>>>> asOf=1703003028000) to ensure that we are listing results at a specific
>>>>>>>> point of time of the warehouse, so the complete result list is fixed. 
>>>>>>>> (This
>>>>>>>> is also the missing piece I forgot to mention in the start index 
>>>>>>>> approach
>>>>>>>> to ensure it works in distributed settings)
>>>>>>>>
>>>>>>>> -Jack
>>>>>>>>
>>>>>>>> On Tue, Dec 19, 2023, 9:51 AM Micah Kornfield <
>>>>>>>> emkornfi...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> I tried to cover these in more details at:
>>>>>>>> https://docs.google.com/document/d/1bbfoLssY1szCO_Hm3_93ZcN0UAMpf7kjmpwHQngqQJ0/edit
>>>>>>>>
>>>>>>>> On Sun, Dec 17, 2023 at 6:07 PM Renjie Liu <liurenjie2...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> +1 for this approach. I agree that the streaming approach requires
>>>>>>>> that http client and servers have http 2 streaming support, which is 
>>>>>>>> not
>>>>>>>> compatible with old clients.
>>>>>>>>
>>>>>>>> I share the same concern with Micah that only start/limit may not
>>>>>>>> be enough in a distributed environment where modification happens 
>>>>>>>> during
>>>>>>>> iterations. For compatibility, we need to consider several cases:
>>>>>>>>
>>>>>>>> 1. Old client <-> New Server
>>>>>>>> 2. New client <-> Old server
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Dec 16, 2023 at 6:51 AM Daniel Weeks <dwe...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I agree that we want to include this feature and I raised similar
>>>>>>>> concerns to what Micah already presented in talking with Ryan.
>>>>>>>>
>>>>>>>> For backward compatibility, just adding a start and limit implies a
>>>>>>>> deterministic order, which is not a current requirement of the REST 
>>>>>>>> spec.
>>>>>>>>
>>>>>>>> Also, we need to consider whether the start/limit would need to be
>>>>>>>> respected by the server.  If existing implementations simply return 
>>>>>>>> all the
>>>>>>>> results, will that be sufficient?  There are a few edge cases that 
>>>>>>>> need to
>>>>>>>> be considered here.
>>>>>>>>
>>>>>>>> For the opaque key approach, I think adding a query param to
>>>>>>>> trigger/continue and introducing a continuation token in
>>>>>>>> the ListNamespacesResponse might allow for more backward 
>>>>>>>> compatibility.  In
>>>>>>>> that scenario, pagination would only take place for clients who know 
>>>>>>>> how to
>>>>>>>> paginate and the ordering would not need to be deterministic.
>>>>>>>>
>>>>>>>> -Dan
>>>>>>>>
>>>>>>>> On Fri, Dec 15, 2023, 10:33 AM Micah Kornfield <
>>>>>>>> emkornfi...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Just to clarify and add a small suggestion:
>>>>>>>>
>>>>>>>> The behavior with no additional parameters requires the operations
>>>>>>>> to happen as they do today for backwards compatibility (i.e either all
>>>>>>>> responses are returned or a failure occurs).
>>>>>>>>
>>>>>>>> For new parameters, I'd suggest an opaque start token (instead of
>>>>>>>> specific numeric offset) that can be returned by the service and a 
>>>>>>>> limit
>>>>>>>> (as proposed above). If a start token is provided without a limit a
>>>>>>>> default limit can be chosen by the server.  Servers might return less 
>>>>>>>> than
>>>>>>>> limit (i.e. clients are required to check for a next token to 
>>>>>>>> determine if
>>>>>>>> iteration is complete).  This enables server side state if it is 
>>>>>>>> desired
>>>>>>>> but also makes deterministic listing much more feasible (deterministic
>>>>>>>> responses are essentially impossible in the face of changing data if 
>>>>>>>> only a
>>>>>>>> start offset is provided).
>>>>>>>>
>>>>>>>> In an ideal world, specifying a limit would result in streaming
>>>>>>>> responses being returned with the last part either containing a token 
>>>>>>>> if
>>>>>>>> continuation is necessary.  Given conversation on the other thread of
>>>>>>>> streaming, I'd imagine this is quite hard to model in an Open API REST
>>>>>>>> service.
>>>>>>>>
>>>>>>>> Therefore it seems like using pagination with token and offset
>>>>>>>> would be preferred.  If skipping someplace in the middle of the 
>>>>>>>> namespaces
>>>>>>>> is required then I would suggest modelling those as first class query
>>>>>>>> parameters (e.g. "startAfterNamespace")
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Micah
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 15, 2023 at 10:08 AM Ryan Blue <b...@tabular.io> wrote:
>>>>>>>>
>>>>>>>> +1 for this approach
>>>>>>>>
>>>>>>>> I think it's good to use query params because it can be
>>>>>>>> backward-compatible with the current behavior. If you get more than the
>>>>>>>> limit back, then the service probably doesn't support pagination. And 
>>>>>>>> if a
>>>>>>>> client doesn't support pagination they get the same results that they 
>>>>>>>> would
>>>>>>>> today. A streaming approach with a continuation link like in the scan 
>>>>>>>> API
>>>>>>>> discussion wouldn't work because old clients don't know to make a 
>>>>>>>> second
>>>>>>>> request.
>>>>>>>>
>>>>>>>> On Thu, Dec 14, 2023 at 10:07 AM Jack Ye <yezhao...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> During the conversation of the Scan API for REST spec, we touched
>>>>>>>> on the topic of pagination when REST response is large or takes time 
>>>>>>>> to be
>>>>>>>> produced.
>>>>>>>>
>>>>>>>> I just want to discuss this separately, since we also see the issue
>>>>>>>> for ListNamespaces and ListTables/Views, when integrating with a large
>>>>>>>> organization that has over 100k namespaces, and also a lot of tables in
>>>>>>>> some namespaces.
>>>>>>>>
>>>>>>>> Pagination requires either keeping state, or the response to be
>>>>>>>> deterministic such that the client can request a range of the full
>>>>>>>> response. If we want to avoid keeping state, I think we need to allow 
>>>>>>>> some
>>>>>>>> query parameters like:
>>>>>>>> - *start*: the start index of the item in the response
>>>>>>>> - *limit*: the number of items to be returned in the response
>>>>>>>>
>>>>>>>> So we can send a request like:
>>>>>>>>
>>>>>>>> *GET /namespaces?start=300&limit=100*
>>>>>>>>
>>>>>>>> *GET /namespaces/ns/tables?start=300&limit=100*
>>>>>>>>
>>>>>>>> And the REST spec should enforce that the response returned for the
>>>>>>>> paginated GET should be deterministic.
>>>>>>>>
>>>>>>>> Any thoughts on this?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Jack Ye
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>>
>>>>>>>> Xuanwo
>>>>>>>>
>>>>>>>>

Reply via email to