Hi all, is there an ETA for this? thanks On Wed, Dec 20, 2023 at 6:03 PM Renjie Liu <liurenjie2...@gmail.com> wrote:
> I think if servers provide a meaningful error message on expiration >> hopefully, this would be a good first step in debugging. I think saying >> tokens should generally support O(Minutes) at least should cover most >> use-cases? >> > > Sounds reasonable to me. Clients just need to be aware that the token is > for transient usage and should not store it for too long. > > On Thu, Dec 21, 2023 at 8:43 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > >> Overall, I don't think it's a good idea to add parallel listing for >>> things like tables and namespaces as it just adds complexity for an >>> incredibly narrow (and possibly poorly designed) use case. >> >> >> +1 I think that there are likely a few ways parallelization of table and >> namespace listing can be incorporated in the future into the API if >> necessary. >> >> I think the one place where parallelization is important immediately is >> for Planning, but that is already a separate thread. Apologies if I >> forked the conversation too far from that. >> >> On Wed, Dec 20, 2023 at 4:06 PM Daniel Weeks <dwe...@apache.org> wrote: >> >>> Overall, I don't think it's a good idea to add parallel listing for >>> things like tables and namespaces as it just adds complexity for an >>> incredibly narrow (and possibly poorly designed) use case. >>> >>> I feel we should leave it up to the server to define whether it will >>> provide consistency across paginated listing and avoid >>> bleeding time-travel like concepts (like 'asOf') into the API. I really >>> just don't see what practical value it provides as there are no explicit or >>> consistently held guarantees around these operations. >>> >>> I'd agree with Micah's argument that if the server does provide stronger >>> guarantees, it should manage those via the opaque token and respond with >>> meaningful errors if it cannot satisfy the internal constraints it imposes >>> (like timeouts). >>> >>> It would help to have articulable use cases to really invest in more >>> complexity in this area and I feel like we're drifting a little into the >>> speculative at this point. >>> >>> -Dan >>> >>> >>> >>> On Wed, Dec 20, 2023 at 3:27 PM Micah Kornfield <emkornfi...@gmail.com> >>> wrote: >>> >>>> I agree that this is not quite useful for clients at this moment. But >>>>> I'm thinking that maybe exposing this will help debugging or diagnosing, >>>>> user just need to be aware of this potential expiration. >>>> >>>> >>>> I think if servers provide a meaningful error message on expiration >>>> hopefully, this would be a good first step in debugging. I think saying >>>> tokens should generally support O(Minutes) at least should cover most >>>> use-cases? >>>> >>>> On Tue, Dec 19, 2023 at 9:18 PM Renjie Liu <liurenjie2...@gmail.com> >>>> wrote: >>>> >>>>> If we choose to manage state on the server side, I recommend not >>>>>> revealing the expiration time to the client, at least not for now. We can >>>>>> introduce it when there's a practical need. It wouldn't constitute a >>>>>> breaking change, would it? >>>>> >>>>> >>>>> I agree that this is not quite useful for clients at this moment. But >>>>> I'm thinking that maybe exposing this will help debugging or diagnosing, >>>>> user just need to be aware of this potential expiration. >>>>> >>>>> On Wed, Dec 20, 2023 at 11:09 AM Xuanwo <xua...@apache.org> wrote: >>>>> >>>>>> > For the continuation token, I think one missing part is about the >>>>>> expiration time of this token, since this may affect the state cleaning >>>>>> process of the server. >>>>>> >>>>>> Some storage services use a continuation token as a binary >>>>>> representation of internal states. For example, they serialize a >>>>>> structure >>>>>> into binary and then perform base64 encoding. Services don't need to >>>>>> maintain state, eliminating the need for state cleaning. >>>>>> >>>>>> > Do servers need to expose the expiration time to clients? >>>>>> >>>>>> If we choose to manage state on the server side, I recommend not >>>>>> revealing the expiration time to the client, at least not for now. We can >>>>>> introduce it when there's a practical need. It wouldn't constitute a >>>>>> breaking change, would it? >>>>>> >>>>>> On Wed, Dec 20, 2023, at 10:57, Renjie Liu wrote: >>>>>> >>>>>> For the continuation token, I think one missing part is about the >>>>>> expiration time of this token, since this may affect the state >>>>>> cleaning process of the server. There are several things to discuss: >>>>>> >>>>>> 1. Should we leave it to the server to decide it or allow the client >>>>>> to config in api? >>>>>> >>>>>> Personally I think it would be enough for the server to determine it >>>>>> for now, since I don't see any usage to allow clients to set the >>>>>> expiration >>>>>> time in api. >>>>>> >>>>>> 2. Do servers need to expose the expiration time to clients? >>>>>> >>>>>> Personally I think it would be enough to expose this through the >>>>>> getConfig api to let users know this. For now there is no requirement for >>>>>> per request expiration time. >>>>>> >>>>>> On Wed, Dec 20, 2023 at 2:49 AM Micah Kornfield < >>>>>> emkornfi...@gmail.com> wrote: >>>>>> >>>>>> IMO, parallelization needs to be a first class entity in the end >>>>>> point/service design to allow for flexibility (I scanned through the >>>>>> original proposal for the scan planning and it looked like it was on the >>>>>> right track). Using offsets for parallelization is problematic from >>>>>> both a >>>>>> consistency and scalability perspective if you want to allow for >>>>>> flexibility in implementation. >>>>>> >>>>>> In particular, I think the server needs an APIs like: >>>>>> >>>>>> DoScan - returns a list of partitions (represented by an opaque >>>>>> entity). The list of partitions should support pagination (in an ideal >>>>>> world, it would be streaming). >>>>>> GetTasksForPartition - Returns scan tasks for a partition (should >>>>>> also be paginated/streaming, but this is up for debate). I think it is >>>>>> an >>>>>> important consideration to allow for empty partitions. >>>>>> >>>>>> With this implementation you don't necessarily require separate >>>>>> server side state (objects in GCS should be sufficient), I think as Ryan >>>>>> suggested, one implementation could be to have each partition correspond >>>>>> to >>>>>> a byte-range in a manifest file for returning the tasks. >>>>>> >>>>>> Thanks, >>>>>> Micah >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 19, 2023 at 9:55 AM Walaa Eldin Moustafa < >>>>>> wa.moust...@gmail.com> wrote: >>>>>> >>>>>> Not necessarily. That is more of a general statement. The pagination >>>>>> discussion forked from server side scan planning. >>>>>> >>>>>> On Tue, Dec 19, 2023 at 9:52 AM Ryan Blue <b...@tabular.io> wrote: >>>>>> >>>>>> > With start/limit each client can query for own's chunk without >>>>>> coordination. >>>>>> >>>>>> Okay, I understand now. Would you need to parallelize the client for >>>>>> listing namespaces or tables? That seems odd to me. >>>>>> >>>>>> On Tue, Dec 19, 2023 at 9:48 AM Walaa Eldin Moustafa < >>>>>> wa.moust...@gmail.com> wrote: >>>>>> >>>>>> > You can parallelize with opaque tokens by sending a starting point >>>>>> for the next request. >>>>>> >>>>>> I meant we would have to wait for the server to return this starting >>>>>> point from the past request? With start/limit each client can query for >>>>>> own's chunk without coordination. >>>>>> >>>>>> >>>>>> On Tue, Dec 19, 2023 at 9:44 AM Ryan Blue <b...@tabular.io> wrote: >>>>>> >>>>>> > I think start and offset has the advantage of being parallelizable >>>>>> (as compared to continuation tokens). >>>>>> >>>>>> You can parallelize with opaque tokens by sending a starting point >>>>>> for the next request. >>>>>> >>>>>> > On the other hand, using "asOf" can be complex to implement and >>>>>> may be too powerful for the pagination use case >>>>>> >>>>>> I don't think that we want to add `asOf`. If the service chooses to >>>>>> do this, it would send a continuation token that has the >>>>>> information embedded. >>>>>> >>>>>> On Tue, Dec 19, 2023 at 9:42 AM Walaa Eldin Moustafa < >>>>>> wa.moust...@gmail.com> wrote: >>>>>> >>>>>> Can we assume it is the responsibility of the server to ensure >>>>>> determinism (e.g., by caching the results along with query ID)? I think >>>>>> start and offset has the advantage of being parallelizable (as compared >>>>>> to >>>>>> continuation tokens). On the other hand, using "asOf" can be complex to >>>>>> implement and may be too powerful for the pagination use case (because >>>>>> it >>>>>> allows to query the warehouse as of any point of time, not just now). >>>>>> >>>>>> Thanks, >>>>>> Walaa. >>>>>> >>>>>> On Tue, Dec 19, 2023 at 9:40 AM Ryan Blue <b...@tabular.io> wrote: >>>>>> >>>>>> I think you can solve the atomicity problem with a continuation token >>>>>> and server-side state. In general, I don't think this is a problem we >>>>>> should worry about a lot since pagination commonly has this problem. But >>>>>> since we can build a system that allows you to solve it if you choose to, >>>>>> we should go with that design. >>>>>> >>>>>> On Tue, Dec 19, 2023 at 9:13 AM Micah Kornfield < >>>>>> emkornfi...@gmail.com> wrote: >>>>>> >>>>>> Hi Jack, >>>>>> Some answers inline. >>>>>> >>>>>> >>>>>> In addition to the start index approach, another potential simple way >>>>>> to implement the continuation token is to use the last item name, when >>>>>> the >>>>>> listing is guaranteed to be in lexicographic order. >>>>>> >>>>>> >>>>>> I think this is one viable implementation, but the reason that the >>>>>> token should be opaque is that it allows several different >>>>>> implementations >>>>>> without client side changes. >>>>>> >>>>>> For example, if an element is added before the continuation token, >>>>>> then all future listing calls with the token would always skip that >>>>>> element. >>>>>> >>>>>> >>>>>> IMO, I think this is fine, for some of the REST APIs it is likely >>>>>> important to put constraints on atomicity requirements, for others (e.g. >>>>>> list namespaces) I think it is OK to have looser requirements. >>>>>> >>>>>> If we want to enforce that level of atomicity, we probably want to >>>>>> introduce another time travel query parameter (e.g. asOf=1703003028000) >>>>>> to >>>>>> ensure that we are listing results at a specific point of time of the >>>>>> warehouse, so the complete result list is fixed. >>>>>> >>>>>> >>>>>> Time travel might be useful in some cases but I think it is >>>>>> orthogonal to services wishing to have guarantees around >>>>>> atomicity/consistency of results. If a server wants to ensure that >>>>>> results >>>>>> are atomic/consistent as of the start of the listing, it can embed the >>>>>> necessary timestamp in the token it returns and parse it out when >>>>>> fetching >>>>>> the next result. >>>>>> >>>>>> I think this does raise a more general point around service >>>>>> definition evolution in general. I think there likely need to be >>>>>> metadata >>>>>> endpoints that expose either: >>>>>> 1. A version of the REST API supported. >>>>>> 2. Features the API supports (e.g. which query parameters are >>>>>> honored for a specific endpoint). >>>>>> >>>>>> There are pros and cons to both approaches (apologies if I missed >>>>>> this in the spec or if it has already been discussed). >>>>>> >>>>>> Cheers, >>>>>> Micah >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 19, 2023 at 8:25 AM Jack Ye <yezhao...@gmail.com> wrote: >>>>>> >>>>>> Yes I agree that it is better to not enforce the implementation to >>>>>> favor any direction, and continuation token is probably better than >>>>>> enforcing a numeric start index. >>>>>> >>>>>> In addition to the start index approach, another potential simple way >>>>>> to implement the continuation token is to use the last item name, when >>>>>> the >>>>>> listing is guaranteed to be in lexicographic order. Compared to the start >>>>>> index approach, it does not need to worry about the change of start index >>>>>> when something in the list is added or removed. >>>>>> >>>>>> However, the issue of concurrent modification could still exist even >>>>>> with a continuation token. For example, if an element is added before the >>>>>> continuation token, then all future listing calls with the token would >>>>>> always skip that element. If we want to enforce that level of atomicity, >>>>>> we >>>>>> probably want to introduce another time travel query parameter (e.g. >>>>>> asOf=1703003028000) to ensure that we are listing results at a specific >>>>>> point of time of the warehouse, so the complete result list is fixed. >>>>>> (This >>>>>> is also the missing piece I forgot to mention in the start index approach >>>>>> to ensure it works in distributed settings) >>>>>> >>>>>> -Jack >>>>>> >>>>>> On Tue, Dec 19, 2023, 9:51 AM Micah Kornfield <emkornfi...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> I tried to cover these in more details at: >>>>>> https://docs.google.com/document/d/1bbfoLssY1szCO_Hm3_93ZcN0UAMpf7kjmpwHQngqQJ0/edit >>>>>> >>>>>> On Sun, Dec 17, 2023 at 6:07 PM Renjie Liu <liurenjie2...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> +1 for this approach. I agree that the streaming approach requires >>>>>> that http client and servers have http 2 streaming support, which is not >>>>>> compatible with old clients. >>>>>> >>>>>> I share the same concern with Micah that only start/limit may not be >>>>>> enough in a distributed environment where modification happens during >>>>>> iterations. For compatibility, we need to consider several cases: >>>>>> >>>>>> 1. Old client <-> New Server >>>>>> 2. New client <-> Old server >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Dec 16, 2023 at 6:51 AM Daniel Weeks <dwe...@apache.org> >>>>>> wrote: >>>>>> >>>>>> I agree that we want to include this feature and I raised similar >>>>>> concerns to what Micah already presented in talking with Ryan. >>>>>> >>>>>> For backward compatibility, just adding a start and limit implies a >>>>>> deterministic order, which is not a current requirement of the REST spec. >>>>>> >>>>>> Also, we need to consider whether the start/limit would need to be >>>>>> respected by the server. If existing implementations simply return all >>>>>> the >>>>>> results, will that be sufficient? There are a few edge cases that need >>>>>> to >>>>>> be considered here. >>>>>> >>>>>> For the opaque key approach, I think adding a query param to >>>>>> trigger/continue and introducing a continuation token in >>>>>> the ListNamespacesResponse might allow for more backward compatibility. >>>>>> In >>>>>> that scenario, pagination would only take place for clients who know how >>>>>> to >>>>>> paginate and the ordering would not need to be deterministic. >>>>>> >>>>>> -Dan >>>>>> >>>>>> On Fri, Dec 15, 2023, 10:33 AM Micah Kornfield <emkornfi...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Just to clarify and add a small suggestion: >>>>>> >>>>>> The behavior with no additional parameters requires the operations to >>>>>> happen as they do today for backwards compatibility (i.e either all >>>>>> responses are returned or a failure occurs). >>>>>> >>>>>> For new parameters, I'd suggest an opaque start token (instead of >>>>>> specific numeric offset) that can be returned by the service and a limit >>>>>> (as proposed above). If a start token is provided without a limit a >>>>>> default limit can be chosen by the server. Servers might return less >>>>>> than >>>>>> limit (i.e. clients are required to check for a next token to determine >>>>>> if >>>>>> iteration is complete). This enables server side state if it is desired >>>>>> but also makes deterministic listing much more feasible (deterministic >>>>>> responses are essentially impossible in the face of changing data if >>>>>> only a >>>>>> start offset is provided). >>>>>> >>>>>> In an ideal world, specifying a limit would result in streaming >>>>>> responses being returned with the last part either containing a token if >>>>>> continuation is necessary. Given conversation on the other thread of >>>>>> streaming, I'd imagine this is quite hard to model in an Open API REST >>>>>> service. >>>>>> >>>>>> Therefore it seems like using pagination with token and offset would >>>>>> be preferred. If skipping someplace in the middle of the namespaces is >>>>>> required then I would suggest modelling those as first class query >>>>>> parameters (e.g. "startAfterNamespace") >>>>>> >>>>>> Cheers, >>>>>> Micah >>>>>> >>>>>> >>>>>> On Fri, Dec 15, 2023 at 10:08 AM Ryan Blue <b...@tabular.io> wrote: >>>>>> >>>>>> +1 for this approach >>>>>> >>>>>> I think it's good to use query params because it can be >>>>>> backward-compatible with the current behavior. If you get more than the >>>>>> limit back, then the service probably doesn't support pagination. And if >>>>>> a >>>>>> client doesn't support pagination they get the same results that they >>>>>> would >>>>>> today. A streaming approach with a continuation link like in the scan API >>>>>> discussion wouldn't work because old clients don't know to make a second >>>>>> request. >>>>>> >>>>>> On Thu, Dec 14, 2023 at 10:07 AM Jack Ye <yezhao...@gmail.com> wrote: >>>>>> >>>>>> Hi everyone, >>>>>> >>>>>> During the conversation of the Scan API for REST spec, we touched on >>>>>> the topic of pagination when REST response is large or takes time to be >>>>>> produced. >>>>>> >>>>>> I just want to discuss this separately, since we also see the issue >>>>>> for ListNamespaces and ListTables/Views, when integrating with a large >>>>>> organization that has over 100k namespaces, and also a lot of tables in >>>>>> some namespaces. >>>>>> >>>>>> Pagination requires either keeping state, or the response to be >>>>>> deterministic such that the client can request a range of the full >>>>>> response. If we want to avoid keeping state, I think we need to allow >>>>>> some >>>>>> query parameters like: >>>>>> - *start*: the start index of the item in the response >>>>>> - *limit*: the number of items to be returned in the response >>>>>> >>>>>> So we can send a request like: >>>>>> >>>>>> *GET /namespaces?start=300&limit=100* >>>>>> >>>>>> *GET /namespaces/ns/tables?start=300&limit=100* >>>>>> >>>>>> And the REST spec should enforce that the response returned for the >>>>>> paginated GET should be deterministic. >>>>>> >>>>>> Any thoughts on this? >>>>>> >>>>>> Best, >>>>>> Jack Ye >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Tabular >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Tabular >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Tabular >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Tabular >>>>>> >>>>>> >>>>>> Xuanwo >>>>>> >>>>>>
