I agree that we want to include this feature and I raised similar concerns to what Micah already presented in talking with Ryan.
For backward compatibility, just adding a start and limit implies a deterministic order, which is not a current requirement of the REST spec. Also, we need to consider whether the start/limit would need to be respected by the server. If existing implementations simply return all the results, will that be sufficient? There are a few edge cases that need to be considered here. For the opaque key approach, I think adding a query param to trigger/continue and introducing a continuation token in the ListNamespacesResponse might allow for more backward compatibility. In that scenario, pagination would only take place for clients who know how to paginate and the ordering would not need to be deterministic. -Dan On Fri, Dec 15, 2023, 10:33 AM Micah Kornfield <emkornfi...@gmail.com> wrote: > Just to clarify and add a small suggestion: > > The behavior with no additional parameters requires the operations to > happen as they do today for backwards compatibility (i.e either all > responses are returned or a failure occurs). > > For new parameters, I'd suggest an opaque start token (instead of specific > numeric offset) that can be returned by the service and a limit (as > proposed above). If a start token is provided without a limit a > default limit can be chosen by the server. Servers might return less than > limit (i.e. clients are required to check for a next token to determine if > iteration is complete). This enables server side state if it is desired > but also makes deterministic listing much more feasible (deterministic > responses are essentially impossible in the face of changing data if only a > start offset is provided). > > In an ideal world, specifying a limit would result in streaming responses > being returned with the last part either containing a token if continuation > is necessary. Given conversation on the other thread of streaming, I'd > imagine this is quite hard to model in an Open API REST service. > > Therefore it seems like using pagination with token and offset would be > preferred. If skipping someplace in the middle of the namespaces is > required then I would suggest modelling those as first class query > parameters (e.g. "startAfterNamespace") > > Cheers, > Micah > > > On Fri, Dec 15, 2023 at 10:08 AM Ryan Blue <b...@tabular.io> wrote: > >> +1 for this approach >> >> I think it's good to use query params because it can be >> backward-compatible with the current behavior. If you get more than the >> limit back, then the service probably doesn't support pagination. And if a >> client doesn't support pagination they get the same results that they would >> today. A streaming approach with a continuation link like in the scan API >> discussion wouldn't work because old clients don't know to make a second >> request. >> >> On Thu, Dec 14, 2023 at 10:07 AM Jack Ye <yezhao...@gmail.com> wrote: >> >>> Hi everyone, >>> >>> During the conversation of the Scan API for REST spec, we touched on the >>> topic of pagination when REST response is large or takes time to be >>> produced. >>> >>> I just want to discuss this separately, since we also see the issue for >>> ListNamespaces and ListTables/Views, when integrating with a large >>> organization that has over 100k namespaces, and also a lot of tables in >>> some namespaces. >>> >>> Pagination requires either keeping state, or the response to be >>> deterministic such that the client can request a range of the full >>> response. If we want to avoid keeping state, I think we need to allow some >>> query parameters like: >>> - *start*: the start index of the item in the response >>> - *limit*: the number of items to be returned in the response >>> >>> So we can send a request like: >>> >>> *GET /namespaces?start=300&limit=100* >>> >>> *GET /namespaces/ns/tables?start=300&limit=100* >>> >>> And the REST spec should enforce that the response returned for the >>> paginated GET should be deterministic. >>> >>> Any thoughts on this? >>> >>> Best, >>> Jack Ye >>> >>> >> >> -- >> Ryan Blue >> Tabular >> >
