+1 for this approach I think it's good to use query params because it can be backward-compatible with the current behavior. If you get more than the limit back, then the service probably doesn't support pagination. And if a client doesn't support pagination they get the same results that they would today. A streaming approach with a continuation link like in the scan API discussion wouldn't work because old clients don't know to make a second request.
On Thu, Dec 14, 2023 at 10:07 AM Jack Ye <yezhao...@gmail.com> wrote: > Hi everyone, > > During the conversation of the Scan API for REST spec, we touched on the > topic of pagination when REST response is large or takes time to be > produced. > > I just want to discuss this separately, since we also see the issue for > ListNamespaces and ListTables/Views, when integrating with a large > organization that has over 100k namespaces, and also a lot of tables in > some namespaces. > > Pagination requires either keeping state, or the response to be > deterministic such that the client can request a range of the full > response. If we want to avoid keeping state, I think we need to allow some > query parameters like: > - *start*: the start index of the item in the response > - *limit*: the number of items to be returned in the response > > So we can send a request like: > > *GET /namespaces?start=300&limit=100* > > *GET /namespaces/ns/tables?start=300&limit=100* > > And the REST spec should enforce that the response returned for the > paginated GET should be deterministic. > > Any thoughts on this? > > Best, > Jack Ye > > -- Ryan Blue Tabular
