Junwang,
Thanks for sharing your feedback. The serverless engineering use case seems
very similar to catalog federations. I assume after the initial bootstrap
the engine still needs to discover new tables and refresh the existing
tables (like schema change).
I agree that clients will need some pagination like logic. e.g.
list namespaces
for each namespace:
list tables and split into batches
for each batch of tables:
call the batch load endpoint
Catalog federation / refresh will likely do something similar.
Thanks,
Steven
On Tue, Feb 3, 2026 at 4:13 AM Junwang Zhao <[email protected]> wrote:
> Hi Steven,
>
> On Tue, Feb 3, 2026 at 2:04 AM Steven Wu <[email protected]> wrote:
> >
> > Hi,
> >
> > I would like to discuss a proposal to add batch load endpoints for
> tables and views to the REST spec.
> >
> https://docs.google.com/document/d/1VW5hgaaajRWtp5KbOU3s83YyoyPi5WOSvHtoJ_yXzJs/edit?tab=t.0
> >
> > It can help with the use cases
> > * Catalog federation: improve refresh throughput and reduce load on the
> source catalog
> > * Improve planning performance by loading multiple referenced tables in
> one request.
> > * Improve MV freshness evaluation if referencing multiple source tables
> >
> > Thanks,
> > Steven
>
> I can think of one scenario where this batch load would be very
> useful: for serverless engines, which may need to reload all their
> tables from the catalog once they get back online(for mapping its
> internal schema using the batch load response).
>
> After looking at the batch-load API, it seems to require users to
> provide an array of table identifiers in the payload, which sounds
> reasonable to me. However, I have a question: could it offer an option
> to load all tables at once without specifying the table array? And if
> the number of tables is large, could it support pagination options
> such as **limit** and **offset**?
>
> I imagine the pagination could be tricky, since other engines may
> create tables dynamically. One approach I can think of is to first
> issue a listTables request and then follow up with additional
> batchLoadTables calls. I'm just wondering whether that's the only
> approach.
>
> --
> Regards
> Junwang Zhao
>
