GitHub user jklaise added a comment to the discussion: Superset pod crashes and
high download latency with large Presto query results (>5M rows) — is streaming
download supported?
I've done some digging and as far as I understand Superset currently does not
support server-side streaming of results.
The endpoint `/api/v1/sqllab/export/{client_id}` downloads the CSV, but if you
look in the
[code](https://github.com/apache/superset/blob/732506b3fa974bd0524d6879cd241641f660a108/superset/commands/sql_lab/export.py#L94-L109),
the results are loaded into memory and a pandas dataframe is created before
creating the CSV.
I was hoping to use the `/api/v1/query/sqllab/results/` endpoint to stream the
json result blob over REST and do CSV parsing client side, but this is a
similar story as the result blob is fully loaded into memory on server side
before being sent over, see
[code](https://github.com/apache/superset/blob/732506b3fa974bd0524d6879cd241641f660a108/superset/commands/sql_lab/results.py#L106-L109).
GitHub link:
https://github.com/apache/superset/discussions/33243#discussioncomment-14073481
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
