GitHub user jklaise added a comment to the discussion: How to Handle Large Data Exports (10M+ Rows) in Superset?
I was recently investigating this and it seems this is not something that's supported server-side - see more info in my comment here: https://github.com/apache/superset/discussions/33243#discussioncomment-14073481 The way I would do it as a workaround would be building a client SDK with some logic to split large tables on known primary keys and download them piece-by-piece, e.g. - Estimate size of the table using some `pg_relation_size` commands - Execute `select count(*) from table;` to get the number of rows - Use the above two to define an appropriate chunk size - Queue up queries to `/api/v1/sqllab/execute` which get chunks of the large table based on primary key (need to be smart about how to do this for it to be quick) - Poll `/api/v1/query/$query_id` to see when each query is finished - Download query results from `/api/v1/sqllab/export/$client_id` - Write to file(s) GitHub link: https://github.com/apache/superset/discussions/33530#discussioncomment-14084006 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
