wiedld commented on issue #11042: URL: https://github.com/apache/datafusion/issues/11042#issuecomment-2234271367
> I also found https://github.com/apache/arrow-rs/issues/5828 which might be related and/or relevant. @hveiga is correct that this is one suspected place with extra memory usage (specifically in the dict_encoder) when processing many rows per page. But that is not the only place, and as such that's why we focused on changing the config setting `data_page_row_limit=20k`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
