adriangb commented on issue #8078: URL: https://github.com/apache/datafusion/issues/8078#issuecomment-3392275880
My thought was to take a random sample *after* sorting (thus duplicating random rows). Then make that its own row group. This means you can also use the table sample as a proxy for the sortedness of the data. Then you don't have to read a fraction of a row group. You'd read the metadata (which is then cached), make an access plan that reads only the last row group and you've got your table sample. Then for the actual scan you'd initialize the access plan with `Skip(row_group_count)` so you always skip the sample row group. This could happen at the same time statistics are collected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
