akashjainn commented on issue #18195: URL: https://github.com/apache/datafusion/issues/18195#issuecomment-5736242136
I'd like to take the remaining item, "Fix `elapsed_compute` baseline metrics not counting issue". It is only partly fixed on main. #20767 added an `elapsed_compute` timer in `PushDecoderStreamState::transition`, but it wraps `copy_arrow_reader_metrics` and `project_batch` only. The decoding happens before the timer starts (in `try_decode` at the time of that PR, in `reader.next()` today), so the metric went from a few nanoseconds to a fraction of a millisecond and is still far too low. For a single-partition `SELECT *` over a 1M row file, the scan takes about 155 ms and reports `elapsed_compute` of about 0.1 ms. Plan: run one timer for the whole transition (decoding, row group pruning and decoder rebuilds, building readers, projection) and pause it only across `get_byte_ranges(...).await`, which is the I/O. That is the same idea as the CSV fix in #18901, which times `reader.next()`. With that change the same scan reports about 151 ms. The number of timer starts per batch stays at one, so the overhead that was discussed on #20767 does not change. I'll add a test that checks the scan's `elapsed_compute` against its wall time, since the existing tests only assert that it is greater than zero. Will have a PR up soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
