cshuo commented on issue #20014: URL: https://github.com/apache/hudi/issues/20014#issuecomment-5756915736
The small-file problem is real, and local spilling is a viable optimization to explore. However, it introduces a tradeoff: fewer remote files and potentially less compaction work come at the cost of additional local I/O on the writer. It also defers remote writes, potentially concentrating more work at checkpoint time and increasing checkpoint latency or even causing timeouts. Measurements from the internal implementation would help establish where this tradeoff pays off, particularly the reduction in small files alongside end-to-end throughput, checkpoint tail latency, and timeout rates. If the results demonstrate meaningful benefits for specific workloads, this could be offered as an optional optimization feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
