nsivabalan commented on PR #5129: URL: https://github.com/apache/hudi/pull/5129#issuecomment-1123690948
I went through the code again wrt this patch and https://github.com/apache/hudi/pull/5497. Probably we should bring back the WrapperFileSystem again so that we don't hit the parquetWriter to fetch the size. if we ensure we flush at regular intervals, wrapperFileSystem.getBytesWritten(Path file) should give us the right size of data that got written. this will also ensure we don't hit the disk or incur the cost due to column meta refresh within parquetWriter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
