kosiew commented on code in PR #24924:
URL: https://github.com/apache/datafusion/pull/24924#discussion_r4043365164
##########
datafusion/datasource/src/write/orchestration.rs:
##########
@@ -111,9 +112,11 @@ pub(crate) async fn serialize_rb_stream_to_object_store(
});
let mut row_count = 0;
+ let mut serialized_bytes = 0;
while let Some(task) = rx.recv().await {
match task.join().await {
Ok(Ok((cnt, bytes))) => {
+ serialized_bytes += bytes.len();
Review Comment:
`serialized_bytes` is counting the bytes returned by the CSV/JSON serializer
before they go through the compression wrapper installed by
`ObjectWriterBuilder`.
That means `COPY ... TO` with gzip, bzip2, xz, or zstd can still rotate
based on the uncompressed size. For highly compressible data, this reproduces
the original issue outside Parquet.
Could we account for bytes at the compressed writer/object-store boundary
and add a regression test for a compressed stateless format? Alternatively, if
that is not intended to be supported, I think the option should be explicitly
restricted to uncompressed stateless writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
