andygrove opened a new issue, #6960:
URL: https://github.com/apache/datafusion-comet/issues/6960
### Describe the bug
The eligibility gate admits a gzip table with a
`write.parquet.compression-level` from 0 to 9, and any brotli table, for the
native writer. On Comet's default build (Spark 4.1, Hadoop 3.4.2, Iceberg
1.11.0) iceberg-java fails both writes, so with native Iceberg writes enabled
(the default since #6664), a write that fails without Comet succeeds with it:
- **gzip with a numeric level.** iceberg-java passes the level to Hadoop as
`zlib.compress.level`, and Hadoop reads that key as a
`ZlibCompressor.CompressionLevel` enum name (`Configuration.getEnum` in
`ZlibFactory.getCompressionLevel`). A number fails the task with
`IllegalArgumentException: No enum constant
org.apache.hadoop.io.compress.zlib.ZlibCompressor.CompressionLevel.5`. An enum
name such as `BEST_SPEED` works, and Comet already falls back on that because
it is not a Java int. On Hadoop 3.3.4 (Spark 3.4 and 3.5), `GzipCodec` reads
the level the same way when native zlib is loaded, and ignores it when it is
not.
- **brotli.** parquet-mr loads `org.apache.hadoop.io.compress.BrotliCodec`,
which neither Spark nor Iceberg ships, so the task fails with
`BadConfigurationException: Class org.apache.hadoop.io.compress.BrotliCodec was
not found` unless the deployment adds a brotli codec jar.
A numeric `zlib.compress.level` in the writer's Hadoop configuration (for
example `spark.hadoop.zlib.compress.level` with `HadoopFileIO`) reaches the
same `ZlibFactory.getCompressionLevel` call for a gzip table. The gate does not
look at that key, because the user guide treats codec settings in the Hadoop
configuration as only shifting compressed bytes.
### Steps to reproduce
Spark 4.1.1 with `iceberg-spark-runtime-4.1_2.13:1.11.0` and a Hadoop
catalog:
```sql
CREATE TABLE local.db.g5 (id bigint, s string) USING iceberg
TBLPROPERTIES ('write.parquet.compression-codec'='gzip',
'write.parquet.compression-level'='5');
INSERT INTO local.db.g5 VALUES (1, 'a'), (2, 'b');
CREATE TABLE local.db.b (id bigint, s string) USING iceberg
TBLPROPERTIES ('write.parquet.compression-codec'='brotli');
INSERT INTO local.db.b VALUES (1, 'a'), (2, 'b');
```
Without Comet, each `INSERT` fails with the error above. With Comet, the
native writer engaged (`spark.comet.exec.localTableScan.enabled=true`, so the
`VALUES` input runs in Comet): both plans show `CometIcebergWrite`, both
inserts succeed, and the data files carry Comet's `created_by`.
### Expected behavior
The native writer should reproduce iceberg-java's outcome, so both writes
should fall back and fail as they do without Comet:
- decline gzip with a numeric `write.parquet.compression-level`, and with a
numeric `zlib.compress.level` in the writer's Hadoop configuration
- decline brotli unless `org.apache.hadoop.io.compress.BrotliCodec` can be
loaded
and update the user guide: the compression-level row of the eligibility
table, and the accepted-divergences paragraph about codec settings in the
Hadoop configuration.
### Additional context
Found while deciding #5643. The lz4, snappy and zstd codecs, and zstd levels
0 and -3, all write successfully on iceberg-java in the same setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
