L. C. Hsieh created SPARK-60316:
-----------------------------------
Summary: Validate the dictionary page against its size when
reading Parquet
Key: SPARK-60316
URL: https://issues.apache.org/jira/browse/SPARK-60316
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.8, 4.2.0, 4.1.3, 4.0.4, 4.3.0, 4.4.0
Reporter: L. C. Hsieh
Spark passes a Parquet dictionary page to parquet-java's
{{Encoding.initDictionary}} without checking the declared dictionary size or
the entry lengths against the bytes of the page. A corrupted dictionary page
can then:
# Kill the executor. A huge positive {{num_values}} in the dictionary page
header makes {{PlainValuesDictionary}} allocate {{new int[size]}}, {{new
Binary[size]}} and so on. For example, an INT32 dictionary page with
{{num_values = 0x7FFFFFF0}} throws {{OutOfMemoryError: Java heap space}}. The
executor exits, every retry kills another executor, and
{{spark.sql.files.ignoreCorruptFiles}} cannot skip the file.
# Return wrong data without an error. {{PlainBinaryDictionary}} does not check
an entry's length against the page. In an UNCOMPRESSED column chunk the
dictionary buffer is a slice of the buffer of the chunk, so an entry that
claims more bytes than are left reads bytes of the next page. For example, a
string dictionary {{02 00 00 00 'a' 'b' 05 00 00 00 'c' 'd'}} returns {{cd}}
followed by 3 bytes of the next page header, with both the vectorized and the
non-vectorized reader. With a compressed chunk the same entry fails with an
{{ArrayIndexOutOfBoundsException}} that names no column, and a truncated
FIXED_LEN_BYTE_ARRAY dictionary fails with an
{{ArrayIndexOutOfBoundsException}} in {{VectorizedColumnReader.readBatch}}.
Before {{initDictionary}}, the vectorized reader could reject a negative size
or a size that cannot fit in the page ({{size * minEntryWidth > page size}}),
check that every BINARY entry ends within the page and that
FIXED_LEN_BYTE_ARRAY and INT96 entries fit ({{size * length}}), and throw a
{{ParquetDecodingException}} that names the column. This runs once per column
chunk. The non-vectorized reader builds the dictionary in parquet-java, so it
needs the same check there.
Found in the review of SPARK-59854 (https://github.com/apache/spark/pull/59542).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
