dotnwat opened a new issue, #51669:
URL: https://github.com/apache/arrow/issues/51669
### Describe the bug, including details regarding any error messages,
version, and platform.
### Describe the bug, including details regarding any error messages,
version, and platform.
A Parquet file written with modular encryption in plaintext-footer mode and
the `AES_GCM_CTR_V1` algorithm cannot be read back, by Arrow or by any other
reader. The writer encrypts the pages with AES-CTR, as it should, but records
`AES_GCM_V1` in `FileMetaData.encryption_algorithm`. A reader takes the
algorithm from that field, tries to open the CTR pages as GCM modules, and
fails authentication.
Nothing is reported at write time, so the file looks fine until someone
tries to read an encrypted column.
#### Reproduction
```python
import pyarrow as pa, pyarrow.parquet as pq, pyarrow.parquet.encryption as pe
key = b"0123456789012345"
table = pa.table({"a": [1, 2, 3]})
for algo in ("AES_GCM_V1", "AES_GCM_CTR_V1"):
props = pe.create_encryption_properties(
key, plaintext_footer=True, encryption_algorithm=algo)
pq.write_table(table, "t.parquet", encryption_properties=props)
try:
pq.read_table(
"t.parquet",
decryption_properties=pe.create_decryption_properties(key))
print(pa.__version__, algo, "ok")
except OSError as e:
print(pa.__version__, algo, "FAILED:", e)
```
Output:
```
25.0.1 AES_GCM_V1 ok
25.0.1 AES_GCM_CTR_V1 FAILED: Failed decryption finalization
```
The same two algorithms with an encrypted footer (`plaintext_footer=False`)
both round-trip, as does `AES_GCM_V1` with a plaintext footer. Only the
combination of a plaintext footer and `AES_GCM_CTR_V1` fails.
#### Cause
`FileMetaDataBuilder::FileMetaDataBuilderImpl::Finish` in
`cpp/src/parquet/metadata.cc` builds the footer's algorithm for
plaintext-footer mode like this (current `main`):
```cpp
// if plaintext footer, set footer signing algorithm
auto file_encryption_properties = properties_->file_encryption_properties();
if (file_encryption_properties &&
!file_encryption_properties->encrypted_footer()) {
EncryptionAlgorithm signing_algorithm;
EncryptionAlgorithm algo = file_encryption_properties->algorithm();
signing_algorithm.aad.aad_file_unique = algo.aad.aad_file_unique;
signing_algorithm.aad.supply_aad_prefix = algo.aad.supply_aad_prefix;
if (!algo.aad.supply_aad_prefix) {
signing_algorithm.aad.aad_prefix = algo.aad.aad_prefix;
}
signing_algorithm.algorithm = ParquetCipher::AES_GCM_V1;
metadata_->__set_encryption_algorithm(ToThrift(signing_algorithm));
```
The hard-coded `AES_GCM_V1` treats `FileMetaData.encryption_algorithm` as
the algorithm of the footer signature, which is indeed always GCM. But the
field is the file's encryption algorithm. `parquet.thrift` describes it as:
```
/**
* Encryption algorithm. This field is set only in encrypted files
* with plaintext footer. Files with encrypted footer store algorithm id
* in FileCryptoMetaData structure.
*/
8: optional EncryptionAlgorithm encryption_algorithm
```
and the reader uses it that way:
`SerializedFile::ParseMetaDataOfEncryptedFileWithPlaintextFooter` in
`cpp/src/parquet/file_reader.cc` passes
`file_metadata_->encryption_algorithm().algorithm` to the
`InternalFileDecryptor`, which then builds the page decryptors from it.
parquet-java writes the file's actual algorithm in this field
(`ParquetFileWriter.serializeFooter`:
`parquetMetadata.setEncryption_algorithm(fileEncryptor.getEncryptionAlgorithm())`).
#### Evidence that the pages are fine and only the field is wrong
Taking a file produced by the reproduction above, changing the one byte that
selects the `EncryptionAlgorithm` union member in the footer from `AES_GCM_V1`
to `AES_GCM_CTR_V1`, and recomputing the footer signature for the changed
bytes, gives a file that pyarrow 25.0.1 reads correctly, with footer
verification enabled and the data equal to what was written. So the reader
needs no change, and the pages in affected files are valid CTR modules.
#### Suggested fix
Write the file's algorithm instead of the constant:
```cpp
signing_algorithm.algorithm = algo.algorithm;
```
The signature itself does not depend on this value: both the writer and
`FileMetaData::VerifySignature` use the GCM cipher for the footer regardless of
the file's algorithm (`metadata = true`).
The existing encryption configurations in
`cpp/src/parquet/encryption/write_configurations_test.cc` use `AES_GCM_CTR_V1`
only with an encrypted footer and a plaintext footer only with `AES_GCM_V1`,
which is why the round-trip tests do not catch this. A configuration combining
the two would cover it.
Files already written with this combination stay unreadable after the fix,
since their footers name the wrong algorithm.
#### Version and platform
- pyarrow 25.0.1 (PyPI wheel, manylinux_2_28 x86_64), Python 3.14.4, Linux
x86_64 (Fedora 44).
- The hard-coded line is present on `main` as of commit fcac3dfd8e
(2026-09-30).
### Component(s)
C++, Parquet
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
