alkis commented on code in PR #603:
URL: https://github.com/apache/parquet-format/pull/603#discussion_r3719603274
##########
Encryption.md:
##########
@@ -270,43 +272,54 @@ The following module types are defined:
* OffsetIndex (7)
* BloomFilter Header (8)
* BloomFilter Bitset (9)
+ * Self-Reference (10)
-| | Internal File ID | Module type | Row group ordinal |
Column ordinal | Page ordinal|
-|----------------------|------------------|-------------|-------------------|----------------|-------------|
-| Footer | yes | yes (0) | no |
no | no |
-| ColumnMetaData | yes | yes (1) | yes |
yes | no |
-| Data Page | yes | yes (2) | yes |
yes | yes |
-| Dictionary Page | yes | yes (3) | yes |
yes | no |
-| Data Page Header | yes | yes (4) | yes |
yes | yes |
-| Dictionary Page Header| yes | yes (5) | yes |
yes | no |
-| ColumnIndex | yes | yes (6) | yes |
yes | no |
-| OffsetIndex | yes | yes (7) | yes |
yes | no |
-| BloomFilter Header | yes | yes (8) | yes |
yes | no |
-| BloomFilter Bitset | yes | yes (9) | yes |
yes | no |
+| | Internal File ID | Module type | Row group ordinal |
Column ordinal | Page ordinal | Self-reference ordinal |
+|-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------|
+| Footer | yes | yes (0) | no |
no | no | no |
+| ColumnMetaData | yes | yes (1) | yes |
yes | no | no |
+| Data Page | yes | yes (2) | yes |
yes | yes | no |
+| Dictionary Page | yes | yes (3) | yes |
yes | no | no |
+| Data Page Header | yes | yes (4) | yes |
yes | yes | no |
+| Dictionary Page Header| yes | yes (5) | yes |
yes | no | no |
+| ColumnIndex | yes | yes (6) | yes |
yes | no | no |
+| OffsetIndex | yes | yes (7) | yes |
yes | no | no |
+| BloomFilter Header | yes | yes (8) | yes |
yes | no | no |
+| BloomFilter Bitset | yes | yes (9) | yes |
yes | no | no |
+| Self-Reference | yes | yes (10) | yes |
yes | no | yes |
+
+For a self-reference, the column ordinal is that of the `inline` column. The
+self-reference ordinal is the zero-based position representing the same `FILE`
value
+in that column chunk's repetition and definition level stream, including null
+positions. It is derived by the reader and is not stored separately.
## 5. File Format
### 5.1 Encrypted module serialization
-All modules, except column pages, are encrypted with the GCM cipher. In the
AES_GCM_V1 algorithm,
-the column pages are also encrypted with AES GCM. For each module, the GCM
encryption
+All modules, except column pages and self-references, are encrypted with the
GCM cipher. In the
+AES_GCM_V1 algorithm, column pages and self-references are also encrypted with
AES GCM. For each module, the GCM encryption
buffer is comprised of a nonce, ciphertext and tag, described in the
Algorithms section. The length of
the encryption buffer (a 4-byte little endian) is written to the output
stream, followed by the buffer itself.
|length (4 bytes) | nonce (12 bytes) | ciphertext (length-28 bytes) | tag (16
bytes) |
|-----------------|------------------|------------------------------|----------------|
-In the AES_GCM_CTR_V1 algorithm, the column pages are encrypted with AES CTR.
-For each page, the CTR encryption buffer is comprised of a nonce and
ciphertext,
+In the AES_GCM_CTR_V1 algorithm, column pages and self-references are
encrypted with AES CTR.
+For each page or self-reference, the CTR encryption buffer is comprised of a
nonce and ciphertext,
described in the Algorithms section. The length of the encryption buffer
(a 4-byte little endian) is written to the output stream, followed by the
buffer itself.
|length (4 bytes) | nonce (12 bytes) | ciphertext (length-12 bytes) |
|-----------------|------------------|------------------------------|
+For a self-reference, `offset` points to the beginning of the 4-byte length
and `size`
+is the size of the complete encrypted module, including the length, nonce,
ciphertext,
+and GCM tag when present.
Review Comment:
The opposite — with this PR self-references *are* modular encryption modules
(module type 10, keyed on the `inline` column key), so modular encryption works
for them. This PR removes the prohibition that the merged spec had.
Two limits are worth noting and are now stated:
- The encryption buffer must fit the 4-byte length field, so an encrypted
self-reference is capped at 2 GiB as for any other module. Larger values use an
external reference.
- The AAD includes the row group and column ordinals, so an encrypted stored
representation is bound to one column chunk. A writer must not share one
encrypted byte range between column chunks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
