alkis commented on code in PR #603:
URL: https://github.com/apache/parquet-format/pull/603#discussion_r3719602189
##########
Encryption.md:
##########
@@ -270,43 +272,54 @@ The following module types are defined:
* OffsetIndex (7)
* BloomFilter Header (8)
* BloomFilter Bitset (9)
+ * Self-Reference (10)
-| | Internal File ID | Module type | Row group ordinal |
Column ordinal | Page ordinal|
-|----------------------|------------------|-------------|-------------------|----------------|-------------|
-| Footer | yes | yes (0) | no |
no | no |
-| ColumnMetaData | yes | yes (1) | yes |
yes | no |
-| Data Page | yes | yes (2) | yes |
yes | yes |
-| Dictionary Page | yes | yes (3) | yes |
yes | no |
-| Data Page Header | yes | yes (4) | yes |
yes | yes |
-| Dictionary Page Header| yes | yes (5) | yes |
yes | no |
-| ColumnIndex | yes | yes (6) | yes |
yes | no |
-| OffsetIndex | yes | yes (7) | yes |
yes | no |
-| BloomFilter Header | yes | yes (8) | yes |
yes | no |
-| BloomFilter Bitset | yes | yes (9) | yes |
yes | no |
+| | Internal File ID | Module type | Row group ordinal |
Column ordinal | Page ordinal | Self-reference ordinal |
+|-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------|
+| Footer | yes | yes (0) | no |
no | no | no |
+| ColumnMetaData | yes | yes (1) | yes |
yes | no | no |
+| Data Page | yes | yes (2) | yes |
yes | yes | no |
+| Dictionary Page | yes | yes (3) | yes |
yes | no | no |
+| Data Page Header | yes | yes (4) | yes |
yes | yes | no |
+| Dictionary Page Header| yes | yes (5) | yes |
yes | no | no |
+| ColumnIndex | yes | yes (6) | yes |
yes | no | no |
+| OffsetIndex | yes | yes (7) | yes |
yes | no | no |
+| BloomFilter Header | yes | yes (8) | yes |
yes | no | no |
+| BloomFilter Bitset | yes | yes (9) | yes |
yes | no | no |
+| Self-Reference | yes | yes (10) | yes |
yes | no | yes |
Review Comment:
You are right that a derived ordinal is wrong for exactly this reason. I
switched the AAD to key on the self-reference `offset` instead. The offset is a
field of the `FILE` value being read, so the reader has it in hand and never
counts values in pages it skips.
`Page ordinal` would also work for a reader — you know the ordinal of the
page you are decoding — but it ties the out-of-band range to page layout, which
the rest of this PR deliberately avoids: compression and encryption are
properties of the column chunk, not of the page a value happens to land in.
Keying on `offset` keeps the stored representation independent of page
boundaries, and the offset already uniquely identifies it within the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
