alkis commented on code in PR #603: URL: https://github.com/apache/parquet-format/pull/603#discussion_r3731450382
########## LogicalTypes.md: ########## @@ -756,10 +756,52 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +A schema that permits self-references must include the `inline` field. + +Each self-reference inherits the compression and encryption settings of the `inline` +column chunk of the same row group. These settings are properties of the column chunk, +so all self-references of a column chunk share them regardless of the page a value is +stored in. + +Each referenced byte range is compressed independently using the `CompressionCodec` +of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. + +Each compressed byte range is an independent compression block. Compression state is +not shared with the data page or with other referenced ranges. + +For an unencrypted self-reference, `offset` and `size` identify either the independent +compressed block or the uncompressed bytes. For a compressed block, the complete range +is supplied to the codec, and its decompressed output is the resolved value. + +The decompressed size of a self-reference is not stored. Readers must rely on the +framing of the codec where it provides one, or decompress into a dynamically sized +buffer. A future revision of this specification may add an explicit decompressed size. Review Comment: Applied in aac4d72 — the forward-looking sentence is gone. The paragraph now just states that the decompressed size is not stored and how a reader copes. ########## LogicalTypes.md: ########## @@ -756,10 +756,44 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +A schema that permits self-references must include the `inline` field. + +Each self-reference inherits the compression and encryption settings of the `inline` +column chunk in the same row group. The corresponding position is the position +representing the same `FILE` value in the `inline` column's repetition and definition +level stream. + +Each referenced byte range is compressed independently using the `CompressionCodec` +of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. + +Each compressed byte range is an independent compression block. Compression state is +not shared with the data page or with other referenced ranges. + +For an unencrypted self-reference, `offset` and `size` identify either the independent +compressed block or the uncompressed bytes. For a compressed block, the complete range +is supplied to the codec, and its decompressed output is the resolved value. Review Comment: Agreed, and that is the conclusion I came to as well. The paragraph no longer speculates about a future field — per your other comment it just states that the decompressed size is not stored and that readers use the codec framing or a dynamically sized buffer. @rok, flagging since removing the sentence also removes the "may be specified later" hedge you asked for. The substance you wanted is unchanged: nothing here precludes adding an explicit decompressed size in a later revision, we just are not advertising it in the spec text. ########## LogicalTypes.md: ########## @@ -756,10 +756,52 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +A schema that permits self-references must include the `inline` field. + +Each self-reference inherits the compression and encryption settings of the `inline` +column chunk of the same row group. These settings are properties of the column chunk, +so all self-references of a column chunk share them regardless of the page a value is +stored in. + +Each referenced byte range is compressed independently using the `CompressionCodec` +of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. + +Each compressed byte range is an independent compression block. Compression state is +not shared with the data page or with other referenced ranges. + +For an unencrypted self-reference, `offset` and `size` identify either the independent +compressed block or the uncompressed bytes. For a compressed block, the complete range +is supplied to the codec, and its decompressed output is the resolved value. + +The decompressed size of a self-reference is not stored. Readers must rely on the +framing of the codec where it provides one, or decompress into a dynamically sized +buffer. A future revision of this specification may add an explicit decompressed size. + +The encryption state and key are inherited from the `inline` column chunk. If the +column chunk is encrypted, each self-reference is encrypted independently using the +same column key and file encryption algorithm. Compression is applied before +encryption. If the column chunk is not encrypted, its self-references are not +encrypted. An encrypted self-reference is limited to 2 GiB by the length field of the +encrypted module; a value too large to store this way must use an external reference +(`uri`). An encrypted stored representation is bound to a single column chunk and must +not be shared between column chunks. See +[Parquet Modular Encryption](Encryption.md) for the encryption layout and AAD +construction. + +A self-reference identifies a stored representation, not necessarily the resolved +bytes. Consumers must use a Parquet reader to resolve a self-reference; copying Review Comment: Applied in aac4d72. The sentence now reads "Copying `[offset, offset + size)` directly may return compressed or encrypted data." — describing the consequence rather than mandating who does the resolution. ########## LogicalTypes.md: ########## @@ -756,10 +756,52 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +A schema that permits self-references must include the `inline` field. + +Each self-reference inherits the compression and encryption settings of the `inline` +column chunk of the same row group. These settings are properties of the column chunk, +so all self-references of a column chunk share them regardless of the page a value is +stored in. + +Each referenced byte range is compressed independently using the `CompressionCodec` +of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. + +Each compressed byte range is an independent compression block. Compression state is +not shared with the data page or with other referenced ranges. + +For an unencrypted self-reference, `offset` and `size` identify either the independent +compressed block or the uncompressed bytes. For a compressed block, the complete range +is supplied to the codec, and its decompressed output is the resolved value. + +The decompressed size of a self-reference is not stored. Readers must rely on the +framing of the codec where it provides one, or decompress into a dynamically sized +buffer. A future revision of this specification may add an explicit decompressed size. + +The encryption state and key are inherited from the `inline` column chunk. If the +column chunk is encrypted, each self-reference is encrypted independently using the +same column key and file encryption algorithm. Compression is applied before +encryption. If the column chunk is not encrypted, its self-references are not +encrypted. An encrypted self-reference is limited to 2 GiB by the length field of the Review Comment: Added in aac4d72: "For an encrypted self-reference, `offset` and `size` identify the encrypted module, so `size` is the size after compression and encryption." One caveat on precision: `size` also covers the 4-byte length prefix and the nonce, which are not themselves products of compression or encryption, so "after compression and encryption" is a shorthand. `Encryption.md` has the exact layout — `offset` points at the 4-byte length, and `size` spans the length, nonce, ciphertext, and GCM tag when present. Happy to replace the shorthand with a pointer to that layout if you would rather the normative statement live in one place. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
