alkis commented on code in PR #603: URL: https://github.com/apache/parquet-format/pull/603#discussion_r3731449479
########## LogicalTypes.md: ########## @@ -688,11 +688,11 @@ external reference (`uri` set). `offset` must not be < 0. ##### size -The byte length of the referenced data. Must be zero or a positive integer if set; a -value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set. -It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), -in which case the range runs to the end of the referenced file. Because a self-reference -always sets `offset`, it always sets `size` as well. +The byte length of the stored representation. Must be zero or a positive integer if set. +A value of 0 indicates an empty stored representation. `size` must be set whenever +`offset` is set. It may be omitted only for a whole-file external reference (`uri` set, +`offset` not set), in which case the range runs to the end of the referenced file. +Because a self-reference always sets `offset`, it always sets `size` as well. Review Comment: Applied in aac4d72. It also lines up with the validation bullet further down, which already stated this with "must". ########## LogicalTypes.md: ########## @@ -756,10 +756,52 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +A schema that permits self-references must include the `inline` field. + +Each self-reference inherits the compression and encryption settings of the `inline` +column chunk of the same row group. These settings are properties of the column chunk, +so all self-references of a column chunk share them regardless of the page a value is +stored in. Review Comment: Good catch — applied in aac4d72. You are right that the old wording implied the payload lives in a page, which is exactly what it does not do. The point I wanted to make is that the settings come from the column chunk, so your phrasing says it without the misleading implication. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
