alkis commented on code in PR #603:
URL: https://github.com/apache/parquet-format/pull/603#discussion_r3823291340
##########
LogicalTypes.md:
##########
@@ -735,41 +732,35 @@ only.
A value resolves to bytes based on which of `inline`, `uri`, `offset`, and
`size` are
set:
-| `inline` | `uri` | `offset` | `size` | Resolves to
|
-|----------|-------|----------|--------|-------------------------------------------------------|
-| set | - | - | - | the inline bytes
|
-| - | set | - | - | whole external file at `uri`
|
-| - | set | set | - | invalid
|
-| - | set | - | set | external `uri`, `[0, size)`
|
-| - | set | set | set | external `uri`, `[offset, offset +
size)` |
-| - | - | set | - | invalid
|
-| - | - | - | set | invalid
|
-| - | - | set | set | this file, `[offset, offset + size)`
(self-reference) |
-| - | - | - | - | nothing - invalid
|
+| `inline` | `uri` | `offset` | `size` | Resolves to
|
+|----------|-------|----------|--------|-------------------------------------------|
+| set | - | - | - | the inline bytes
|
Review Comment:
Agreed on both counts. An inline prefix of a larger object at `uri` is a
genuinely useful pattern for filtering, but it needs its own field to say "this
is a prefix, not the value" — overloading `inline` would make every reader
guess which it is. An application can do it today with a separate column, and
if there is demand later it can be a new optional field with explicit semantics
rather than a reinterpretation of this one.
##########
LogicalTypes.md:
##########
@@ -735,41 +732,35 @@ only.
A value resolves to bytes based on which of `inline`, `uri`, `offset`, and
`size` are
set:
-| `inline` | `uri` | `offset` | `size` | Resolves to
|
-|----------|-------|----------|--------|-------------------------------------------------------|
-| set | - | - | - | the inline bytes
|
-| - | set | - | - | whole external file at `uri`
|
-| - | set | set | - | invalid
|
-| - | set | - | set | external `uri`, `[0, size)`
|
-| - | set | set | set | external `uri`, `[offset, offset +
size)` |
-| - | - | set | - | invalid
|
-| - | - | - | set | invalid
|
-| - | - | set | set | this file, `[offset, offset + size)`
(self-reference) |
-| - | - | - | - | nothing - invalid
|
+| `inline` | `uri` | `offset` | `size` | Resolves to
|
+|----------|-------|----------|--------|-------------------------------------------|
+| set | - | - | - | the inline bytes
|
+| - | set | - | - | whole external file at `uri`
|
+| - | set | set | - | invalid
|
+| - | set | - | set | external `uri`, `[0, size)`
|
+| - | set | set | set | external `uri`, `[offset, offset +
size)` |
+| - | - | set | - | invalid
|
+| - | - | - | set | invalid
|
+| - | - | set | set | invalid
|
+| - | - | - | - | nothing - invalid
|
`size` must be set whenever `offset` is set, so any offset-based read always
carries an
-explicit `size`. A self-reference (`uri` not set) must set `offset`, and
therefore also
-`size`. `size` may be omitted only for a whole-file external reference, where
the range
-runs to the end of the referenced file.
+explicit `size`. `size` may be omitted only for a whole-file external
reference, where
+the range runs to the end of the referenced file. A byte range within the
current file
+cannot be referenced: `offset` and `size` apply only to data referenced by
`uri`.
-A self-reference points within the same Parquet file using `offset` and `size`
(both
-required). A self-reference is when `uri` is not set. A file containing
self-references
-can be renamed or relocated as a single unit.
-
-Parquet files containing self-references must not use Parquet modular
encryption.
-Self-referenced byte ranges are not Parquet encryption modules and therefore
cannot
-be encrypted or authenticated independently. Encryption of external files
referenced
-by `uri` is outside the scope of the Parquet format.
+Encryption of external files referenced by `uri` is outside the scope of the
Parquet
+format.
Review Comment:
Yes, out of scope here. Worth noting the reason it is a bigger change than
it looks: a key for the bytes behind a `uri` means Parquet describing the
encryption of data it does not own and cannot re-encrypt on rewrite, so key
rotation and compaction both need answers. Happy to look at it as a separate
proposal if you want to write it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
