emkornfield commented on code in PR #461:
URL: https://github.com/apache/parquet-format/pull/461#discussion_r1894292367
##########
VariantShredding.md:
##########
@@ -25,290 +25,320 @@
The Variant type is designed to store and process semi-structured data
efficiently, even with heterogeneous values.
Query engines encode each Variant value in a self-describing format, and store
it as a group containing `value` and `metadata` binary fields in Parquet.
Since data is often partially homogenous, it can be beneficial to extract
certain fields into separate Parquet columns to further improve performance.
-We refer to this process as **shredding**.
-Each Parquet file remains fully self-describing, with no additional metadata
required to read or fully reconstruct the Variant data from the file.
-Combining shredding with a binary residual provides the flexibility to
represent complex, evolving data with an unbounded number of unique fields
while limiting the size of file schemas, and retaining the performance benefits
of a columnar format.
+This process is **shredding**.
-This document focuses on the shredding semantics, Parquet representation,
implications for readers and writers, as well as the Variant reconstruction.
-For now, it does not discuss which fields to shred, user-facing API changes,
or any engine-specific considerations like how to use shredded columns.
-The approach builds upon the [Variant Binary Encoding](VariantEncoding.md),
and leverages the existing Parquet specification.
+Shredding enables the use of Parquet's columnar representation for more
compact data encoding, column statistics for data skipping, and partial
projections.
-At a high level, we replace the `value` field of the Variant Parquet group
with one or more fields called `object`, `array`, `typed_value`, and
`variant_value`.
-These represent a fixed schema suitable for constructing the full Variant
value for each row.
+For example, the query `SELECT variant_get(event, '$.event_ts', 'timestamp')
FROM tbl` only needs to load field `event_ts`, and if that column is shredded,
it can be read by columnar projection without reading or deserializing the rest
of the `event` Variant.
+Similarly, for the query `SELECT * FROM tbl WHERE variant_get(event,
'$.event_type', 'string') = 'signup'`, the `event_type` shredded column
metadata can be used for skipping and to lazily load the rest of the Variant.
-Shredding allows a query engine to reap the full benefits of Parquet's
columnar representation, such as more compact data encoding, min/max statistics
for data skipping, and I/O and CPU savings from pruning unnecessary fields not
accessed by a query (including the non-shredded Variant binary data).
-Without shredding, any query that accesses a Variant column must fetch all
bytes of the full binary buffer.
-With shredding, we can get nearly equivalent performance as in a relational
(scalar) data model.
+## Variant Metadata
-For example, `select variant_get(variant_col, ‘$.field1.inner_field2’,
‘string’) from tbl` only needs to access `inner_field2`, and the file scan
could avoid fetching the rest of the Variant value if this field was shredded
into a separate column in the Parquet schema.
-Similarly, for the query `select * from tbl where variant_get(variant_col,
‘$.id’, ‘integer’) = 123`, the scan could first decode the shredded `id`
column, and only fetch/decode the full Variant value for rows that pass the
filter.
+Variant metadata is stored in the top-level Variant group in a binary
`metadata` column regardless of whether the Variant value is shredded.
-# Parquet Example
+All `value` columns within the Variant must use the same `metadata`.
+All field names of a Variant, whether shredded or not, must be present in the
metadata.
-Consider the following Parquet schema together with how Variant values might
be mapped to it.
-Notice that we represent each shredded field in `object` as a group of two
fields, `typed_value` and `variant_value`.
-We extract all homogenous data items of a certain path into `typed_value`, and
set aside incompatible data items in `variant_value`.
-Intuitively, incompatibilities within the same path may occur because we store
the shredding schema per Parquet file, and each file can contain several row
groups.
-Selecting a type for each field that is acceptable for all rows would be
impractical because it would require buffering the contents of an entire file
before writing.
+## Value Shredding
-Typically, the expectation is that `variant_value` exists at every level as an
option, along with one of `object`, `array` or `typed_value`.
-If the actual Variant value contains a type that does not match the provided
schema, it is stored in `variant_value`.
-An `variant_value` may also be populated if an object can be partially
represented: any fields that are present in the schema must be written to those
fields, and any missing fields are written to `variant_value`.
-
-The `metadata` column is unchanged from its unshredded representation, and may
be referenced in `variant_value` fields in the shredded data.
+Variant values are stored in Parquet fields named `value`.
+Each `value` field may have an associated shredded field named `typed_value`
that stores the value when it matches a specific type.
+When `typed_value` is present, readers **must** reconstruct shredded values
according to this specification.
+For example, a Variant field, `measurement` may be shredded as long values by
adding `typed_value` with type `int64`:
```
-optional group variant_col {
- required binary metadata;
- optional binary variant_value;
- optional group object {
- optional group a {
- optional binary variant_value;
- optional int64 typed_value;
- }
- optional group b {
- optional binary variant_value;
- optional group object {
- optional group c {
- optional binary variant_value;
- optional binary typed_value (STRING);
- }
- }
- }
- }
+required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
}
```
-| Variant Value | Top-level variant_value | b.variant_value | a.typed_value |
a.variant_value | b.object.c.typed_value | b.object.c.variant_value | Notes |
-|---------------|-------------------------|-----------------|---------------|-----------------|------------------------|--------------------------|-------|
-| {a: 123, b: {c: “hello”}} | null | null | 123 | null | hello | null | All
values shredded |
-| {a: 1.23, b: {c: “123”}} | null | null | null | 1.23 | 123 | null | a is not
an integer |
-| {a: 123, b: {c: null}} | null | null | null | 123 | null | null | b.object.c
set to non-null to indicate VariantNull |
-| {a: 123, b: {} | null | null | null | 123 | null | null | b.object.c set to
null, to indicate that c is missing |
-| {a: 123, d: 456} | {d: 456} | null | 123 | null | null | null | Extra field
d is stored as variant_value |
-| [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | [{a: 1, b: {c: 2}}, {a: 3, b: {c:
4}}] | null | null | null | null | null | Not an object |
-
-# Parquet Layout
+The Parquet columns used to store variant metadata and values must be accessed
by name, not by position.
-The `array` and `object` fields represent Variant array and object types,
respectively.
-Arrays must use the three-level list structure described in
[LogicalTypes.md](LogicalTypes.md).
+The series of measurements `34, null, "n/a", 100` would be stored as:
-An `object` field must be a group.
-Each field name of this inner group corresponds to the Variant value's object
field name.
-Each inner field's type is a recursively shredded variant value: that is, the
fields of each object field must be one or more of `object`, `array`,
`typed_value` or `variant_value`.
+| Value | `metadata` | `value` | `typed_value` |
+|---------|------------------|-----------------------|---------------|
+| 34 | `01 00` v1/empty | null | `34` |
+| null | `01 00` v1/empty | `00` (null) | null |
+| "n/a" | `01 00` v1/empty | `13 6E 2F 61` (`n/a`) | null |
+| 100 | `01 00` v1/empty | null | `100` |
-Similarly the elements of an `array` must be a group containing one or more of
`object`, `array`, `typed_value` or `variant_value`.
+Both `value` and `typed_value` are optional fields used together to encode a
single value.
+Values in the two fields must be interpreted according to the following table:
-Each leaf in the schema can store an arbitrary Variant value.
-It contains an `variant_value` binary field and a `typed_value` field.
-If non-null, `variant_value` represents the value stored as a Variant binary.
-The `typed_value` field may be any type that has a corresponding Variant type.
-For each value in the data, at most one of the `typed_value` and
`variant_value` may be non-null.
-A writer may omit either field, which is equivalent to all rows being null.
+| `value` | `typed_value` | Meaning
|
+|----------|---------------|-------------------------------------------------------------|
+| null | null | The value is missing; only valid for shredded
object fields |
+| non-null | null | The value is present and may be any type,
including null |
+| null | non-null | The value is present and is the shredded type
|
+| non-null | non-null | The value is present and is a partially shredded
object |
-Dictionary IDs in a `variant_value` field refer to entries in the top-level
`metadata` field.
+An object is _partially shredded_ when the `value` is an object and the
`typed_value` is a shredded object.
-For an `object`, a null field means that the field does not exist in the
reconstructed Variant object.
-All elements of an `array` must be non-null, since array elements cannote be
missing.
+If both fields are non-null and either is not an object, the value is invalid.
Readers must either fail or return the `typed_value`.
Review Comment:
To maybe go a little bit deeper here, I think it is worth enumerating a
matrix of potential operations and data inconsistencies (see below). The main
purpose of these examples is to show that:
1. Requiring using `typed_value` or `failure` pessimizes some cases or
requires that different operations will return inconsistent result (making the
value of trying to force consistency in some cases have less utility).
2. Preferring `typed_value` in the presence of some inconsistencies sseems
like an arbitrary choice simply for the sake of not exposing bugs in the
underlying data.
I think the way of rectifying this is either to add more cases that are
considered "valid" (e.g. shredded object fields might overlap with fields in
value as long as they are consistent) and/or leave the behavior undefined. My
preference would simply be to do the latter.
I think there are three main operations that will be performed with Variant:
1. Project as primitive non-strict - Return a field as a specific primitive
value, if a field mismatches the type return null in its place.
4. Project as primitive strict - Return a field specific primitive value if
a field mismatches the type the query is failed.
5. Project as variant - project a field as a variant type allowing for mixed
types to be returned (this would also be projecting the top level variant to
get the original value).
# Primitive inconsistencies
Assume for all these cases a column is shredded as int32 as its typed value.
## Inconsistency: Both typed_value and value are present for the same cell
and the values are consistent with one another (e.g. all values in `value`
column are int32 and exactly equal to those in `typed_value`
*Operation*: Project field as int32 non-strict. In this case engines would
prefer to always use typed_value and never need to read `value`
*Operation*: Project field as string not-strict. In this case only `value`
column would need to be read but would return consistent results as using
`typed_value`
*Operation*: Project field as int32 as strict. If it is required that a
field that can be projected always is (IIUC this isn't currently mandated by
the spec), then this operation should fail on statististics as there would be a
`non-null` `value` column. Otherwise this effectively requires the same logic
projecting a variant described below.
*Operation*: Project field as string strict. This would fail based on
stastistics from `typed_value`.
*Operation*: Project field as variant. In this case both `typed_value` and
`value` would need to be read and merged. When merging the reader could chose
to always take the non-null value from `typed_value` ignoring any present
values in `value` (assuming corresponding values are null). This would however
lead to a strange state where after all values are merged, fewer cells from
`value` would be read then are present. An alternative would be check for
consistency between repetition/definition levels, and realize they are both
defined and still favor `typed_value`, or check for consistency and continue
reading as long as `value` and `typed_value` are consistent.
## Inconsistency: Both type_value and value are present for the same cell
and are inconsistent with one another (i.e. `value` contains a string).
*Operation*: Project field as int32 non-strict. In this case engines would
prefer to always use typed_value and never need to read `value`. int32 value
is returned over the string value
*Operation*: Project field as string not-strict. In this case only `value`
column would need to be read but would return inconsistent string value in the
cells that conflict.
*Operation*: Project field as int32 as strict. Same as data inconsistency
above.
*Operation*: Project field as string as strict. Same as data inconsistency
above (fail based on statistics).
*Operation*: Project as variant. Same as data inconsistency above, but if
the reader is checking for consistency between values to decide on what to do,
as the current spec is written it would arbitrarily choose int32 value (valid
alternatives seem like failure or returning the `string` value.
# Object inconsistencies
In this case assume there the object is shredded with a single child field
"a" as int32. Objects overlap with the cases outlined above for
consistency/inconsistency so they are not covered here, only somewhat novel
cases are discussed below (merging is taken one step further because field
overlap either must be detected or duplicate fields could be added when
reconstructing the full variant).
## Inconsistency: The `value` object contains a non-object value (e.g.
string) in the same cell that has a present value for the shredded column "a".
*Operation*: Project as as variant. In this case as I read the spec, it is
valid for readers to return `{a: <some int value>}`. Other valid options seem
like (use the non-object `value` or fail).
## Inconsistency: The `typed_value` object is marked as non-present but the
`value column` is present and contains object.
*Operation*: Project field as object only keeping fields "a" non-strict. In
this case readers would ignore the `value` column based on the spec and return
null for the the inconsistency fields.
*Operation*: Project field only keep field "c" as variant. In this case the
reader should be able to ignore `typed_value` (it only contains an object "a"
as a shredded column so it can be ignored) and return `{"c": <some value>}` but
that is not what the specification states.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@parquet.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@parquet.apache.org
For additional commands, e-mail: issues-h...@parquet.apache.org
