rdblue commented on code in PR #461:
URL: https://github.com/apache/parquet-format/pull/461#discussion_r1859143649
##########
VariantShredding.md:
##########
@@ -25,290 +25,316 @@
The Variant type is designed to store and process semi-structured data
efficiently, even with heterogeneous values.
Query engines encode each Variant value in a self-describing format, and store
it as a group containing `value` and `metadata` binary fields in Parquet.
Since data is often partially homogenous, it can be beneficial to extract
certain fields into separate Parquet columns to further improve performance.
-We refer to this process as **shredding**.
-Each Parquet file remains fully self-describing, with no additional metadata
required to read or fully reconstruct the Variant data from the file.
-Combining shredding with a binary residual provides the flexibility to
represent complex, evolving data with an unbounded number of unique fields
while limiting the size of file schemas, and retaining the performance benefits
of a columnar format.
+This process is **shredding**.
-This document focuses on the shredding semantics, Parquet representation,
implications for readers and writers, as well as the Variant reconstruction.
-For now, it does not discuss which fields to shred, user-facing API changes,
or any engine-specific considerations like how to use shredded columns.
-The approach builds upon the [Variant Binary Encoding](VariantEncoding.md),
and leverages the existing Parquet specification.
+Shredding enables the use of Parquet's columnar representation for more
compact data encoding, column statistics for data skipping, and partial
projections.
-At a high level, we replace the `value` field of the Variant Parquet group
with one or more fields called `object`, `array`, `typed_value`, and
`variant_value`.
-These represent a fixed schema suitable for constructing the full Variant
value for each row.
+For example, the query `SELECT variant_get(event, '$.event_ts', 'timestamp')
FROM tbl` only needs to load field `event_ts`, and if that column is shredded,
it can be read by columnar projection without reading or deserializing the rest
of the `event` Variant.
+Similarly, for the query `SELECT * FROM tbl WHERE variant_get(event,
'$.event_type', 'string') = 'signup'`, the `event_type` shredded column
metadata can be used for skipping and to lazily load the rest of the Variant.
-Shredding allows a query engine to reap the full benefits of Parquet's
columnar representation, such as more compact data encoding, min/max statistics
for data skipping, and I/O and CPU savings from pruning unnecessary fields not
accessed by a query (including the non-shredded Variant binary data).
-Without shredding, any query that accesses a Variant column must fetch all
bytes of the full binary buffer.
-With shredding, we can get nearly equivalent performance as in a relational
(scalar) data model.
+## Variant Metadata
-For example, `select variant_get(variant_col, ‘$.field1.inner_field2’,
‘string’) from tbl` only needs to access `inner_field2`, and the file scan
could avoid fetching the rest of the Variant value if this field was shredded
into a separate column in the Parquet schema.
-Similarly, for the query `select * from tbl where variant_get(variant_col,
‘$.id’, ‘integer’) = 123`, the scan could first decode the shredded `id`
column, and only fetch/decode the full Variant value for rows that pass the
filter.
+Variant metadata is stored in the top-level Variant group in a binary
`metadata` column regardless of whether the Variant value is shredded.
-# Parquet Example
+All `value` columns within the Variant must use the same `metadata`.
+All field names of a Variant, whether shredded or not, must be present in the
metadata.
-Consider the following Parquet schema together with how Variant values might
be mapped to it.
-Notice that we represent each shredded field in `object` as a group of two
fields, `typed_value` and `variant_value`.
-We extract all homogenous data items of a certain path into `typed_value`, and
set aside incompatible data items in `variant_value`.
-Intuitively, incompatibilities within the same path may occur because we store
the shredding schema per Parquet file, and each file can contain several row
groups.
-Selecting a type for each field that is acceptable for all rows would be
impractical because it would require buffering the contents of an entire file
before writing.
+## Value Shredding
-Typically, the expectation is that `variant_value` exists at every level as an
option, along with one of `object`, `array` or `typed_value`.
-If the actual Variant value contains a type that does not match the provided
schema, it is stored in `variant_value`.
-An `variant_value` may also be populated if an object can be partially
represented: any fields that are present in the schema must be written to those
fields, and any missing fields are written to `variant_value`.
-
-The `metadata` column is unchanged from its unshredded representation, and may
be referenced in `variant_value` fields in the shredded data.
+Variant values are stored in Parquet fields named `value`.
+Each `value` field may have an associated shredded field named `typed_value`
that stores the value when it matches a specific type.
+When `typed_value` is present, readers **must** reconstruct shredded values
according to this specification.
+For example, a Variant field, `measurement` may be shredded as long values by
adding `typed_value` with type `int64`:
```
-optional group variant_col {
- required binary metadata;
- optional binary variant_value;
- optional group object {
- optional group a {
- optional binary variant_value;
- optional int64 typed_value;
- }
- optional group b {
- optional binary variant_value;
- optional group object {
- optional group c {
- optional binary variant_value;
- optional binary typed_value (STRING);
- }
- }
- }
- }
+required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
}
```
-| Variant Value | Top-level variant_value | b.variant_value | a.typed_value |
a.variant_value | b.object.c.typed_value | b.object.c.variant_value | Notes |
-|---------------|-------------------------|-----------------|---------------|-----------------|------------------------|--------------------------|-------|
-| {a: 123, b: {c: “hello”}} | null | null | 123 | null | hello | null | All
values shredded |
-| {a: 1.23, b: {c: “123”}} | null | null | null | 1.23 | 123 | null | a is not
an integer |
-| {a: 123, b: {c: null}} | null | null | null | 123 | null | null | b.object.c
set to non-null to indicate VariantNull |
-| {a: 123, b: {} | null | null | null | 123 | null | null | b.object.c set to
null, to indicate that c is missing |
-| {a: 123, d: 456} | {d: 456} | null | 123 | null | null | null | Extra field
d is stored as variant_value |
-| [{a: 1, b: {c: 2}}, {a: 3, b: {c: 4}}] | [{a: 1, b: {c: 2}}, {a: 3, b: {c:
4}}] | null | null | null | null | null | Not an object |
+The series of measurements `34, null, "n/a", 100` would be stored as:
-# Parquet Layout
+| Value | `metadata` | `value` | `typed_value` |
+|---------|------------------|-----------------------|---------------|
+| 34 | `01 00` v1/empty | null | `34` |
+| null | `01 00` v1/empty | `00` (null) | null |
+| "n/a" | `01 00` v1/empty | `13 6E 2F 61` (`n/a`) | null |
+| 100 | `01 00` v1/empty | null | `100` |
-The `array` and `object` fields represent Variant array and object types,
respectively.
-Arrays must use the three-level list structure described in
[LogicalTypes.md](LogicalTypes.md).
+Both `value` and `typed_value` are optional fields used together to encode a
single value.
+Values in the two fields must be interpreted according to the following table:
-An `object` field must be a group.
-Each field name of this inner group corresponds to the Variant value's object
field name.
-Each inner field's type is a recursively shredded variant value: that is, the
fields of each object field must be one or more of `object`, `array`,
`typed_value` or `variant_value`.
+| `value` | `typed_value` | Meaning
|
+|----------|---------------|-------------------------------------------------------------|
+| null | null | The value is missing; only valid for shredded
object fields |
+| non-null | null | The value is present and may be any type,
including null |
+| null | non-null | The value is present and is the shredded type
|
+| non-null | non-null | The value is present and is a partially shredded
object |
-Similarly the elements of an `array` must be a group containing one or more of
`object`, `array`, `typed_value` or `variant_value`.
+An object is _partially shredded_ when the `value` is an object and the
`typed_value` is a shredded object.
-Each leaf in the schema can store an arbitrary Variant value.
-It contains an `variant_value` binary field and a `typed_value` field.
-If non-null, `variant_value` represents the value stored as a Variant binary.
-The `typed_value` field may be any type that has a corresponding Variant type.
-For each value in the data, at most one of the `typed_value` and
`variant_value` may be non-null.
-A writer may omit either field, which is equivalent to all rows being null.
+If both fields are non-null and either is not an object, the value is invalid.
Readers must either fail or return the `typed_value`.
-Dictionary IDs in a `variant_value` field refer to entries in the top-level
`metadata` field.
+If a Variant is missing in a context where a value is required, readers must
either fail or return a Variant null: basic type 0 (primitive) and physical
type 0 (null).
+For example, if a Variant is required (like `measurement` above) and both
`value` and `typed_value` are null, the returned `value` must be `00` (Variant
null).
-For an `object`, a null field means that the field does not exist in the
reconstructed Variant object.
-All elements of an `array` must be non-null, since array elements cannote be
missing.
+### Shredded Value Types
-| typed_value | variant_value | Meaning |
-|-------------|----------------|---------|
-| null | null | Field is Variant Null (not missing) in the reconstructed
Variant. |
-| null | non-null | Field may be any type in the reconstructed Variant. |
-| non-null | null | Field has this column’s type in the reconstructed Variant.
|
-| non-null | non-null | Invalid |
+Shredded values must use the following Parquet types:
-The `typed_value` may be absent from the Parquet schema for any field, which
is equivalent to its value being always null (in which case the shredded field
is always stored as a Variant binary).
-By the same token, `variant_value` may be absent, which is equivalent to their
value being always null (in which case the field will always have the value
Null or have the type of the `typed_value` column).
+| Variant Type | Equivalent Parquet Type |
+|-----------------------------|------------------------------|
+| boolean | BOOLEAN |
+| int8 | INT(8, signed=true) |
+| int16 | INT(16, signed=true) |
+| int32 | INT32 / INT(32, signed=true) |
+| int64 | INT64 / INT(64, signed=true) |
+| float | FLOAT |
+| double | DOUBLE |
+| decimal4 | DECIMAL(precision, scale) |
+| decimal8 | DECIMAL(precision, scale) |
+| decimal16 | DECIMAL(precision, scale) |
+| date | DATE |
+| timestamp | TIMESTAMP(true, MICROS) |
+| timestamp without time zone | TIMESTAMP(false, MICROS) |
+| binary | BINARY |
+| string | STRING |
+| array | LIST; see Arrays below |
+| object | GROUP; see Objects below |
-# Unshredded values
+#### Primitive Types
-If all values can be represented at a given level by whichever of `object`,
`array`, or `typed_value` is present, `variant_value` is set to null.
+Primitive values can be shredded using the equivalent Parquet primitive type
from the table above for `typed_value`.
-If a value cannot be represented by whichever of `object`, `array`, or
`typed_value` is present in the schema, then it is stored in `variant_value`,
and the other fields are set to null.
-In the Parquet example above, if field `a` was an object or array, or a
non-integer scalar, it would be stored in `variant_value`.
+Unless the value is shredded as an object (see [Objects](#objects)),
`typed_value` or `value` (but not both) must be non-null.
-If a value is an object, and the `object` field is present but does not
contain all of the fields in the value, then any remaining fields are stored in
an object in `variant_value`.
-In the Parquet example above, if field `b` was an object of the form `{"c": 1,
"d": 2}"`, then the object `{"d": 2}` would be stored in `variant_value`, and
the `c` field would be shredded recursively under `object.c`.
+#### Arrays
-Note that an array is always fully shredded if there is an `array` field, so
the above consideration for `object` is not relevant for arrays: only one of
`array` or `variant_value` may be non-null at a given level.
+Arrays can be shredded using a 3-level Parquet list for `typed_value`.
-# Using variant_value vs. typed_value
+If the value is not an array, `typed_value` must be null.
+If the value is an array, `value` must be null.
-In general, it is desirable to store values in the `typed_value` field rather
than the `variant_value` whenever possible.
-This will typically improve encoding efficiency, and allow the use of Parquet
statistics to filter at the row group or page level.
-In the best case, the `variant_value` fields are all null and the engine does
not need to read them (or it can omit them from the schema on write entirely).
-There are two main motivations for including the `variant_value` column:
+The list `element` must be a required group that contains `value` and
`typed_value` fields.
+The element's `value` field stores the element as Variant-encoded `binary`
when the `typed_value` is not present or cannot represent it.
+The `typed_value` field may be omitted when not shredding elements as a
specific type.
+When `typed_value` is omitted, `value` must be `required`.
-1) In a case where there are rare type mismatches (for example, a numeric
field with rare strings like “n/a”), we allow the field to be shredded, which
could still be a significant performance benefit compared to fetching and
decoding the full value/metadata binary.
-2) Since there is a single schema per file, there would be no easy way to
recover from a type mismatch encountered late in a file write. Parquet files
can be large, and buffering all file data before starting to write could be
expensive. Including a variant column for every field guarantees we can adhere
to the requested shredding schema.
+For example, a `tags` Variant may be shredded as a list of strings using the
following definition:
+```
+optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # must be optional to allow a null list
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+}
+```
-# Top-level metadata
+All elements of an array must be present (not missing) because the `array`
Variant encoding does not allow missing elements.
+That is, either `typed_value` or `value` (but not both) must be non-null.
+Null elements must be encoded in `value` as Variant null: basic type 0
(primitive) and physical type 0 (null).
-Any values stored in a shredded `variant_value` field may have dictionary IDs
referring to the metadata.
-There is one metadata value for the entire Variant record, and that is stored
in the top-level `metadata` field.
-This means any `variant_value` values in the shredded representation is only
the "value" portion of the [Variant Binary Encoding](VariantEncoding.md).
+The series of `tags` arrays `["comedy", "drama"], ["horror", null], ["comedy",
"drama", "romance"], null` would be stored as:
-The metadata is kept at the top-level, instead of shredding the metadata with
the shredded variant values because:
-* Simplified shredding scheme and specification. No need for additional
struct-of-binary values, or custom concatenated binary scheme for
`variant_value`.
-* Simplified and good performance for write shredding. No need to rebuild the
metadata, or re-encode IDs for `variant_value`.
-* Simplified and good performance for Variant reconstruction. No need to
re-encode IDs for `variant_value`.
+| Array | `value` | `typed_value `|
`typed_value...value` | `typed_value...typed_value` |
+|----------------------------------|-------------|---------------|-----------------------|--------------------------------|
+| `["comedy", "drama"]` | null | non-null | [null,
null] | [`comedy`, `drama`] |
+| `["horror", null]` | null | non-null | [null,
`00`] | [`horror`, null] |
+| `["comedy", "drama", "romance"]` | null | non-null | [null,
null, null] | [`comedy`, `drama`, `romance`] |
+| null | `00` (null) | null |
| |
-# Data Skipping
+#### Objects
-Shredded columns are expected to store statistics in the same format as a
normal Parquet column.
-In general, the engine can only skip a row group or page if all rows in the
`variant_value` field are null, since it is possible for a `variant_get`
expression to successfully cast a value from the `variant_value` to the target
type.
-For example, if `typed_value` is of type `int64`, then the string “123” might
be contained in `variant_value`, which would not be reflected in statistics,
but could be retained by a filter like `where variant_get(col, “$.field”,
“long”) = 123`.
-If `variant_value` is all-null, then the engine can prune pages or row groups
based on `typed_value`.
-This specification is not strict about what values may be stored in
`variant_value` rather than `typed_value`, so it is not safe to skip rows based
on `typed_value` unless the corresponding `variant_value` column is all-null,
or the engine has specific knowledge of the behavior of the writer that
produced the shredded data.
+Fields of an object can be shredded using a Parquet group for `typed_value`
that contains shredded fields.
-# Shredding Semantics
+If the value is an object, `typed_value` must be non-null.
+If the value is not an object, `typed_value` must be null.
-Reconstruction of Variant value from a shredded representation is not expected
to produce a bit-for-bit identical binary to the original unshredded value.
-For example, in a reconstructed Variant value, the order of object field
values may be different from the original binary.
-This is allowed since the [Variant Binary
Encoding](VariantEncoding.md#object-field-id-order-and-uniqueness) does not
require an ordering of the field values, but the field IDs will still be
ordered lexicographically according to the corresponding field names.
+Each shredded field in the `typed_value` group is represented as a required
group that contains optional `value` and `typed_value` fields.
+The `value` field stores the value as Variant-encoded `binary` when the
`typed_value` cannot represent the field.
+This layout enables readers to skip data based on the field statistics for
`value` and `typed_value`.
-The physical representation of scalar values may also be different in the
reconstructed Variant binary.
-In particular, the [Variant Binary Encoding](VariantEncoding.md) considers all
integer and decimal representations to represent a single logical type.
-This flexibility enables shredding to be applicable in more scenarios, while
maintaining all information and values losslessly.
-As a result, it is valid to shred a decimal into a decimal column with a
different scale, or to shred an integer as a decimal, as long as no numeric
precision is lost.
-For example, it would be valid to write the value 123 to a Decimal(9, 2)
column, but the value 1.234 would need to be written to the `variant_value`
column.
-When reconstructing, it would be valid for a reader to reconstruct 123 as an
integer, or as a Decimal(9, 2).
-Engines should not depend on the physical type of a Variant value, only the
logical type.
+If the value is a partially shredded object, the `value` must not contain the
shredded fields.
+If shredded fields are present in the variant object, it is invalid and
readers must either fail or use the shredded values.
-On the other hand, shredding as a different logical type is not allowed.
-For example, the integer value 123 could not be shredded to a string
`typed_value` column as the string "123", since that would lose type
information.
-It would need to be written to the `variant_value` column.
+For example, a Variant `event` field may shred `event_type` (`string`) and
`event_ts` (`timestamp`) columns using the following definition:
+```
+optional group event (VARIANT) {
+ required binary metadata;
+ optional binary value; # a variant, expected to be an object
+ optional group typed_value { # shredded fields for the variant
object
+ required group event_type { # shredded field for event_type
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ required group event_ts { # shredded field for event_ts
+ optional binary value;
+ optional int64 typed_value (TIMESTAMP(true, MICROS));
+ }
+ }
+}
+```
-# Reconstructing a Variant
+The group for each named field must be required.
-It is possible to recover a full Variant value using a recursive algorithm,
where the initial call is to `ConstructVariant` with the top-level fields,
which are assumed to be null if they are not present in the schema.
+A field's `value` and `typed_value` are set to null (missing) to indicate that
the field does not exist in the variant.
+To encode a field that is present with a null value, the `value` must contain
a Variant null: basic type 0 (primitive) and physical type 0 (null).
-```
-# Constructs a Variant from `variant_value`, `object`, `array` and
`typed_value`.
-# Only one of object, array and typed_value may be non-null.
-def ConstructVariant(variant_value, object, array, typed_value):
- if object is null and array is null and typed_value is null and
variant_value is null: return VariantNull
- if object is not null:
- return ConstructObject(variant_value, object)
- elif array is not null:
- return ConstructArray(array)
- elif typed_value is not null:
- return cast(typed_value as Variant)
- else:
- variant_value
-
-# Construct an object from an `object` group, and a (possibly null) Variant
variant_value
-def ConstructObject(variant_value, object):
- # If variant_value is present and is not an Object, then the result is
ambiguous.
- assert(variant_value is null or is_object(variant_value))
- # Null fields in the object are missing from the reconstructed Variant.
- nonnull_object_fields = object.fields.filter(field -> field is not null)
- all_keys = Union(variant_value.keys, non_null_object_fields)
- return VariantObject(all_keys.map { key ->
- if key in object: (key, ConstructVariant(object[key].variant_value,
object[key].object, object[key].array, object[key].typed_value))
- else: (key, variant_value[key])
- })
-
-def ConstructArray(array):
- newVariantArray = VariantArray()
- for i in range(array.size):
- newVariantArray.append(ConstructVariant(array[i].variant_value,
array[i].object, array[i].array, array[i].typed_value)
-```
+The series of objects below would be stored as:
-# Nested Parquet Example
+| Event object
| `value` | `typed_value` |
`typed_value.event_type.value` | `typed_value.event_type.typed_value` |
`typed_value.event_ts.value` | `typed_value.event_ts.typed_value` | Notes
|
+|------------------------------------------------------------------------------------|-----------------------------------|---------------|--------------------------------|--------------------------------------|------------------------------|------------------------------------|--------------------------------------------------|
+| `{"event_type": "noop", "event_ts": 1729794114937}`
| null | non-null | null
| `noop` | null
| 1729794114937 | Fully shredded object
|
+| `{"event_type": "login", "event_ts": 1729794146402, "email":
"u...@example.com"}` | `{"email": "u...@example.com"}` | non-null |
null | `login` | null
| 1729794146402 | Partially shredded
object |
+| `{"error_msg": "malformed: ..."}`
| `{"error_msg", "malformed: ..."}` | non-null | null
| null | null
| null | Object with all shredded fields missing
|
+| `"malformed: not an object"`
| `malformed: not an object` | null |
| |
| | Not an object (stored as Variant
string) |
+| `{"event_ts": 1729794240241, "click": "_button"}`
| `{"click": "_button"}` | non-null | null
| null | null
| 1729794240241 | Field `event_type` is missing
|
+| `{"event_type": null, "event_ts": 1729794954163}`
| null | non-null | `00` (field exists,
is null) | null | null
| 1729794954163 | Field `event_type` is present and is
null |
+| `{"event_type": "noop", "event_ts": "2024-10-24"`
| null | non-null | null
| `noop` | `"2024-10-24"`
| null | Field `event_ts` is present but not a
timestamp |
+| `{ }`
| null | non-null | null
| null | null
| null | Object is present but empty
|
+| null
| `00` (null) | null |
| |
| | Object/value is null
|
+| missing
| null | null |
| |
| | Object/value is missing
|
+| INVALID
| `{"event_type": "login"}` | non-null | null
| `login` | null
| 1729795057774 | INVALID: Shredded field is present in
`value` |
+| INVALID
| `"a"` | non-null | null
| `login` | null
| 1729795057774 | INVALID: `typed_value` is present for
non-object |
+| INVALID
| `02 00` (object with 0 fields) | null |
| |
| | INVALID: `typed_value` is null for
object |
-This section describes a more deeply nested example, using a top-level array
as the shredding type.
+Invalid cases in the table above must not be produced by writers.
+Readers must return an object when `typed_value` is non-null containing the
shredded fields.
+If `typed_value` is null and `value` is an object, readers may read the
encoded object but are not required to do so.
-Below is a sample of JSON that would be fully shredded in this example.
-It contains an array of objects, containing an `a` field shredded as an array,
and a `b` field shredded as an integer.
+Readers can assume that a value is not an object if `typed_value` is null and
that `typed_value` field values are correct; that is, readers do not need to
read the `value` column if `typed_value` fields satisfy the required fields.
-```
-[
- {
- "a": [1, 2, 3],
- "b": 100
- },
- {
- "a": [4, 5, 6],
- "b": 200
- }
-]
-```
+## Nesting
+The `typed_value` associated with any Variant `value` field can be any
shredded type, as shown in the sections above.
-The corresponding Parquet schema with “a” and “b” as leaf types is:
+For example, the `event` object above may also shred sub-fields as object
(`location`) or array (`tags`).
```
-optional group variant_col {
- required binary metadata;
- optional binary variant_value;
- optional group array (LIST) {
- repeated group list {
- optional group element {
- optional binary variant_value;
- optional group object {
- optional group a {
- optional binary variant_value;
- optional group array (LIST) {
- repeated group list {
- optional group element {
- optional int64 typed_value;
- optional binary variant_value;
+optional group event (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value {
+ required group event_type {
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ required group event_ts {
+ optional binary value;
+ optional int64 typed_value (TIMESTAMP(true, MICROS));
+ }
+ required group location {
+ optional binary value;
+ optional group typed_value {
+ required group latitude {
+ optional binary value;
+ optional double typed_value;
+ }
+ required group longitude {
+ optional binary value;
+ optional double typed_value;
+ }
+ }
+ }
+ required group tags {
+ optional binary value;
+ optional group typed_value (LIST) {
+ repeated group list {
+ required group element {
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
}
- }
}
- }
- optional group b {
- optional int64 typed_value;
- optional binary variant_value;
- }
}
- }
}
- }
}
```
-In the above example schema, if “a” is an array containing a mix of integer
and non-integer values, the engine will shred individual elements appropriately
into either `typed_value` or `variant_value`.
-If the top-level Variant is not an array (for example, an object), the engine
cannot shred the value and it will store it in the top-level `variant_value`.
-Similarly, if "a" is not an array, it will be stored in the `variant_value`
under "a".
+# Data Skipping
-Consider the following example:
+Statistics for `typed_value` columns can be used for file, row group, or page
skipping when `value` is always null (missing).
-```
-[
- {
- "a": [1, 2, 3],
- "b": 100,
- “c”: “unexpected”
- },
- {
- "a": [4, 5, 6],
- "b": 200
- },
- “not an object”
-]
-```
+When the corresponding `value` column is all nulls, all values must be the
shredded `typed_value` field's type.
+Because the type is known, comparisons with values of that type are valid.
+`IS NULL`/`IS NOT NULL` and `IS NAN`/`IS NOT NAN` filter results are also
valid.
-The second array element can be fully shredded, but the first and third cannot
be. The contents of `variant_col.array[*].variant_value` would be as follows:
+Comparisons with values of other types are not necessarily valid and data
should not be skipped.
-```
-[
- { “c”: “unexpected” },
- NULL,
- “not an object”
-]
+Casting behavior for Variant is delegated to processing engines.
+For example, the interpretation of a string as a timestamp may depend on the
engine's SQL session time zone.
+
+## Reconstructing a Shredded Variant
+
+It is possible to recover an unshredded Variant value using a recursive
algorithm, where the initial call is to `construct_variant` with the top-level
Variant group fields.
+
+```python
+def construct_variant(metadata: Metadata, value: Variant, typed_value: Any) ->
Variant:
+ """Constructs a Variant from value and typed_value"""
+ if typed_value is not None:
+ if isinstance(typed_value, dict):
+ # this is a shredded object
+ object_fields = {
+ name: construct_variant(metadata, field.value,
field.typed_value)
+ for (name, field) in typed_value
+ }
+
+ if value is not None:
+ # this is a partially shredded object
+ assert isinstance(value, VariantObject), "partially shredded
value must be an object"
+ assert typed_value.keys().isdisjoint(value.keys()), "object
keys must be disjoint"
+
+ # union the shredded fields and non-shredded fields
+ return VariantObject(metadata,
object_fields).union(VariantObject(metadata, value))
+
+ else:
+ return VariantObject(metadata, object_fields)
+
+ elif isinstance(typed_value, list):
+ # this is a shredded array
+ assert value is None, "shredded array must not conflict with
variant value"
+
+ elements = [
+ construct_variant(metadata, elem.value, elem.typed_value)
+ for elem in list(typed_value)
+ ]
+ return VariantArray(metadata, elements)
+
+ else:
+ # this is a shredded primitive
+ assert value is None, "shredded primitive must not conflict with
variant value"
+
+ return primitive_to_variant(typed_value)
+
+ elif value is not None:
+ return Variant(metadata, value)
+
+ else:
+ # value is missing
+ return None
+
+def primitive_to_variant(typed_value: Any): Variant:
+ if isinstance(typed_value, int):
+ return VariantInteger(typed_value)
+ elif isinstance(typed_value, str):
+ return VariantString(typed_value)
+ ...
```
-# Backward and forward compatibility
-Shredding is an optional feature of Variant, and readers must continue to be
able to read a group containing only a `value` and `metadata` field.
+## Backward and forward compatibility
-Any fields in the same group as `typed_value`/`variant_value` that start with
`_` (underscore) can be ignored.
-This is intended to allow future backwards-compatible extensions.
-In particular, the field names `_metadata_key_paths` and any name starting
with `_spark` are reserved, and should not be used by other implementations.
-Any extra field names that do not start with an underscore should be assumed
to be backwards incompatible, and readers should fail when reading such a
schema.
+Shredding is an optional feature of Variant, and readers must continue to be
able to read a group containing only `value` and `metadata` fields.
Engines without shredding support are not expected to be able to read Parquet
files that use shredding.
-Since different files may contain conflicting schemas (e.g. a `typed_value`
column with incompatible types in two files), it may not be possible to infer
or specify a single schema that would allow all Parquet files for a table to be
read.
+Different files may contain conflicting schemas.
+That is, files may contain different `typed_value` columns for the same
Variant with incompatible types.
Review Comment:
Utilities for concatenating Parquet files still require the same physical
schema. For instance, concatenating a list of required elements with a list of
optional elements is not possible without rewriting because the repetition
level of the list element must match the encoding. This would be no different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@parquet.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@parquet.apache.org
For additional commands, e-mail: issues-h...@parquet.apache.org
