This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/parquet-format.git
The following commit(s) were added to refs/heads/master by this push:
new 325b9f4 Fix errors and inconsistencies in Variant format
documentation (#574)
325b9f4 is described below
commit 325b9f4a70663c88485d4d59fdc1a121d7497943
Author: Ismaël Mejía <[email protected]>
AuthorDate: Mon Jun 15 18:23:34 2026 +0200
Fix errors and inconsistencies in Variant format documentation (#574)
* Fix errors and inconsistencies in Variant format documentation
VariantEncoding.md:
- Fix BINARY -> BYTE_ARRAY (BINARY is not a Parquet physical type)
- Add note on decimal little-endian vs big-endian difference
- Fix decimal implied-precision formula for val <= 0
- Label undocumented reserved bits in metadata/object/array headers
- Make sorted_strings description consistent across three definitions
- Use INT(N, true) notation consistent with LogicalTypes.md
- Hyphenate compound adjectives ("3 byte" -> "3-byte", etc.)
VariantShredding.md:
- Fix Python syntax error: iterating dict yields keys only;
add .items() for (name, field) unpacking
- Replace BINARY with BYTE_ARRAY
- Fix comma -> colon inside JSON-like literal in table cell
- Remove trailing space inside backticks in table header
- Use INT(N, true) notation consistent with LogicalTypes.md
* Address review: drop redundant parentheses in dict comprehension
* Address review: drop writer mandate for reserved bits
Revert 'must be set to 0 by writers' wording for reserved bits.
As noted in review, mandating writer behavior changes the spec and
would need mailing list discussion. Keep only the reader requirement.
---
VariantEncoding.md | 49 ++++++++++++++++++++++++++++---------------------
VariantShredding.md | 16 ++++++++--------
2 files changed, 36 insertions(+), 29 deletions(-)
diff --git a/VariantEncoding.md b/VariantEncoding.md
index b78c02e..adfac53 100644
--- a/VariantEncoding.md
+++ b/VariantEncoding.md
@@ -77,7 +77,7 @@ The encoded metadata always starts with a header byte.
```
7 6 5 4 3 0
+-------+---+---+---------------+
-header | | | | version |
+header | | R | | version |
+-------+---+---+---------------+
^ ^
| +-- sorted_strings
@@ -87,6 +87,7 @@ The `version` is a 4-bit value that must always contain the
value `1`.
`sorted_strings` is a 1-bit value indicating whether dictionary strings are
sorted and unique.
`offset_size_minus_one` is a 2-bit value providing the number of bytes per
dictionary size and offset field.
The actual number of bytes, `offset_size`, is `offset_size_minus_one + 1`.
+Bit 5 (marked `R`) is reserved; it must be ignored by readers.
The entire metadata is encoded as the following diagram shows:
```
@@ -129,7 +130,7 @@ The grammar for encoded metadata is as follows
metadata: <header> <dictionary_size> <dictionary>
header: 1 byte (<version> | <sorted_strings> << 4 | (<offset_size_minus_one>
<< 6))
version: a 4-bit version ID. Currently, must always contain the value 1
-sorted_strings: a 1-bit value indicating whether metadata strings are sorted
+sorted_strings: a 1-bit value indicating whether dictionary strings are sorted
and unique
offset_size_minus_one: 2-bit value providing the number of bytes per
dictionary size and offset field.
dictionary_size: `offset_size` bytes. unsigned little-endian value indicating
the number of strings in the dictionary
dictionary: <offset>* <bytes>
@@ -141,7 +142,7 @@ Notes:
- Offsets are relative to the start of the `bytes` array.
- The length of the ith string can be computed as `offset[i+1] - offset[i]`.
- The offset of the first string is always equal to 0 and is therefore
redundant. It is included in the spec to simplify in-memory-processing.
-- `offset_size_minus_one` indicates the number of bytes per `dictionary_size`
and `offset` entry. I.e. a value of 0 indicates 1-byte offsets, 1 indicates
2-byte offsets, 2 indicates 3 byte offsets and 3 indicates 4-byte offsets.
+- `offset_size_minus_one` indicates the number of bytes per `dictionary_size`
and `offset` entry. I.e. a value of 0 indicates 1-byte offsets, 1 indicates
2-byte offsets, 2 indicates 3-byte offsets and 3 indicates 4-byte offsets.
- If `sorted_strings` is set to 1, strings in the dictionary must be unique
and sorted in lexicographic order. If the value is set to 0, readers may not
make any assumptions about string order or uniqueness.
@@ -195,7 +196,7 @@ When `basic_type` is `2`, `value_header` is made up of
`field_offset_size_minus_
```
5 4 3 2 1 0
+---+---+-------+-------+
-value_header | | | | |
+value_header | R | | | |
+---+---+-------+-------+
^ ^ ^
| | +-- field_offset_size_minus_one
@@ -206,6 +207,7 @@ value_header | | | | |
The actual number of bytes is computed as `field_offset_size_minus_one + 1`
and `field_id_size_minus_one + 1`.
`is_large` is a 1-bit value that indicates how many bytes are used to encode
the number of elements.
If `is_large` is `0`, 1 byte is used, and if `is_large` is `1`, 4 bytes are
used.
+Bit 5 (marked `R`) is reserved; it must be ignored by readers.
#### Value Header for Array (`basic_type`=3)
@@ -213,7 +215,7 @@ When `basic_type` is `3`, `value_header` is made up of
`field_offset_size_minus_
```
5 3 2 1 0
+-----------+---+-------+
-value_header | | | |
+value_header | RRR | | |
+-----------+---+-------+
^ ^
| +-- field_offset_size_minus_one
@@ -223,6 +225,7 @@ value_header | | | |
The actual number of bytes is computed as `field_offset_size_minus_one + 1`.
`is_large` is a 1-bit value that indicates how many bytes are used to encode
the number of elements.
If `is_large` is `0`, 1 byte is used, and if `is_large` is `1`, 4 bytes are
used.
+Bits 5-3 (marked `RRR`) are reserved; they must be ignored by readers.
### Value Data
@@ -367,9 +370,9 @@ primitive_val: see table for binary representation
short_string_val: UTF-8 encoded bytes
object_val: <num_elements> <field_id>* <field_offset>* <fields>
array_val: <num_elements> <field_offset>* <fields>
-num_elements: a 1 or 4 byte unsigned little-endian value (depending on
is_large in <object_header>/<array_header>)
-field_id: a 1, 2, 3 or 4 byte unsigned little-endian value (depending on
field_id_size_minus_one in <object_header>), indexing into the dictionary
-field_offset: a 1, 2, 3 or 4 byte unsigned little-endian value (depending on
field_offset_size_minus_one in <object_header>/<array_header>), providing the
offset in bytes within fields
+num_elements: a 1- or 4-byte unsigned little-endian value (depending on
is_large in <object_header>/<array_header>)
+field_id: a 1-, 2-, 3-, or 4-byte unsigned little-endian value (depending on
field_id_size_minus_one in <object_header>), indexing into the dictionary
+field_offset: a 1-, 2-, 3-, or 4-byte unsigned little-endian value (depending
on field_offset_size_minus_one in <object_header>/<array_header>), providing
the offset in bytes within fields
fields: <value>*
```
@@ -380,15 +383,19 @@ The last entry is the offset that is one byte past the
last field (i.e. the tota
All offsets are relative to the first byte of the first field in the
object/array.
`field_id_size_minus_one` and `field_offset_size_minus_one` indicate the
number of bytes per field ID/offset.
-For example, a value of 0 indicates 1-byte IDs, 1 indicates 2-byte IDs, 2
indicates 3 byte IDs and 3 indicates 4-byte IDs.
-The `is_large` flag for arrays and objects is used to indicate whether the
number of elements is indicated using a one or four byte value.
+For example, a value of 0 indicates 1-byte IDs, 1 indicates 2-byte IDs, 2
indicates 3-byte IDs and 3 indicates 4-byte IDs.
+The `is_large` flag for arrays and objects is used to indicate whether the
number of elements is indicated using a one- or four-byte value.
When more than 255 elements are present, `is_large` must be set to true.
It is valid for an implementation to use a larger value than necessary for any
of these fields (e.g. `is_large` may be true for an object with less than 256
elements).
The "short string" basic type may be used as an optimization to fold string
length into the type byte for strings less than 64 bytes.
It is semantically identical to the "string" primitive type.
-The Decimal type contains a scale, but no precision. The implied precision of
a decimal value is `floor(log_10(val)) + 1`.
+The Decimal type contains a scale, but no precision. The implied precision of
a decimal value is `floor(log_10(|val|)) + 1` (and `1` when `val` is `0`).
+
+Note: Decimal values in the Variant binary encoding use little-endian byte
order for the
+unscaled value. This differs from Parquet's DECIMAL logical type which uses
big-endian
+two's complement encoding for `BYTE_ARRAY` and `FIXED_LEN_BYTE_ARRAY` physical
types.
## Encoding types
*Variant basic types*
@@ -407,20 +414,20 @@ The Decimal type contains a scale, but no precision. The
implied precision of a
| NullType | null | `0` | UNKNOWN
| none
|
| Boolean | boolean (True) | `1` | BOOLEAN
| none
|
| Boolean | boolean (False) | `2` | BOOLEAN
| none
|
-| Exact Numeric | int8 | `3` | INT(8,
signed) | 1 byte
|
-| Exact Numeric | int16 | `4` | INT(16,
signed) | 2 byte little-endian
|
-| Exact Numeric | int32 | `5` | INT(32,
signed) | 4 byte little-endian
|
-| Exact Numeric | int64 | `6` | INT(64,
signed) | 8 byte little-endian
|
+| Exact Numeric | int8 | `3` | INT(8, true)
| 1-byte
|
+| Exact Numeric | int16 | `4` | INT(16, true)
| 2-byte little-endian
|
+| Exact Numeric | int32 | `5` | INT(32, true)
| 4-byte little-endian
|
+| Exact Numeric | int64 | `6` | INT(64, true)
| 8-byte little-endian
|
| Double | double | `7` | DOUBLE
| IEEE little-endian
|
-| Exact Numeric | decimal4 | `8` |
DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by
little-endian unscaled value (see decimal table) |
-| Exact Numeric | decimal8 | `9` |
DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by
little-endian unscaled value (see decimal table) |
-| Exact Numeric | decimal16 | `10` |
DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by
little-endian unscaled value (see decimal table) |
-| Date | date | `11` | DATE
| 4 byte little-endian
|
+| Exact Numeric | decimal4 | `8` |
DECIMAL(precision, scale) | 1-byte scale in range [0, 38], followed by
little-endian unscaled value (see decimal table) |
+| Exact Numeric | decimal8 | `9` |
DECIMAL(precision, scale) | 1-byte scale in range [0, 38], followed by
little-endian unscaled value (see decimal table) |
+| Exact Numeric | decimal16 | `10` |
DECIMAL(precision, scale) | 1-byte scale in range [0, 38], followed by
little-endian unscaled value (see decimal table) |
+| Date | date | `11` | DATE
| 4-byte little-endian
|
| Timestamp | timestamp | `12` |
TIMESTAMP(isAdjustedToUTC=true, MICROS) | 8-byte little-endian
|
| TimestampNTZ | timestamp without time zone | `13` |
TIMESTAMP(isAdjustedToUTC=false, MICROS) | 8-byte little-endian
|
| Float | float | `14` | FLOAT
| IEEE little-endian
|
-| Binary | binary | `15` | BINARY
| 4 byte little-endian size, followed by bytes
|
-| String | string | `16` | STRING
| 4 byte little-endian size, followed by UTF-8 encoded bytes
|
+| Binary | binary | `15` | BYTE_ARRAY
| 4-byte little-endian size, followed by bytes
|
+| String | string | `16` | STRING
| 4-byte little-endian size, followed by UTF-8 encoded bytes
|
| TimeNTZ | time without time zone | `17` |
TIME(isAdjustedToUTC=false, MICROS) | 8-byte little-endian
|
| Timestamp | timestamp with time zone | `18` |
TIMESTAMP(isAdjustedToUTC=true, NANOS) | 8-byte little-endian
|
| TimestampNTZ | timestamp without time zone | `19` |
TIMESTAMP(isAdjustedToUTC=false, NANOS) | 8-byte little-endian
|
diff --git a/VariantShredding.md b/VariantShredding.md
index 4f7d614..fe84044 100644
--- a/VariantShredding.md
+++ b/VariantShredding.md
@@ -85,8 +85,8 @@ Shredded values must use the following Parquet types:
| Variant Type | Parquet Physical Type | Parquet
Logical Type |
|-----------------------------|-----------------------------------|--------------------------|
| boolean | BOOLEAN |
|
-| int8 | INT32 | INT(8,
signed=true) |
-| int16 | INT32 | INT(16,
signed=true) |
+| int8 | INT32 | INT(8,
true) |
+| int16 | INT32 | INT(16,
true) |
| int32 | INT32 |
|
| int64 | INT64 |
|
| float | FLOAT |
|
@@ -100,8 +100,8 @@ Shredded values must use the following Parquet types:
| timestamptz(9) | INT64 |
TIMESTAMP(true, NANOS) |
| timestampntz(6) | INT64 |
TIMESTAMP(false, MICROS) |
| timestampntz(9) | INT64 |
TIMESTAMP(false, NANOS) |
-| binary | BINARY |
|
-| string | BINARY | STRING
|
+| binary | BYTE_ARRAY |
|
+| string | BYTE_ARRAY | STRING
|
| uuid | FIXED_LEN_BYTE_ARRAY[len=16] | UUID
|
| array | GROUP; see Arrays below | LIST
|
| object | GROUP; see Objects below |
|
@@ -148,7 +148,7 @@ Null elements must be encoded in `value` as Variant null:
basic type 0 (primitiv
The series of `tags` arrays `["comedy", "drama"], ["horror", null], ["comedy",
"drama", "romance"], null` would be stored as:
-| Array | `value` | `typed_value `|
`typed_value...value` | `typed_value...typed_value` |
+| Array | `value` | `typed_value` |
`typed_value...value` | `typed_value...typed_value` |
|----------------------------------|-------------|---------------|-----------------------|--------------------------------|
| `["comedy", "drama"]` | null | non-null | [null,
null] | [`comedy`, `drama`] |
| `["horror", null]` | null | non-null | [null,
`00`] | [`horror`, null] |
@@ -206,7 +206,7 @@ The table below shows how the series of objects in the
first column would be sto
|-----------------------------------------------------------------------------------|-----------------------------------|---------------|--------------------------------|--------------------------------------|------------------------------|------------------------------------|----------------------------------------------------------------------------|
| `{"event_type": "noop", "event_ts": 1729794114937}`
| null | non-null | null
| `noop` | null
| 1729794114937 | Fully shredded object
|
| `{"event_type": "login", "event_ts": 1729794146402, "email":
"[email protected]"}` | `{"email": "[email protected]"}` | non-null | null
| `login` | null
| 1729794146402 | Partially shredded
object |
-| `{"error_msg": "malformed: ..."}`
| `{"error_msg", "malformed: ..."}` | non-null | null
| null | null
| null | Object with all shredded fields missing
|
+| `{"error_msg": "malformed: ..."}`
| `{"error_msg": "malformed: ..."}` | non-null | null
| null | null
| null | Object with all shredded fields missing
|
| `"malformed: not an object"`
| `malformed: not an object` | null |
| |
| | Not an object (stored as Variant string)
|
| `{"event_ts": 1729794240241, "click": "_button"}`
| `{"click": "_button"}` | non-null | null
| null | null
| 1729794240241 | Field `event_type` is missing
|
| `{"event_type": null, "event_ts": 1729794954163}`
| null | non-null | `00` (field exists,
is null) | null | null
| 1729794954163 | Field `event_type` is present and is
null |
@@ -294,7 +294,7 @@ def construct_variant(metadata: Metadata, value: Variant,
typed_value: Any) -> V
# this is a shredded object
object_fields = {
name: construct_variant(metadata, field.value,
field.typed_value)
- for (name, field) in typed_value
+ for name, field in typed_value.items()
}
if value is not None:
@@ -334,7 +334,7 @@ def construct_variant(metadata: Metadata, value: Variant,
typed_value: Any) -> V
# value is missing
return None
-def primitive_to_variant(typed_value: Any): Variant:
+def primitive_to_variant(typed_value: Any) -> Variant:
if isinstance(typed_value, int):
return VariantInteger(typed_value)
elif isinstance(typed_value, str):
