Hi all, I found that parquet-format doesn’t allow VARIANT field value to be omitted, not like Arrow.
Arrow’s documents say "At least one (or both) of the following… A field named value… A field named typed_value…”. https://arrow.apache.org/docs/format/CanonicalExtensions.html#parquet-variant But Parquet’s documents just say "The `value` field must be annotated as `required` for unshredded Variant values, or `optional` if parts of the value are [shredded](VariantShredding.md) as typed Parquet columns.”. https://github.com/apache/parquet-format/blob/master/VariantEncoding.md But there are some cases in parquet-testing don’t follow this behavior. For examples, parquet-testing/shredded_variant/case-131.parquet: required group field_id=-1 table { required int32 field_id=1 id; required group field_id=2 var (Variant(1)) { required binary field_id=-1 metadata; optional int32 field_id=-1 typed_value; } } parquet-testing/shredded_variant/case-132.parquet: required group field_id=-1 table { required int32 field_id=1 id; optional group field_id=2 var (Variant(1)) { required binary field_id=-1 metadata; optional binary field_id=-1 value; optional group field_id=-1 typed_value { required group field_id=-1 a { optional int32 field_id=-1 typed_value; } required group field_id=-1 b { optional binary field_id=-1 typed_value (String); } } } } I created an issue which records more cases. https://github.com/apache/parquet-format/issues/590 Should we modify parquet-format’s description and allow value field to be omitted? And I found arrow-rs has already allowed value field to be omitted. https://github.com/apache/arrow-rs/blob/7616e10f12894232eacb807993e55427a067d061/parquet-variant-compute/src/variant_array.rs#L858 I have submitted a PR to adjust the description. https://github.com/apache/parquet-format/pull/591 Best, Zehua Zou
