wgtmac commented on code in PR #186: URL: https://github.com/apache/parquet-site/pull/186#discussion_r3436457967
########## content/en/docs/File Format/versions.md: ########## @@ -0,0 +1,260 @@ +--- +title: "Parquet format versions" +linkTitle: "Features and Versions" +weight: 9 +--- + +This page describes how features are added to the [Parquet format +specification](https://github.com/apache/parquet-format) and how they affect +reader and writer compatibility. See the +[Implementation status](../implementationstatus/) page for which implementations +(arrow, parquet-java, arrow-rs, etc.) support each feature. + +*Note*: If you find out-of-date information, please open an issue or pull request. + +## Feature compatibility + +The Parquet format spec [classifies changes] by their effect on reader and +writer compatibility. Changes differ in their *forward* compatibility — whether +an older reader can read files that use a newer feature. + +**Forward compatible** features remain **readable by older readers**, with a +possibly degraded experience: some metadata may be missing or performance may +suffer, but the reader does not fail. Examples: + +* **Bloom filters**: a reader that ignores them skips the pruning metadata but + still reads the data correctly. +* **Logical type annotations** such as `VARIANT`: an older reader reads the + underlying physical column (e.g. `BYTE_ARRAY`) as raw bytes without applying + the logical type. + +**Forward incompatible** features make the data **unreadable** to older software. +Examples: + +* **New encodings** (e.g. the `DELTA_*` encodings, `BYTE_STREAM_SPLIT`, + `RLE_DICTIONARY`): a reader that does not implement them cannot decode the + column values. +* **Data Page V2 headers**: a reader that only understands `DataPageHeader` + cannot parse `DataPageHeaderV2` pages. + +[classifies changes]: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement + +## `FileMetadata` version field + +Each Parquet file has a `version` field in the [`thrift FileMetadata`]. This +field has historically been used inconsistently: writers populate `1` or `2` +without a consistent relationship to the features actually used. See the +[note in parquet.thrift] and [this discussion][closing-out-2.0] for details. + +## `parquet-format` release versions + +The Thrift definition is released independently of implementations such as +parquet-java or arrow-rs, following the Apache release process. This +release version is not recorded in the FileMetaData. Note that +release numbering **DOES NOT FOLLOW** [semantic versioning]: +minor releases (e.g. `2.10.0` to `2.11.0`) sometimes contain forward +incompatible features. + +## Adding new features + +New features are added by discussion and voting on the [parquet dev mailing list] +(full process in the [contributing guide]). Once approved, a feature is added to the spec and ships in +the next parquet-format release. + +[parquet dev mailing list]: https://lists.apache.org/[email protected] +[semantic versioning]: https://semver.org/ +[`thrift FileMetadata`]: https://github.com/apache/parquet-format/blob/c42c2cb4ecfccb38153375e24b702a82fd763cc0/src/main/thrift/parquet.thrift#L1365-L1373 +[contributing guide]: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#additionschanges-to-the-format +[note in parquet.thrift]: https://github.com/apache/parquet-format/blob/74001e41f5c5a1856b29be115f9c992cab16a4bf/src/main/thrift/parquet.thrift#L1368-L1373 +[closing-out-2.0]: https://lists.apache.org/thread/0bdyyb7qobrxx94x8v7t5z7g2ksnpyr2 + +## Forward incompatible features by version + +Forward incompatible features and the format version each became available in: + +| Feature | Released in | Source | Notes | +| ------------------------------------------ | ----------------------------- | --- | ------------------------- | +| [BOOLEAN] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT32] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT64] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT96 (deprecated)] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [FLOAT] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [DOUBLE] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BYTE_ARRAY] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [FIXED_LEN_BYTE_ARRAY] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [Data Page V1] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [Data Page V2] | [2.0.0] | [1.0.0..2.0.0] | | +| [PLAIN] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [PLAIN_DICTIONARY] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [RLE] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BIT_PACKED (deprecated)] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [RLE_DICTIONARY] | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_BINARY_PACKED] | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_LENGTH_BYTE_ARRAY] | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_BYTE_ARRAY] | [2.0.0] | [1.0.0..2.0.0] | | +| [BYTE_STREAM_SPLIT] | [2.8.0] | [2.7.0..2.8.0] | [Approved 2019-12-03] | +| [BYTE_STREAM_SPLIT<br/>(Additional Types)] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-03-18] | +| [UNCOMPRESSED] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [SNAPPY] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [GZIP] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [LZO] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BROTLI] | [2.4.0] | [2.3.1..2.4.0] | | +| [LZ4 (deprecated)] | [2.4.0] | [2.3.1..2.4.0] | | +| [LZ4_RAW] | [2.9.0] | [2.8.0..2.9.0] | | +| [ZSTD] | [2.4.0] | [2.3.1..2.4.0] | | +| [Modular encryption] | [2.7.0] | [2.6.0..2.7.0] | [Approved 2019-01-16] | + + +> **Note:** Files with an [encrypted footer] use different magic bytes (`PARE` +> instead of `PAR1`), making it clear to readers they must support modular +> encryption to read the file; [plaintext footer] files use `PAR1` so legacy +> readers can still read their unencrypted columns. + +## Forward compatible additions Review Comment: Do we want to include the new `IEEE754TotalOrder` column order just released in 2.13.0? I think it is a compatible change if readers just regard the new column order as an unknown order and ignore its min/max stats. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
