This is an automated email from the ASF dual-hosted git repository.
jamesbognar pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/juneau.git
The following commit(s) were added to refs/heads/docs by this push:
new fe30245f80 feat(marshall): Parquet external-read interop
(multi-page/row-group, dictionary, Snappy) + binary-native logical types
fe30245f80 is described below
commit fe30245f808c3492ff27c71a47611bda25a1ca85
Author: James Bognar <[email protected]>
AuthorDate: Wed Jun 17 14:18:26 2026 -0400
feat(marshall): Parquet external-read interop (multi-page/row-group,
dictionary, Snappy) + binary-native logical types
---
pages/release-notes/10.0.0.md | 22 ++++++++++++++++++++++
pages/topics/02.34.06.ParquetBasics.md | 34 ++++++++++++++++++++++++++++++++++
static/ai/juneau-knowledge.jsonl | 2 +-
static/ai/manifest.json | 4 ++--
4 files changed, 59 insertions(+), 3 deletions(-)
diff --git a/pages/release-notes/10.0.0.md b/pages/release-notes/10.0.0.md
index c3dd7308d6..e7e72c4530 100644
--- a/pages/release-notes/10.0.0.md
+++ b/pages/release-notes/10.0.0.md
@@ -379,6 +379,28 @@ annotation.
See the new [Protobuf Binary Format Basics](/docs/topics/ProtobufBinaryBasics)
topic page for the full
scalar/container mapping tables, field-number rules, and round-trip guarantees.
+#### Parquet external-read interop + binary-native logical types
+
+Juneau 10.0 extends the Parquet codec to read spec-compliant files produced by
external writers
+(parquet-mr / Spark / Hive), not just files Juneau itself emits, and to
understand the binary-native
+logical types those tools use.
+
+- **External-read fidelity** — the reader now handles **multi-page** column
chunks, **multi-row-group**
+ files, and **dictionary-encoded** pages (`DICTIONARY_PAGE` +
`PLAIN_DICTIONARY`/`RLE_DICTIONARY`, the
+ parquet-mr default). Previously only the single-page / single-row-group /
PLAIN shape Juneau wrote was
+ readable.
+- **Snappy decompression** — a built-in, dependency-free Snappy **block**
decoder reads Snappy-compressed
+ pages (the default Spark/parquet-mr compression). Decode-only; write stays
`UNCOMPRESSED`/`GZIP`. LZ4 /
+ Zstd / Brotli / LZO remain hard errors (no silent fallback).
+- **Binary-native logical types** — INT96 timestamps, DECIMAL
(INT32/INT64-backed, scale-aware),
+ DATE, TIME(micros), and TIMESTAMP(micros) are always decoded on read. On
write they are opt-in via
+ `ParquetSerializer.create().nativeLogicalTypes(true)`; the default keeps the
existing portable
+ string / `TIMESTAMP_MILLIS` normalization, so default output is
byte-for-byte unchanged.
+- **Write knobs** — `pageSize` and `rowGroupSize` are now honored on write,
splitting output into
+ multiple pages / row groups.
+
+See the [Parquet Basics](/docs/topics/ParquetBasics) topic page for details.
+
### Bug Fixes
_TBD — to be filled in as development continues._
diff --git a/pages/topics/02.34.06.ParquetBasics.md
b/pages/topics/02.34.06.ParquetBasics.md
index e87d38c43b..e52354f3d5 100644
--- a/pages/topics/02.34.06.ParquetBasics.md
+++ b/pages/topics/02.34.06.ParquetBasics.md
@@ -10,6 +10,40 @@ The `ParquetSerializer` and `ParquetParser` classes
integrate with the standard
`Serializer` / `Parser` APIs and support the `@Parquet` and `@ParquetConfig`
annotations for
format-specific configuration.
+##### External-read interop
+
+The parser reads spec-compliant Parquet produced by external writers
(parquet-mr / Spark / Hive), not
+just files Juneau itself emits:
+
+- **Multi-page column chunks** — a column chunk split across several data
pages is read end-to-end.
+- **Multi-row-group files** — every row group is read and concatenated, not
just the first.
+- **Dictionary-encoded pages** — a `DICTIONARY_PAGE` followed by
`PLAIN_DICTIONARY` / `RLE_DICTIONARY`
+ data pages (the parquet-mr default) is decoded back through its dictionary.
+- **Snappy compression** — Snappy-compressed pages are decompressed by a
built-in, dependency-free
+ Snappy block decoder. Snappy is **decode-only**; on write use `UNCOMPRESSED`
or `GZIP`. LZ4, Zstd,
+ Brotli, and LZO remain unsupported and are reported as a hard error rather
than silent garbage.
+
+The `pageSize` and `rowGroupSize` builder settings control how Juneau splits
its own output into pages
+and row groups (useful for streaming large datasets and for interop testing).
+
+##### Binary-native logical types
+
+The parser always decodes the binary-native logical types external tools emit:
+
+- **INT96** — legacy 12-byte timestamps (Impala / Hive / older Spark) decode
to instants.
+- **DECIMAL** (INT32 / INT64 backed) decodes to `BigDecimal` with the schema's
scale.
+- **DATE**, **TIME (micros)**, **TIMESTAMP (micros)** decode to the matching
`java.time` types,
+ preserving sub-millisecond precision.
+
+On **write**, these native encodings are opt-in via
`ParquetSerializer.create().nativeLogicalTypes(true)`.
+By default Juneau keeps its portable string / `TIMESTAMP_MILLIS`
normalization, so default output is
+unchanged. Enable `nativeLogicalTypes()` for sub-millisecond-precise,
parquet-mr-interoperable output.
+
+```java
+// Emit binary-native DECIMAL / DATE / TIME / TIMESTAMP-micros columns.
+ParquetSerializer s =
ParquetSerializer.create().nativeLogicalTypes(true).build();
+```
+
:::note
Full documentation for the Parquet serializer/parser is coming soon. In the
meantime, refer to the
[Marshallers overview](/docs/topics/Marshallers) for general serializer/parser
configuration patterns,
diff --git a/static/ai/juneau-knowledge.jsonl b/static/ai/juneau-knowledge.jsonl
index 0e22b8db67..feee001bc1 100644
--- a/static/ai/juneau-knowledge.jsonl
+++ b/static/ai/juneau-knowledge.jsonl
@@ -205,7 +205,7 @@
{"description": "The <a
href=\"/site/apidocs/org/apache/juneau/marshall/oapi/OpenApiParser.html\"
target=\"_blank\">OpenApiParser</a> class is used to convert HTTP parts
back\ninto POJOs.\n\nThe class hierarchy for the builder of this parser
is:\n\n<tree>\n<node-0><java-abstract-class><a
href=\"/site/apidocs/org/apache/juneau/Context.Builder.html\"
target=\"_blank\">Context.Builder</a></java-abstract-class></node-0>\n<node-1><java-abstract-class><a
href=\"/site/apidocs/org/apache/juneau/ [...]
{"description": "Juneau supports converting arbitrary POJOs to and from TOML
(Tom's Obvious Minimal Language) using native serializers and parsers
implemented without any external dependencies.\nTOML is a configuration file
format that maps unambiguously to hash tables, widely used in projects like
Cargo (Rust) and Python's `pyproject.toml`.\nThe <a
href=\"/site/apidocs/org/apache/juneau/marshall/toml/TomlSerializer.html\"
target=\"_blank\">TomlSerializer</a> converts POJOs directly to T [...]
{"description": "Juneau supports converting arbitrary POJOs to and from the
**Protobuf Text Format** using native serializers and parsers implemented
without any external dependencies. This is the human-readable text format (not
the binary wire format) used for protobuf configuration files, debugging, and
interop with tools like `protoc --decode`. The <a
href=\"/site/apidocs/org/apache/juneau/marshall/prototext/PrototextSerializer.html\"
target=\"_blank\">PrototextSerializer</a> converts [...]
-{"description": "The `org.apache.juneau.marshall.parquet` package provides
Juneau serializer and parser implementations for the\n[Apache
Parquet](https://parquet.apache.org/docs/file-format/) columnar storage
format.\n\nThe `ParquetSerializer` and `ParquetParser` classes integrate with
the standard Juneau\n`Serializer` / `Parser` APIs and support the `@Parquet`
and `@ParquetConfig` annotations for\nformat-specific
configuration.\n\n:::note\nFull documentation for the Parquet serializer/p [...]
+{"description": "The `org.apache.juneau.marshall.parquet` package provides
Juneau serializer and parser implementations for the\n[Apache
Parquet](https://parquet.apache.org/docs/file-format/) columnar storage
format.\n\nThe `ParquetSerializer` and `ParquetParser` classes integrate with
the standard Juneau\n`Serializer` / `Parser` APIs and support the `@Parquet`
and `@ParquetConfig` annotations for\nformat-specific configuration.\n\n#####
External-read interop\n\nThe parser reads spec-com [...]
{"description": "Juneau supports converting arbitrary POJOs to and from the
Protocol Buffers **binary** wire format using native, bean-driven serializers
and parsers. No `.proto` schema file or code generation is required — the
field-number/type schema is derived directly from Juneau bean metadata.\n\nThe
<a
href=\"/site/apidocs/org/apache/juneau/marshall/protobuf/ProtobufSerializer.html\"
target=\"_blank\">ProtobufSerializer</a> converts POJOs directly to protobuf
binary bytes, an [...]
{"description": "Juneau supports converting arbitrary POJOs to and from YAML
using native serializers and parsers implemented\nwithout any external
dependencies.\nThe YAML serializer converts POJOs directly to YAML block-style
output using indentation-based structure.\nThe YAML parser creates POJOs
directly from YAML using an indentation-aware state machine.\n\nThe following
example shows YAML for a typical bean:\n\n##### Sample Beans\n\n```java\npublic
class Person {\n\n // Bean prop [...]
{"description": "The YAML data type produced depends on the Java object type
being serialized.\n\n- Primitives and primitive objects are converted to YAML
scalars.\n- Beans and `Maps` are converted to YAML block mappings.\n-
`Collections` and arrays are converted to YAML block sequences.\n- Anything
else is converted to YAML string scalars (with quoting if necessary).\n\n#####
Data type conversions:\n\n| POJO type | YAML type | Example | Serialized form
|\n|-----------|-----------|------ [...]
diff --git a/static/ai/manifest.json b/static/ai/manifest.json
index 15412d675b..dd7676ff67 100644
--- a/static/ai/manifest.json
+++ b/static/ai/manifest.json
@@ -3,8 +3,8 @@
"record_count": 464,
"schema_version": "1.0.0",
"source_commit": {
- "juneau": "2487121feb3ee201c6a27e9791481db0c13b64a1",
- "juneau_docs": "624834ba134ac4e8b1146e377970dfdf88375c29"
+ "juneau": "7fc45280c7657982db8eb1a3f1ee1c45ab6dea13",
+ "juneau_docs": "c3847ad70e26ba516beb8b3593d34c38703b03de"
},
"version": "10.0.0"
}
