During data ingestion or upsert operations, documents are flushed to disk
in batches, creating disk components. In the columnar storage format, each
MegaPage, which represents a leaf logically, begins with a single page
metadata section called `PageZero`.
Currently, `PageZero` stores metadata for every column in the global
schema, even if a column is not present in the documents of the current
batch. This metadata includes a 4-byte offset and a 16-byte filter (min/max
values) per column. This approach leads to significant overhead, especially
for datasets with sparse or wide schemas. The 128KB default size limit of
`PageZero` imposes a practical maximum of approximately 6,500 columns,
which is further reduced in practice by the space required for primary keys.
The proposed enhancement introduces an efficient "Sparse PageZero writer".
This writer's design is to only store metadata for the subset of columns
that are actually present in the current batch of documents being flushed,
plus any others required for correct column assembly (e.g., in union types
or nested structures). This reduces metadata overhead, enabling support for
schemas with a larger number of sparse columns within the existing
`PageZero` size constraint.
Risks and trade-offs include a potential performance impact. The sparse
format requires PageReaders to perform a binary search to look up column
offsets and filters, rather than a direct index lookup, which introduces
CPU overhead. There is also a minor computational overhead from the column
estimation logic.
An alternative is the existing "Default" writer. The proposal includes an
"Adaptive" mode that dynamically evaluates both the Default and Sparse
writers for an incoming batch and selects the one that consumes the least
space.
A limitation of this proposal is that the `PageZero` size remains
constrained to one page, default 128KB, so the hard limit on the number of
columns in a single MegaPage remains ~6,500 by default. This is not removed
by this change.
This APE[
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey=ASTERIXDB&title=APE*22*3A*Sparse*column*metadata*storage__;KyUrKysr!!CzAuKJ42GuquVTTmVmPViYEvSg!LYuLE2dTTaSmQqZFBtmfGfDeQ6wK34-3HPl68Ji6x8BUouvD48jjKQcklfT4D9zoHj615BA1DUAR$
]
would introduce a new "Sparse PageZero writer" that writes metadata for
only the subset of columns present in a given batch.
The source code changes are summarized as follows:
* A new `PageZero Writer Mode` configuration option will be added with
three possible values:
* "*Default*": Always uses the current writer.
* "*Sparse*": Always uses the new sparse writer.
* "*Adaptive*": Dynamically compares the space usage of both writers
for an incoming batch and selects the one that results in a smaller
`PageZero`.
* The sparse layout will store `columnIndex`, `columnOffset`, and
`columnFilter` for each present column.
* Logic will be added to determine the minimum required set of columns
for a batch, accounting for schema evolution, unions, and nested structures
to ensure correct record assembly.
The change is controlled by a new configuration option. Existing disk
components created with the default writer will coexist with new
components. Since the global metadata is maintained at the index level and
used by the column assembler to reconstruct records, the system will be
able to read from components created with either writer, ensuring backward
compatibility.
The following areas will be tested to validate the change:
*Performance Testing*:
Once a prototype is available, performance testing should be done to
evaluate the trade-offs:
1. *Indirect Column Lookup*: Measure the CPU overhead introduced by using
binary search to locate column offsets and filters.
2. *Column Estimation Overhead*: Measure the computational cost of the
column estimation step.
*Functional Testing*:
1. *Default Writer Validation*: Run the existing test suite with documents
containing most or all fields to ensure the default writer's behavior is
unchanged.
2. *Sparse Writer Validation*: Design a new test suite with batches of
sparse documents (where each batch contains a subset of fields) to verify
that the `SparsePageZeroWriter` produces smaller disk components. Tests
will be constructed with a column set less than or equal to the 6,500
column limit.
3. *Correctness Checks*: For both writers, compare query results with row
format collections to ensure correctness, paying special attention to
missing fields, null values, and nested structures (arrays, objects,
unions).
