Thanks for the clarification. So, essentially it means the column count in the current segment. I somehow thought "max column index" as "the last column offset in the current segment".
On Thu, Jun 26, 2025 at 11:37 PM Ritik Raj <ri...@apache.org> wrote: > The columnIndex is essentially an ID assigned to each column within an > LSMIndex, typically ranging from [0, N-1] for N columns. > Metadata for each column (such as offsets, min/max values, etc.) is stored > sequentially based on this index. > For example, if columnIndex = 0 has its offset at position X, then > columnIndex = 1's offset would be at X + Integer.BYTES. > > There are two primary reasons: > ## Not Penalizing Well-Modeled Schemas > > Many customers have a good data model with far fewer than 5000 columns. > Setting a Max Column Index (e.g., 5000) ensures these users are not > penalized with additional overhead from extra metadata segments. > It allows efficient operation for typical use cases without incurring > unnecessary costs. > > ## Simplified Segment Calculation for Column Metadata > > This boundary helps determine the segment in which a column's metadata > resides. > For example: > > Max columns in the first segment = 5000 > Buffer cache page size = 32KB > > If each column's metadata takes X bytes, then the number of > columns per page/segment is: > R = 32KB / X > To find which segment column I resides in: > > If I < 5000: it's in segment 0 > If I ≥ 5000: it's in > segment = ((I - 5000) / R) + 1 > > On 2025/06/26 23:22:01 Taewoo Kim wrote: > > +1 > > > > Q: What's the main point of having the "Max Column Index"? > > > > Best, > > Taewoo > > > > > > On Thu, Jun 26, 2025 at 3:33 PM Ian Maxon <ima...@uci.edu> wrote: > > > > > +1 > > > > > > On Thu, Jun 26, 2025 at 3:08 PM Mike Carey <dtab...@gmail.com> wrote: > > > > > > > > This looks excellent! > > > > > > > > +1 for adopting this extension to our storage system ASAP. > > > > > > > > On 6/26/25 11:53 AM, Ritik Raj wrote: > > > > > *Expanding PageZero to Support Unlimited Columns* > > > > > APE: > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*23*3A*Unlimited*Columns*Support__;KyUrKys!!CzAuKJ42GuquVTTmVmPViYEvSg!P1EHZSwq7hcpOlyHuy7R1F0lAkJK31elLGusrjb58xBVxuuNH4gxpVwKRuJSv9mByOtN5siVn5A6sQ$ > > > > > > > > > > In the columnar storage format, each MegaPage represents a logical > leaf > > > > > node and begins with `PageZero`, a metadata section that captures > > > essential > > > > > column metadata including column offsets and min/max filters. > > > Originally, > > > > > `PageZero` was constrained to reside in a single page (typically > > > 128KB), > > > > > with a fixed layout that stored information for **every column** > in the > > > > > global schema. > > > > > > > > > > Each column entry consumed 4 bytes for offset and 16 bytes for a > > > min/max > > > > > filter, leading to a **metadata footprint of 20 bytes per column**. > > > With > > > > > this layout, the **maximum number of columns supported was capped > at > > > > > ~6,000**, given space constraints and the need to reserve part of > > > > > `PageZero` for primary key metadata and structural headers. > > > > > > > > > > This limitation became problematic for datasets with **wide or > sparse > > > > > schemas**, where many columns may be missing in individual document > > > batches > > > > > but still occupy space in `PageZero`. The presence of unused > metadata > > > > > bloated the footprint and limited scalability. > > > > > > > > > > *Multi-Segment PageZero: Motivation and Layout* > > > > > > > > > > To overcome this limitation, we introduce **multi-segment support > in > > > > > PageZero**. Instead of storing all metadata in a single fixed > block, we > > > > > partition PageZero into multiple **segments**, with the **first > > > (zeroth) > > > > > segment storing primary key metadata and as many column entries as > it > > > can > > > > > fit**, and subsequent segments storing the remaining metadata. > > > > > > > > > > Each segment follows the same layout: column index → offset → min → > > > max, > > > > > stored in an interleaved manner. This structure ensures efficient > scan > > > and > > > > > lookup, while enabling us to scale to **arbitrarily many columns**, > > > bounded > > > > > only by MegaPage size. > > > > > > > > > > *Segment Layout:* > > > > > > > > > > ``` > > > > > [ Segment Header ] > > > > > ├─ Number of Columns > > > > > ├─ Max Column Index in Segment > > > > > [ Interleaved Metadata Entries ] > > > > > ├─ ColumnIndex₁, Offset₁, Min₁, Max₁ > > > > > ├─ ColumnIndex₂, Offset₂, Min₂, Max₂ > > > > > └─ ... > > > > > ``` > > > > > > > > > > A new `DefaultColumnMultiPageZeroWriter` class was introduced to > manage > > > > > this segmented layout. It delegates metadata writing to individual > > > segments > > > > > while maintaining headers at the top-level for navigation. > > > > > > > > > > *Adaptive Writer Selection* > > > > > > > > > > To avoid burdening all batches with this segmented structure, we > > > retain the > > > > > `DefaultColumnPageZeroWriter` for small or dense schemas. A new > > > **adaptive > > > > > selection mechanism** compares space usage of both writers for a > batch > > > and > > > > > picks the optimal one. > > > > > > > > > > The decision logic weighs: > > > > > - Space taken by Default Multi-segment writer (fixed layout for all > > > columns) > > > > > - Space taken by Sparse Multi-Segment writer (compact layout for > > > present > > > > > columns) > > > > > > > > > > This logic is encapsulated in `PageZeroWriterFlavorSelector`. > > > > > > > > > > *New Configuration Options:* > > > > > > > > > > Two new storage configuration parameters have been introduced: > > > > > > > > > > 1. **`STORAGE_MAX_COLUMNS_IN_ZEROTH_SEGMENT`** > (`INTEGER_BYTE_UNIT`, > > > > > default: `5000`) > > > > > Controls the maximum number of columns that can be stored in > the > > > zeroth > > > > > segment of `PageZero`. Remaining columns, if any, are offloaded to > > > > > additional segments. This helps balance lookup performance (fast > for > > > zeroth > > > > > segment) and scalability. This might change based on perf > experiments. > > > > > > > > > > 2. **`STORAGE_PAGE_ZERO_WRITER`** (`STRING`, default: `"default"`) > > > > > Controls the writer strategy used during flush. Accepted values > > > are: > > > > > - `"default"`: Always use the legacy writer. > > > > > - `"sparse"`: Always use the sparse writer (only present > columns). > > > > > - `"adaptive"`: Dynamically compare both and pick the writer > that > > > uses > > > > > less space. > > > > > > > > > > *Summary of Changes* > > > > > > > > > > - Interleaved layout per segment for columnIndex, offset, min, max. > > > > > - Logic to estimate the number of segments and assign columns to > > > segments. > > > > > - Writer is selected dynamically using > `PageZeroWriterFlavorSelector`. > > > > > > > > > > *Benefits* > > > > > > > > > > - Unlocks support for **tens of thousands of columns** per > MegaPage. > > > > > - Better space efficiency for sparse batches. > > > > > - Retains backward compatibility: Already ingested MegaLeafs can > also > > > be > > > > > read. > > > > > > > > > > This change is essential for evolving workloads that increasingly > rely > > > on > > > > > flexible schemas and sparse data layouts. > > > > > > > > > > >
