xiangfu0 opened a new pull request, #19665:
URL: https://github.com/apache/pinot/pull/19665
## What
`GET /tables/{table}/metadata?columns=*` intersects the column sets of all
segments a server holds. It took the first immutable segment's
`getAllColumns()` as the running intersection and `retainAll`'d every later
segment into it. That set is a **live view of the segment's own `Schema`**
(`SegmentMetadata.getAllColumns()` defaults to `Schema.getColumnNames()`, the
navigable key set of the field-spec map), so the endpoint deleted every column
a later segment lacked from the first segment's metadata.
After schema evolution that is the normal case: the newest segment loses its
newly added columns, `ImmutableSegmentImpl.getColumnNames()` /
`getPhysicalColumnNames()` read that schema live, and `SELECT *` on the segment
silently drops the columns until it is reloaded. The controller forwards the
caller's `columns` list verbatim, so the controller endpoint with `columns=*`
triggers it too.
The fix copies the first segment's column set before intersecting
(`TablesResource.java`, two lines).
## Tests
`TablesResourceTest#testTableMetadataWithAllColumnsLeavesSegmentColumnsIntact`
loads three segments with pairwise different column sets
(`column1,column2,column3` / `column1,column3` / `column1,column2`) into one
table, calls the endpoint with `columns=*`, and asserts that every segment's
`getColumnNames()`, `getSegmentMetadata().getAllColumns()` and
`getSchema().getColumnNames()` are what they were before the call. With three
segments the bug reproduces on every iteration order; without the fix the test
fails with `AssertionError: noColumn3` (that segment lost `column2`).
## Why now
Found while reviewing the DATA-3221 stack (#19475 → #19511). #19478 and
#19479 there change `getAllColumns()` to a view of the column metadata map,
which would make this endpoint delete `ColumnMetadata` from a serving segment
instead; #19481 carries this same copy. Landing the fix on master first closes
the window for the whole stack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
