Hi Iceberg Community,
During the last column update sync we discussed the column file metadata
representation at length (link <https://www.youtube.com/watch?v=exC7RCDYtk4>).
Thanks for all the valuable inputs and perspectives! Let me sum up what we
covered:
*Revisited content of the ColumnFile metadata struct*
I added a couple of extra fields to the column file level. The reason is
that the table and its properties might evolve and fields like
'file_format' are not enough on the base file level. Let's say the
'write.format.version' changes after writing the base file, then the
subsequent column files are in a different format and we had no support in
the proposed metadata to represent that. As a result, here is the extended
list of fields:
ColumnFile
format_version: required, int
field_ids: required, list<int>
location: required, string
file_format: required, string
file_size_in_bytes: required, long
key_metadata: optional, binary
split_offsets: optional, list<long>
*Is base file a column file?*
Since many fields are present both on column file level and on base file
level, it's a reasonable idea to make the base file a column file as it
came up earlier. We went back and forth on this sharing opinions and
eventually we *concluded to keep base file separate from column files*,
because the alternative design would introduce constraints and invariants
that aren't that intuitive and easy to get wrong. E.g.:
- we can't populate the 'field_ids' list for the base file (and probably
we don't want to)
- need a mechanism to identify the base file from the column files to
answer requests for location, file size in bytes, split offsets, etc. on
the manifest entry (TrackedFile)
- Can't drop the base file from the list
- Technically it's feasible to judge whether all the fields of the
base file are covered by column files
- However, we can't drop it because we'd need to elect a new base file
- The location and others would change => we use location as the
identifier of a file now => new location means new identifier
Adding all of the above together brought me to the point where we felt that
it's somewhat awkward to maintain such a design because we still want to
keep the "base file + column files" mental model. Revisiting this and
introducing the mental model of logical files is too much at this point.
Hope this helps!
Also, the column file metadata representation PR
<https://github.com/apache/iceberg/pull/16285> is out there. Any review is
appreciated! Let's get this going!
Best Regards,
Gabor
Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. jún. 18.,
Cs, 16:32):
> Hi Iceberg Community,
>
> I'd like to bump this thread and ask for some feedback. In the recent
> syncs (both V4 AMT and column updates) it came up again whether we want to
> a) treat the base file as a regular column file
> b) track base file info on the manifest entry level (TrackedFile),
> while keeping the ColumnFile list purely for column update files
>
> There were pros and cons, I think both ways are easy technically, it's the
> matter of personal preference, and how many invariants we want to add to
> the spec. However, in my opinion this conversation is orthogonal to the
> representation of a ColumnFile. I think the PR that introduces basic
> column file metadata structs
> <https://github.com/apache/iceberg/pull/16285> is ready for review. Any
> feedback is appreciated!
> - We finalized the field IDs
> - We concluded on keeping track of the sequence number for the latest
> column file within Tracking
> - The ColumnFile schema seems mature to me too. We can debate whether to
> include 'split_offsets' or not
>
> If this is merged, I can create a follow-up PR to show how it would look
> if we included base file info 'column_files'.
>
> Regards,
> Gabor
>
>
> Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. jún. 3.,
> Sze, 17:13):
>
>> Hi Iceberg Community,
>>
>> Let me sum up the relevant findings from the sync yesterday!
>>
>> - We want to keep the mental model of "Physical base file with
>> physical column files on top" and don't want to introduce a new concept of
>> "Logical file consisting of a list of physical column files"
>> - This makes most of the questions from the previous mail obsolete.
>> - Open question: Do we want to persist base file information (such as
>> location and file_size_in_bytes) within column files to save a handful of
>> field IDs in the metadata space?
>> - Implementation-wise we would have a list of column files where
>> not all the items are equal: we have to differentiate the base file
>> - We don't want to explicitly keep track of the field IDs covered
>> by the base file. Would be too much for thousands of fields
>> - We could say "The first column file in the list is the base
>> file" or "The column file with an empty field ID list is the base file"
>>
>> I gave this a thought and for me deviating the physical
>> representation (base file within column files) here from the mental model
>> (base file + column files), might cause unnecessary confusion and might not
>> be that intuitive. Plus we have one more requirement/invariant to keep to
>> create the distinction for the base file's column file from the regular
>> column files.
>>
>> I'm leaning towards keeping the TrackedFile level fields for the base
>> file and not moving them into the column file list.
>> I'm wondering WDYT?
>>
>> If we agree on this, here is the PR
>> <https://github.com/apache/iceberg/pull/16285> with the column file
>> metadata structs that we discussed so far.
>>
>> Best Regards,
>> Gabor
>>
>> Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. jún. 2.,
>> K, 13:57):
>>
>>> Hey Iceberg Community,
>>>
>>> Yesterday in the V4 AMT sync there were some discussions that are also
>>> relevant here. Let me summarize!
>>>
>>> 1) No new sequence number(s)
>>> One decision we made is to not introduce a new field for the sequence
>>> number of the latest column file, but to *bump the data sequence number
>>> of TrackedFile* whenever we add a new column file. With this what we
>>> can achieve:
>>>
>>> - Use TrackedFile.dataSequenceNumber() for
>>> last_updated_sequence_number for the null LUSN values in the latest
>>> update
>>> file
>>> - We can get rid of the equality deletes associated to the base file
>>> when adding a column update (current direction is to rewrite eq-deletes
>>> to
>>> DVs when adding column updates)
>>>
>>>
>>> With this, the metadata structures would look like the following:
>>> Tracking
>>> ...
>>> optional latest_column_file_snapshot_id long
>>> ColumnFile
>>> required field_ids list<int>
>>> required location string
>>> required file_size_in_bytes long
>>> 2) Treat base file as a regular column file
>>> This seemed a more exploratory idea: What if we include (some of the)
>>> base file details into TrackedFile.column_files? With this we could have
>>> the following advantages:
>>>
>>> - We won't need the "duplication" of some field in the schema, like
>>> TrackedFile.location vs ColumnFile.location, and the same for
>>> file_size_in_bytes (maybe also for split_offsets)
>>> - This design would be a more natural fit for column families (not
>>> being designed now but seems there is some interest for the future)
>>>
>>> While I think this approach makes sense, there are some areas that *we
>>> have to think through*:
>>>
>>> - *Making base file a first class citizen of column_files would
>>> require the following:*
>>> - Move location, file_size_in_bytes and potentially split_offsets
>>> from TrackedFile into ColumnFile
>>> - Populate fieldIDs list for the base file entry too
>>> - Include row_id into the field IDs too. Don't allow to change
>>> this field to be served by regular column files?
>>> - We tackle wide tables here: 1k cols => 4KB, 10k cols => 40KB
>>> in-memory size. On storage Parquet V2 probably compresses this
>>> efficiently.
>>> - Original projection algorithm: Find whatever projected
>>> fields are covered by column files, leave the rest for the base
>>> file
>>> - We could leave field ID list empty for base file, but then
>>> not all column files are treated equal
>>> - *Do we have to know which column file is the base file?*
>>> - Projections: If we keep a full column ID list, we don't have to
>>> make a distinction of base file from column updates for this
>>> - _file metadata column: We still want to populate this with base
>>> file's location
>>> - We can keep TrackedFile.location() that could delegate to
>>> the relevant column file (or keep an in-memory, not persisted
>>> field in
>>> TrackedFile for this)
>>> - Positional deletes: DVs no longer use file path to find the
>>> relevant data file. No need to find base file's path
>>> - Do we have other use cases where this could cause a mess?
>>> - *Mental model change: Physical file => logical file*
>>> - Instead of saying "TrackedFile is a physical file with
>>> additional physical column files" we switch to "TrackedFile is a
>>> logical
>>> file that is represented by physical column files"
>>> - In the current mental model, the ID of a file is its location
>>> - In a mental model of logical files, do we still want to use
>>> location as an ID?
>>> - Is it the base file's location? Somewhat hurts the mental
>>> model IMO
>>> - Introducing general "file ID"? Is this too invasive to
>>> change?
>>> - *Split offsets:*
>>> - Currently, planning for split reading uses the split offsets of
>>> the base file
>>> - Technically, we can move split_offsets to the ColumnFile-level
>>> and let the planner to decide on which column file to use for splits
>>>
>>> Let me know if I missed anything! Looking for your opinions!
>>> Gabor
>>>
>>>
>>> Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. máj.
>>> 28., Cs, 16:45):
>>>
>>>> Hey Iceberg Community,
>>>>
>>>> I discussed this with Amogh today, and apparently I was trying to solve
>>>> change detection on a lower level (per-column file) than optimal. According
>>>> to our conversation, here is an update of the proposed metadata structure
>>>> for column updates:
>>>> Tracking
>>>> ...
>>>> optional latest_column_file_sequence_number long
>>>> optional latest_column_file_snapshot_id long
>>>>
>>>> ColumnFile
>>>> required field_ids list<int>
>>>> required location string
>>>> required file_size_in_bytes long
>>>> Note, there is no need to track per column file status, sequence
>>>> numbers and snapshot IDs as we see currently.
>>>>
>>>> How change detection would work:
>>>> Whenever a column file is added to an entry (TrackedFile) the old entry
>>>> would be part of the new snapshot with REPLACED status and there would be a
>>>> new entry with MODIFIED status that contains the addition of the new column
>>>> file. With this, finding the matching REPLACED and MODIFIED entries by
>>>> location we could see the differences in the column list. Also
>>>> Tracking.latest_column_file_snapshot_id == current_snapshot_id() also helps
>>>> that we should take a look at column files to see the changes.
>>>>
>>>> @amogh, let me know if I missed something!
>>>>
>>>> Any feedback is appreciated!
>>>> Gabor
>>>>
>>>>
>>>> Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. máj.
>>>> 22., P, 16:48):
>>>>
>>>>> Thank you for taking a look and getting back with your thoughts,
>>>>> Steven!
>>>>>
>>>>> You're right, we shouldn't repurpose the semantics of existing fields.
>>>>> The idea to keep track of* latest_column_file_sequence_number *as you
>>>>> described in the doc makes sense to me. Going with this design, we won't
>>>>> need to keep sequence numbers on a column file level (unless we want to
>>>>> know the order they were added, but I don't see a use case for that).
>>>>>
>>>>> I see the following metadata structure changes:
>>>>> Tracking
>>>>> ...
>>>>> optional latest_column_file_sequence_number long
>>>>> ColumnFile
>>>>> required field_ids list<int>
>>>>> required location string
>>>>> required file_size_in_bytes long
>>>>> required column_file_tracking ColumnFileTracking
>>>>> ColumnFileTracking
>>>>> required status int
>>>>> optional snapshot_id long
>>>>> optional removed_field_ids list<int>
>>>>> ColumnFileTracking.status could have values {ADDED, EXISTING, DELETED,
>>>>> REPLACED} similarly to TrackedFile. With this we could have a clear idea
>>>>> exactly what changed wrt the column files simply taking a look at the
>>>>> column files metadata. See details in my first mail in this thread.
>>>>>
>>>>> Would be nice to hear further feedback on this!
>>>>> Best Regards,
>>>>> Gabor Kaszab
>>>>>
>>>>> Steven Wu <[email protected]> ezt írta (időpont: 2026. máj. 21.,
>>>>> Cs, 23:55):
>>>>>
>>>>>> Gabor, thanks for starting this discussion.
>>>>>>
>>>>>> I have been thinking about this problem independently since the
>>>>>> column update sync. Here is the detailed design document
>>>>>> <https://docs.google.com/document/d/160-FizR6zOASMb86NycfgCm7cZbh6HK7FLcUW8_Xp-0/edit?usp=sharing>
>>>>>> .
>>>>>>
>>>>>> Gabor, I read your section of How to support
>>>>>> _last_updated_sequence_number
>>>>>> <https://docs.google.com/document/d/1Bd7JVzgajA8-DozzeEE24mID_GLuz6iwj0g4TlcVJcs/edit?tab=t.xvm52pv4m7lq#heading=h.k2neg79ocgu>.
>>>>>> If I understand correctly, the proposal is to repurpose
>>>>>> file_sequence_number to capture the snapshot sequence number of the
>>>>>> latest
>>>>>> column file. I suggest we don't change the semantics of the existing
>>>>>> file_sequence_number. Instead, we can introduce a new
>>>>>> latest_column_file_sequence_number field in the tracking struct. My doc
>>>>>> described the reasoning.
>>>>>>
>>>>>> That is the only real difference as far as I can tell. Otherwise, I
>>>>>> think we had the same idea/design.
>>>>>>
>>>>>> On Wed, May 20, 2026 at 11:56 AM Gábor Kaszab <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey Iceberg Community,
>>>>>>>
>>>>>>> Anurag started a separate, focused discussion
>>>>>>> <https://lists.apache.org/thread/jbh1gbrso5h6l4by9rh9poy2cjjtb8j0>
>>>>>>> on the column update file representation, similarly, let me start
>>>>>>> another
>>>>>>> one for the metadata representation. Hopefully, we can make some
>>>>>>> iterations
>>>>>>> on this before the next sync.
>>>>>>>
>>>>>>> We covered this topic in the sync yesterday and agreed on some of
>>>>>>> the fields, but we left the "tracking" information part open. The
>>>>>>> *required* fields we agreed on so far:
>>>>>>>
>>>>>>> ColumnFile
>>>>>>> field_ids list<int>
>>>>>>> location string
>>>>>>> file_size_in_bytes long
>>>>>>>
>>>>>>> *Tracking information*
>>>>>>> Additionally to the above, we discussed the need of tracking
>>>>>>> information. These are the potential ones:
>>>>>>>
>>>>>>> *1) Sequence number*
>>>>>>>
>>>>>>> - Usage for _last_updated_sequence_number
>>>>>>>
>>>>>>> I did think about how to produce _last_updated_sequence_number and I
>>>>>>> think technically we don't need to store the sequence number on the
>>>>>>> update
>>>>>>> file level for that. I wrote up the steps here
>>>>>>> <https://docs.google.com/document/d/1Bd7JVzgajA8-DozzeEE24mID_GLuz6iwj0g4TlcVJcs/edit?pli=1&tab=t.xvm52pv4m7lq>,
>>>>>>> but in a nutshell: we could either fille that from the
>>>>>>> _last_updated_sequence_number written into the latest column file, or if
>>>>>>> null we can use the base file's file_sequence_number.
>>>>>>>
>>>>>>> - Usage for equality deletes
>>>>>>>
>>>>>>> As we agreed previously, we don't want to support update files
>>>>>>> together with equality deletes, so we won't need to store column file
>>>>>>> level
>>>>>>> sequence numbers for this either.
>>>>>>>
>>>>>>> - Usage for CDC, observability, etc.
>>>>>>>
>>>>>>> I'm wondering if there is any use case where we want to see the
>>>>>>> order of the column updates to see the sequence they were created. If
>>>>>>> this
>>>>>>> matters for CDC or reproducibility or anything else, then let's have a
>>>>>>> column file level sequence number too, if not, we can omit this.
>>>>>>>
>>>>>>> *2) Status*
>>>>>>> I think, similarly to TrackedFile, we need the following statuses
>>>>>>> here: EXISTING, ADDED, DELETED, REPLACED
>>>>>>> With these, when the base file's status is REPLACED, taking a look
>>>>>>> at the column_files we can know exactly what has changed wrt the column
>>>>>>> updates. Some examples to demonstrate:
>>>>>>>
>>>>>>> Step 1: Start with an existing base file:
>>>>>>> base file: {location: "file.parquet", seq_num: 1, file_seq_num: 1,
>>>>>>> status: EXISTING, column_files:[]}
>>>>>>>
>>>>>>> Step 2: Adding a column update for field IDs [1, 2]:
>>>>>>> base file: {location: "file.parquet", seq_num: 1, file_seq_num: *2*,
>>>>>>> status: *REPLACED*,
>>>>>>> column_files: [ *{field_ids: [1, 2], location:
>>>>>>> "update1.parquet", status: ADDED}* ]}
>>>>>>>
>>>>>>> Step 3: Adding an overlapping column update with field IDs [2, 3]
>>>>>>> ("de-duplicate" field IDs):
>>>>>>> base file: {location: "file.parquet", seq_num: 1, file_seq_num: *3*,
>>>>>>> status: REPLACED,
>>>>>>> column_files: [ {field_ids: *[1],* location:
>>>>>>> "update1.parquet", status: *REPLACED}, **{field_ids: [2, 3],
>>>>>>> location: "update2.parquet", status: ADDED}* ]}
>>>>>>>
>>>>>>> Step 4: Add another column update for field ID [1] to completely
>>>>>>> eliminate one previous update file from metadata
>>>>>>> base file: {location: "file.parquet", seq_num: 1, file_seq_num: *4*,
>>>>>>> status: REPLACED,
>>>>>>> column_files: [ {field_ids: [1]*,* location:
>>>>>>> "update1.parquet", status: *DELETED},* {field_ids: [2, 3],
>>>>>>> location: "update2.parquet", status: *EXISTING*}, *{field_ids: [1],
>>>>>>> location: "update3.parquet", status: ADDED}* ]}
>>>>>>>
>>>>>>> *Thoughts on REPLACED*
>>>>>>> In step 3, we marked the existing column file as REPLACED while
>>>>>>> reducing the field_ids list to de-duplicate them with the incoming
>>>>>>> update
>>>>>>> file's field_ids. With this, REPLACED indicates that field_ids content
>>>>>>> was
>>>>>>> reduced, however, we won't know exactly what field IDs were removed.
>>>>>>>
>>>>>>> - Alternative approach 1:
>>>>>>> We could use DELETED status leaving the field ID list intact, and
>>>>>>> then create a new ColumnFile with the reduced list. Step 3 would look
>>>>>>> like
>>>>>>> this:
>>>>>>>
>>>>>>> base file: {location: "file.parquet", seq_num: 1, file_seq_num: *3*,
>>>>>>> status: REPLACED,
>>>>>>> column_files: [ {field_ids: [1, 2]*,* location:
>>>>>>> "update1.parquet", status: *DELETED}, **{field_ids: [1], location:
>>>>>>> "update1.parquet", status: ADDED}, **{field_ids: [2, 3], location:
>>>>>>> "update2.parquet", status: ADDED}* ]}
>>>>>>>
>>>>>>> - Alternative approach 2:
>>>>>>> We can use REPLACED as originally, and also have a field in the
>>>>>>> tracking data to *keep track of the removed field IDs* (similarly
>>>>>>> to Tracking.DELETED_POSITIONS). Step 3 would look like this:
>>>>>>>
>>>>>>> base file: {location: "file.parquet", seq_num: 1, file_seq_num: *3*,
>>>>>>> status: REPLACED,
>>>>>>> column_files: [ {field_ids: *[1],* location:
>>>>>>> "update1.parquet", status: *REPLACED, removed_field_ids: [2]},
>>>>>>> **{field_ids:
>>>>>>> [2, 3], location: "update2.parquet", status: ADDED}* ]}
>>>>>>>
>>>>>>> - Preference:
>>>>>>> I think the REPLACED approach is cleaner, I'd prefer that. In case
>>>>>>> we want to track what IDs were removed, we could follow "alternative
>>>>>>> approach 2".
>>>>>>>
>>>>>>> - Additional, note:
>>>>>>> Re-writing the column file as REPLACED shouldn't alter the sequence
>>>>>>> number of the column file (if we decide to have one).
>>>>>>>
>>>>>>> *3) Snapshot ID*
>>>>>>> 'Tracking' has this, I think it could make sense for column files
>>>>>>> too.
>>>>>>>
>>>>>>> *4) First row ID*
>>>>>>> Row IDs should come from the base file's metadata IMO, we shouldn't
>>>>>>> store this for the update files.
>>>>>>>
>>>>>>> *Summary of all the potential tracking fields:*
>>>>>>>
>>>>>>> ColumnFileTracking
>>>>>>> required status int
>>>>>>> optional snapshot_id long
>>>>>>> optional sequence_number long
>>>>>>> optional removed_field_ids list<int>
>>>>>>>
>>>>>>> *Field IDs*
>>>>>>> The first free field ID within TrackedFile is 157. The last used one
>>>>>>> is DeletionVector.CARDINALITY
>>>>>>> <https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/DeletionVector.java#L42>
>>>>>>> with field ID 156.
>>>>>>> I'm working with Amogh to coordinate assigning the required field
>>>>>>> IDs here.
>>>>>>>
>>>>>>> Let me know if I miss anything here! Any feedback is appreciated!
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Gabor
>>>>>>>
>>>>>>>
