yihua commented on PR #13414: URL: https://github.com/apache/hudi/pull/13414#issuecomment-2981238102
@Davis-Zhang-Onehouse at a high level, both the reader and writer paths should follow the new index layout version added in the index definition (i.e., the index layout version is `1` if not set, and the index layout version is `2`, which should be explicitly set for the new secondary index layout in this PR). The table version needs to be upgraded and the secondary index layout should be changed as part of the table upgrade and downgrade process. Here are more details on how it should work: Index definition and layout: - Table version 8: no index layout version; secondary index layout based on hard-coded logic - Table version 9: new index layout version introduced; secondary index layout version 2 (different from table version 8) Upgrade and downgrade: - Upgrade from 8 -> 9: add the index layout version to index definitions; remove secondary index if exists - Downgrade from 9 -> 8: remove the index layout version; remove secondary index if exists MDT reader and write compatibility: - The writer and reader take the index definition and get the index layout version. If not present (meaning table version <= 8), infer the table layout version as 1. Use the table layout version to determine how the record keys are mapped to file groups (i.e., secondary index has different mapping between layout 1 and 2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
