vinishjail97 opened a new pull request, #20188: URL: https://github.com/apache/hudi/pull/20188
### Describe the issue this Pull Request addresses The record level index and the secondary index group a commit's write stats by `HoodieWriteStat::getFileId` alone. A file group is identified by its partition and file id. For files written outside Hudi, which #19869 registers by their file name, the file id is only unique within a partition. Spark names a file `part-<task>-<job uuid>`, so one task of a `partitionBy` write leaves the same name in every partition it writes. The secondary index then sees several base files under one key and fails with `Only one new parquet file expected per file group per commit`. TPC-DS `store_returns` written by Spark has 2,004 files with 200 distinct names. ### Summary and Changelog - `SecondaryIndexRecordGenerationUtils` groups write stats by `HoodieFileGroupId` and takes the partition from the key. The replaced file groups it skips are matched by `HoodieFileGroupId` as well. - `BaseRecordIndexer` groups write stats by `HoodieFileGroupId`. - `TestExternalFileRecordAndSecondaryIndex#testFilesWithTheSameNameInDifferentPartitions` registers files with the same name in two partitions in one commit and checks both indexes. It fails without the change. ### Impact None for files written by Hudi, whose file ids are unique across the table. ### Risk Level low ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
