hudi-agent commented on code in PR #20244:
URL: https://github.com/apache/hudi/pull/20244#discussion_r4235266567
##########
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaBulkInsertHelper.java:
##########
@@ -111,17 +111,31 @@ public List<WriteStatus> bulkInsert(List<HoodieRecord<T>>
inputRecords,
final List<HoodieRecord<T>> repartitionedRecords =
(List<HoodieRecord<T>>) partitioner.repartitionRecords(dedupedRecords,
targetParallelism);
- String fileIdPrefix;
+ List<WriteStatus> writeStatuses = new ArrayList<>();
if (partitioner instanceof JavaHoodieMetadataBulkInsertPartitioner) {
- fileIdPrefix = partitioner.getFileIdPfx(0);
- } else {
- FileIdPrefixProvider fileIdPrefixProvider = (FileIdPrefixProvider)
ReflectionUtils.loadClass(
- config.getFileIdPrefixProviderClassName(),
- config.getProps());
- fileIdPrefix = fileIdPrefixProvider.createFilePrefix("");
+ // Metadata table records are tagged with the file group they hash to,
and the partitioner sorts them by file
+ // group. Each file group's run of records is written with its own file
id prefix and handle factory, so readers
+ // that look a key up in the file group it hashes to find it.
+ int runStart = 0;
+ int runIndex = 0;
+ for (int i = 1; i <= repartitionedRecords.size(); i++) {
Review Comment:
🤖 nit: would it make sense to pull this run-splitting loop into a small
private method (e.g. `bulkInsertMetadataFileGroups`)? It would keep
`bulkInsert` short and avoid recomputing the run's prefix via `getFileIdPrefix`
on every iteration.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
