parisni opened a new issue, #8634: URL: https://github.com/apache/hudi/issues/8634
Hudi > 0.10 (OCC feature) When running multiple concurrent ingress process in a given table, which ads new partitions then it leads sometimes to missing partitions. This happens when some ingress are faster than others. For eg, say we have 2 ingress processes: ``` p1 starts at commit1 and will add part1 p2 starts at commit2 and will add part2 p2 finishes and sync the metastore to adds part2 and set metastore_last_commit = commit2 p1 finishes and consider the metastore as already synced because commit1 < metastore_last_commit ``` A way to mitigate this would be to force sync the current commit even if current_commit < metastore_last_commit We also should consider to not throw an error when a partition already exists (which is already the case for glue metastore) Thought ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
