yihua opened a new issue, #20065: URL: https://github.com/apache/hudi/issues/20065
Engine-agnostic code in `hudi-client-common` (indexes, table services, metadata table writers) runs task functions through `HoodieEngineContext` and `HoodieData`, but has no way to share read-only state with tasks. It captures the `HoodieTable`, the write config or the meta client in the task function instead, so every task deserializes its own copy and rebuilds the transient state behind it. Spark code works around this with `jsc.broadcast`, which engine-agnostic code cannot call. Proposal: add a broadcast to `HoodieEngineContext` (a Spark broadcast on Spark, an identity holder on engines that run tasks in the current JVM) so later changes can move heavy objects out of task closures. part of #20064 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
