skywalker0618 opened a new pull request, #20165: URL: https://github.com/apache/hudi/pull/20165
### Describe the issue this Pull Request addresses > **Stacked on #20148.** Until that merges, this PR also shows its three commits. The change in this PR is the last commit only: [`1b4a585`](https://github.com/skywalker0618/hudi/commit/1b4a5853f3bf2d11608dffad42a17f32423487e5). I'll rebase once #20148 lands. #20148 reports whether each Hive sync run by `StreamWriteOperatorCoordinator` succeeded and how long it took. When a sync gets slow, nothing says where the time went: connecting to the metastore, reading the schema, finding the partitions to sync, or the metastore calls themselves. Nothing reports what a sync changed either. A sync that keeps succeeding while registering no partitions looks healthy. ### Summary and Changelog Flink jobs with Hive sync enabled now report the steps of each sync next to the metrics from #20148. * New `HiveSyncStats` in `hudi-hive-sync`, filled in by `HiveSyncTool` during `syncHoodieTable()` and read through a new public `getSyncStats()`. It holds: * time reading the table schema from storage; * time finding the partitions to sync, from the timeline or by listing storage; * the rest of the sync's time, which is the metastore calls; * the number of partitions added; * whether the storage schema differed from the metastore's and was pushed. * A MERGE_ON_READ table is registered as several Hive tables in one sync. The durations add up over them. The partition count and the schema flag describe one table, since each gets the same partitions and schema. * A step that did not run reports nothing, instead of 0 ms. For example, a table already synced to the latest commit reads no schema. * `StreamWriteOperatorCoordinator#doSyncHive()` times building the sync tool, and records the stats after the sync, including a failed one: * `hiveSyncInitDurationMs`: histogram * `hiveSyncSchemaReadDurationMs`: histogram * `hiveSyncPartitionScanDurationMs`: histogram * `hiveSyncMetastoreDurationMs`: histogram * `hiveSyncPartitionsAddedCount`: counter * `hiveSyncSchemaEvolvedCount`: counter Tests: `TestHiveSyncStats`; `TestHiveSyncTool` checks the stats of a first sync, an incremental sync with schema evolution, an already-synced sync, and a MERGE_ON_READ sync, in each sync mode; `TestFlinkHiveSyncMetrics` and `TestStreamWriteOperatorCoordinator` cover the mapping to metrics and the failure paths. ### Impact No config change. `HiveSyncTool` gets one new public method, `getSyncStats()`, and `HiveSyncStats` is new. Jobs with `hive_sync.enabled=true` report six more metrics under the write operator's coordinator scope. ### Risk Level low The sync logic is unchanged. `HiveSyncTool` adds timers around the schema read and the partition scan, and records the partitions added and the schema update after they happen. `hudi-hive-sync` tests pass (409); checkstyle passes. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
