Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 )
Change subject: IMPALA-11553: Add event specific metrics in the table metrics ...................................................................... Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG@22 PS2, Line 22: Tests: > last-synced-event-id on table metric makes sense to me. I think the lag of event processing at table level will be helpful. It shows how much a table is impacted when event processing is lagging behind. Users can check if some critical tables have stale metadata and trigger manual REFRESH/INVALIDATE on them. However, we don't have a light-weight HMS API to get the latest event time of a table. So measuring the lag at table level is expensive. "events-consuming-delay-ms" is a cheaper counter to reflect the lag. "events-skipped" at table level helps to find tables that don't need event processing. E.g. if events on a table are all skipped, we can disable HMS event sync on it. To get this, we also need the counter of total events at table level. BTW, db level metrics also help to find such dbs. I'm ok to add some of them in a separate JIRA if they require more work. http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG@23 PS8, Line 23: - Manually verified the metrics on catalogD UI page when running some Can we add a simple test in tests/webserver/test_web_pages.py to fetch the metrics of a table and verify the existence of these event metrics? http://gerrit.cloudera.org:8080/#/c/20473/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20473/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2529 PS8, Line 2529: return; If we return here, we should bump the counter of EVENTS_SKIPPED_METRIC. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Thu, 21 Dec 2023 07:58:29 +0000 Gerrit-HasComments: Yes
