Quanlong Huang created KUDU-3650:
------------------------------------
Summary: Skip fetching HMS notification events in unwanted types
Key: KUDU-3650
URL: https://issues.apache.org/jira/browse/KUDU-3650
Project: Kudu
Issue Type: Improvement
Components: server
Reporter: Quanlong Huang
Currently, when HMS integration is enabled, Kudu fetches events in all types
from HMS. However, only DROP_TABLE and ALTER_TABLE events are consumed. Some
events like ALTER_PARTITION events could be pretty large due to the incremental
stats of Impala is also sent inside the message body. Fetching such events is a
burden for both HMS and Kudu side.
Start from Hive 4.0.0, NotificationEventRequest has an optional list for
skipped event types:
{code}
struct NotificationEventRequest {
1: required i64 lastEvent,
2: optional i32 maxEvents,
3: optional list<string> eventTypeSkipList,
4: optional string catName,
5: optional string dbName,
6: optional list<string> tableNames
}{code}
https://github.com/apache/hive/blob/rel/release-4.0.0/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L1489
Kudu can use this to skip some common event types like ALTER_PARTITION. All
event types:
https://github.com/apache/hive/blob/183f8cb41d3dbed961ffd27999876468ff06690c/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/EventMessage.java#L33-L77
Once HIVE-28146 is done, the request can be simpler by just specifying the two
wanted event types.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
