[
https://issues.apache.org/jira/browse/IMPALA-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18043102#comment-18043102
]
ASF subversion and git services commented on IMPALA-13801:
----------------------------------------------------------
Commit bff814f0790eedc2ab8040c8ff2b9ad0218c9c69 in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=bff814f07 ]
IMPALA-14562: Enable Hierarchical event processing by default
IMPALA-12709 Added support for hierarchical metastore event processing.
This commit enables hierarchical event processing by default.
hms_event_polling_interval_s can now be set to decimal value (eg: 0.5)
to support millisecond precision interval. Along with that others
configs can be fine tuned, such as:
num_db_event_executors: To set the number of database level event
executors.
num_table_event_executors_per_db_event_executor: To set the number of
table level event executors within a database event executor.
min_event_processor_idle_ms: To set the minimum time to retain idle db
processors and table processors.
max_outstanding_events_on_executors: To set the limit of maximum
outstanding events to process on event executors.
Testing:
- All the testing required to enable this flag is done in IMPALA-12709
and IMPALA-13801.
Change-Id: Ie9a28f863ef17456817e0a335215450e514b1f5b
Reviewed-on: http://gerrit.cloudera.org:8080/23687
Reviewed-by: <[email protected]>
Reviewed-by: Quanlong Huang <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Enhance Lag mechanism in hierarchical metastore event processing
> ----------------------------------------------------------------
>
> Key: IMPALA-13801
> URL: https://issues.apache.org/jira/browse/IMPALA-13801
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Venugopal Reddy K
> Assignee: Venugopal Reddy K
> Priority: Critical
> Labels: catalog-2024
>
> 1. Need to redefine the lag in the hierarchical processing mode. Hierarchical
> mode currently have a mechanism to check the total number of outstanding
> events at the moment so that memory usage pressure on catalogd can be
> controlled when event processing becomes slow i.e., iff outstanding event
> count exceeds max_outstanding_events_on_executors configured value.
> 2. Currently, with enable_hierarchical_event_processing as true(introduced
> with https://issues.apache.org/jira/browse/IMPALA-12709), lastSyncedEventId_
> and lastSyncedEventTimeSecs_ are updated upon event dispatch to
> EventExecutorService for processing on respective DbEventExecutor and/or
> TableEventExecutor. So lastSyncedEventId_ and lastSyncedEventTimeSecs_
> doesn't actually mean events are processed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
