[
https://issues.apache.org/jira/browse/IMPALA-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venugopal Reddy K updated IMPALA-13801:
---------------------------------------
Description:
1. Need to redefine the lag in the hierarchical processing mode. Hierarchical
mode currently have a mechanism to check the total number of outstanding events
at the moment so that memory usage pressure on catalogd can be controlled when
event processing becomes slow i.e., iff outstanding event count exceeds
max_outstanding_events_on_executors configured value.
2. Currently, with enable_hierarchical_event_processing as true(introduced with
https://issues.apache.org/jira/browse/IMPALA-12709), lastSyncedEventId_ and
lastSyncedEventTimeSecs_ are updated upon event dispatch to
EventExecutorService for processing on respective DbEventExecutor and/or
TableEventExecutor. So lastSyncedEventId_ and lastSyncedEventTimeSecs_ doesn't
actually mean events are processed.
was:
1. Need to redefine the lag in the hierarchical processing mode. Hierarchical
mode currently have a mechanism to check the total number of outstanding events
at the moment so that memory usage pressure on catalogd can be controlled when
event processing becomes slow i.e., iff outstanding event count exceeds
max_outstanding_events_on_executors configured value.
2. Currently, with enable_hierarchical_event_processing as true,
lastSyncedEventId_ and lastSyncedEventTimeSecs_ are updated upon event dispatch
to EventExecutorService for processing on respective DbEventExecutor and/or
TableEventExecutor. So lastSyncedEventId_ and lastSyncedEventTimeSecs_ doesn't
actually mean events are processed.
> Enhance Lag mechanism in hierarchical metastore event processing
> ----------------------------------------------------------------
>
> Key: IMPALA-13801
> URL: https://issues.apache.org/jira/browse/IMPALA-13801
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Venugopal Reddy K
> Priority: Major
> Labels: catalog-2024
>
> 1. Need to redefine the lag in the hierarchical processing mode. Hierarchical
> mode currently have a mechanism to check the total number of outstanding
> events at the moment so that memory usage pressure on catalogd can be
> controlled when event processing becomes slow i.e., iff outstanding event
> count exceeds max_outstanding_events_on_executors configured value.
> 2. Currently, with enable_hierarchical_event_processing as true(introduced
> with https://issues.apache.org/jira/browse/IMPALA-12709), lastSyncedEventId_
> and lastSyncedEventTimeSecs_ are updated upon event dispatch to
> EventExecutorService for processing on respective DbEventExecutor and/or
> TableEventExecutor. So lastSyncedEventId_ and lastSyncedEventTimeSecs_
> doesn't actually mean events are processed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
