[
https://issues.apache.org/jira/browse/IMPALA-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016706#comment-18016706
]
ASF subversion and git services commented on IMPALA-13593:
----------------------------------------------------------
Commit b67a9cecb33a7241ad99b33262173dd0287cfe3b in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b67a9cecb ]
IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events
from metastore
HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.
This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.
Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.
Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Reviewed-on: http://gerrit.cloudera.org:8080/22554
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Csaba Ringhofer <[email protected]>
> Enable event processor to consume ALTER_PARTITIONS events from metastore
> ------------------------------------------------------------------------
>
> Key: IMPALA-13593
> URL: https://issues.apache.org/jira/browse/IMPALA-13593
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Sai Hemanth Gantasala
> Assignee: Sai Hemanth Gantasala
> Priority: Major
>
> HIVE-27746 introduced emitting ALTER_PARTIONS events which contain all the
> partition names involved in the alter event. Before this metastore emits one
> ALTER_PARTITION event for a single partition, so for a table having huge
> partition count Impala would have to process a ton of alter_partition events.
> Instead, we can now consume single ALTER_PARTITIONS events and then refresh
> metadata for those partitions in the CatalogD's cache.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
