[
https://issues.apache.org/jira/browse/IMPALA-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17950551#comment-17950551
]
ASF subversion and git services commented on IMPALA-13718:
----------------------------------------------------------
Commit 04735598d660ef2063d3f3ddb78cfd08a7b6f6a9 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=04735598d ]
IMPALA-13718: Skip reloading Iceberg tables when metadata JSON file is the same
With this patch Impala skips reloading Iceberg tables when metadata
JSON file is the same, as this means that the table is essentially
unchanged.
This can help in situations when the event processor is lagging behind
and we have an Iceberg table that is updated frequently. Imagine the
case when Impala gets 100 events for an Iceberg table. In this case
after processing the first event, our internal representation of
the Iceberg table is already up-to-date, there is no need to do the
reload 100 times.
We cannot use the internal icebergApiTable_'s metadata location,
as the following statement might silently refresh the metadata
in 'current()':
icebergApiTable_.operations().current().metadataFileLocation()
To guarantee that we check against the actual loaded metadata
this patch introduces a new member to store the metadata location.
Testing
* added e2e tests for REFRESH, also for event processing
Change-Id: I16727000cb11d1c0591875a6542d428564dce664
Reviewed-on: http://gerrit.cloudera.org:8080/22432
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Noemi Pap-Takacs <[email protected]>
> Skip reloading Iceberg tables when metadata JSON file is the same
> -----------------------------------------------------------------
>
> Key: IMPALA-13718
> URL: https://issues.apache.org/jira/browse/IMPALA-13718
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Skip reloading Iceberg tables when metadata JSON file is the same.
> If the Iceberg table refers to the same metadata JSON this means we shouldn't
> reload it as it is unchanged.
> This can help in situations when the event processor is lagging behind.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
