Hello Peter Rozsa, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22432
to look at the new patch set (#5).
Change subject: IMPALA-13718: Skip reloading Iceberg tables when metadata JSON
file is the same
......................................................................
IMPALA-13718: Skip reloading Iceberg tables when metadata JSON file is the same
With this patch Impala skips reloading Iceberg tables when metadata
JSON file is the same, as this means that the table is essentially
unchanged.
This can help in situations when the event processor is lagging behind
and we have an Iceberg table that is updated frequently. Imagine the
case when Impala gets 100 events for an Iceberg table. In this case
after processing the first event, our internal representation of
the Iceberg table is already up-to-date, there is no need to do the
reload 100 times.
We cannot use the internal icebergApiTable_'s metadata location,
as the following statement might silently refresh the metadata
in 'current()':
icebergApiTable_.operations().current().metadataFileLocation()
To guarantee that we check against the actual loaded metadata
this patch introduces a new member to store the metadata location.
Testing
* added e2e tests for REFRESH, also for event processing
Change-Id: I16727000cb11d1c0591875a6542d428564dce664
---
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
M tests/metadata/test_event_processing.py
3 files changed, 108 insertions(+), 47 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/22432/5
--
To view, visit http://gerrit.cloudera.org:8080/22432
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16727000cb11d1c0591875a6542d428564dce664
Gerrit-Change-Number: 22432
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <npaptak...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pro...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
