Noemi Pap-Takacs has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22432 )
Change subject: IMPALA-13718: Skip reloading Iceberg tables when metadata JSON file is the same ...................................................................... IMPALA-13718: Skip reloading Iceberg tables when metadata JSON file is the same With this patch Impala skips reloading Iceberg tables when metadata JSON file is the same, as this means that the table is essentially unchanged. This can help in situations when the event processor is lagging behind and we have an Iceberg table that is updated frequently. Imagine the case when Impala gets 100 events for an Iceberg table. In this case after processing the first event, our internal representation of the Iceberg table is already up-to-date, there is no need to do the reload 100 times. We cannot use the internal icebergApiTable_'s metadata location, as the following statement might silently refresh the metadata in 'current()': icebergApiTable_.operations().current().metadataFileLocation() To guarantee that we check against the actual loaded metadata this patch introduces a new member to store the metadata location. Testing * added e2e tests for REFRESH, also for event processing Change-Id: I16727000cb11d1c0591875a6542d428564dce664 Reviewed-on: http://gerrit.cloudera.org:8080/22432 Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-by: Noemi Pap-Takacs <npaptak...@cloudera.com> --- M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test M tests/metadata/test_event_processing.py 3 files changed, 108 insertions(+), 47 deletions(-) Approvals: Impala Public Jenkins: Verified Noemi Pap-Takacs: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/22432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I16727000cb11d1c0591875a6542d428564dce664 Gerrit-Change-Number: 22432 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Noemi Pap-Takacs <npaptak...@cloudera.com> Gerrit-Reviewer: Peter Rozsa <pro...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
