[
https://issues.apache.org/jira/browse/IMPALA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17983766#comment-17983766
]
ASF subversion and git services commented on IMPALA-14107:
----------------------------------------------------------
Commit 384a664f54f6fa32dc39d3368ad19fda225bff01 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=384a664f5 ]
IMPALA-14107: Increment table version in fireReloadEvent
test_reload_events_with_transient_partitions can get stuck when running
in local catalog mode. From the catalod.INFO, it looks like catalog keep
looking for non-existent partition id.
In local catalog mode, CatalogLookupStatus.PARTITION_NOT_FOUND response
from CatalogD will be followed by Coordinator invalidating table
metadata cache and table list of database. However, it does not
invalidate the partition list of requested table. This cause Coordinator
to re-request the same partition id that is not exist anymore in
CatalogD. This patch attempt to fix the issue from CatalogD side by
incrementing table version in fireReloadEventAndUpdateRefreshEventId if
HdfsTable.updatePartition() do detect any partition change. CatalogD
will send new topic update that reset Coordinator cache, and Coordinator
will retry the same partition request with an updated partition ID.
Reworded the InconsistentMetadataFetchException message to clarify the
lookup_status failure vs the req.object_desc.
This patch also tweak HdfsTable.getPartialInfo() to directly gather all
partitions when Coordinator does not request for specific partitions.
Testing:
Split test_reload_events_with_transient_partitions into two setup:
legacy catalog mode and local catalog mode. Confirm that both tests
pass.
Change-Id: I1bc91023d46a0b57916202d2b64e2cf07b687b74
Reviewed-on: http://gerrit.cloudera.org:8080/22956
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> test_reload_events_with_transient_partitions stuck in local catalog mode
> ------------------------------------------------------------------------
>
> Key: IMPALA-14107
> URL: https://issues.apache.org/jira/browse/IMPALA-14107
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
> Attachments: IMPALA-14107-debug.patch,
> catalogd-IMPALA-14107-debug.INFO, catalogd.INFO,
> impalad-IMPALA-14107-debug.INFO, impalad.INFO
>
>
> test_reload_events_with_transient_partitions can get stuck when running in
> local catalog mode. From the catalod.INFO, it looks like catalog keep looking
> for non-existent partition id.
> {noformat}
> W20250527 16:04:11.311885 1240999 HdfsTable.java:2302]
> 5b44f2b8f845ceb7:2febe44200000000] Missing partition ID: 59, Table:
> test_reload_events_with_transient_partitions_local_catalog_77dd977e.tbl
> W20250527 16:04:12.118695 1240999 HdfsTable.java:2302]
> 5b44f2b8f845ceb7:2febe44200000000] Missing partition ID: 59, Table:
> test_reload_events_with_transient_partitions_local_catalog_77dd977e.tbl
> W20250527 16:04:13.121974 1240999 HdfsTable.java:2302]
> 5b44f2b8f845ceb7:2febe44200000000] Missing partition ID: 59, Table:
> test_reload_events_with_transient_partitions_local_catalog_77dd977e.tbl
> W20250527 16:04:14.325909 1240999 HdfsTable.java:2302]
> 5b44f2b8f845ceb7:2febe44200000000] Missing partition ID: 59, Table:
> test_reload_events_with_transient_partitions_local_catalog_77dd977e.tbl
> ...
> {noformat}
> Attached are both impalad and catalogd log when the issue happen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
