[
https://issues.apache.org/jira/browse/IMPALA-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18013686#comment-18013686
]
Riza Suminto commented on IMPALA-14307:
---------------------------------------
Thanks for digging into this issue, [~stigahuang].
Yes, we have suspected that EP is interfering with the rename operation. But
has not quite figure it out on how to prevent it from happening.
Disabling EP for this testcase also does not seems right, since it just hide
the issue.
> invalidateTable could add table to old Db object while concurrent reset is
> running
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-14307
> URL: https://issues.apache.org/jira/browse/IMPALA-14307
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> Saw a failure in TestConcurrentDdls.test_ddls_with_invalidate_metadata:
> {code:python}
> custom_cluster/test_concurrent_ddls.py:50: in
> test_ddls_with_invalidate_metadata
> self._run_ddls_with_invalidation(unique_database, sync_ddl=False)
> custom_cluster/test_concurrent_ddls.py:179: in _run_ddls_with_invalidation
> worker[i].get(timeout=100)
> /data/jenkins/workspace/impala-asf-master-core-ubsan-arm/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/multiprocessing/pool.py:572:
> in get
> raise self._value
> E AssertionError: Query b148f272b5821772:ace4369900000000 failed:
> E AnalysisException: Table does not exist:
> test_ddls_with_invalidate_metadata_9525e717.test_6
> E
> E
> E assert <bound method type.is_acceptable_error of <class
> 'test_concurrent_ddls.TestConcurrentDdls'>>('Query
> b148f272b5821772:ace4369900000000 failed:\nAnalysisException: Table does not
> exist: test_ddls_with_invalidate_metadata_9525e717.test_6\n\n', False)
> E + where <bound method type.is_acceptable_error of <class
> 'test_concurrent_ddls.TestConcurrentDdls'>> =
> <test_concurrent_ddls.TestConcurrentDdls object at
> 0xffff1ff7d810>.is_acceptable_error{code}
> The test runs in legacy catalog mode. Coordinator logs show that it receives
> a deletion update on the table before analying the REFRESH statement on it:
> {noformat}
> I20250811 16:22:58.429529 3608823 ImpaladCatalog.java:227] Deleting:
> TABLE:test_ddls_with_invalidate_metadata_9525e717.test_6 version: 8523 size:
> 97
> ...
> I20250811 16:22:58.540377 3609121 Frontend.java:2398]
> b148f272b5821772:ace4369900000000] Analyzing query: refresh
> test_ddls_with_invalidate_metadata_9525e717.test_6 db: default
> ...
> I20250811 16:22:58.542122 3609121 jni-util.cc:321]
> b148f272b5821772:ace4369900000000]
> org.apache.impala.common.AnalysisException: Table does not exist:
> test_ddls_with_invalidate_metadata_9525e717.test_6
> at
> org.apache.impala.analysis.ResetMetadataStmt.analyze(ResetMetadataStmt.java:173)
> at
> org.apache.impala.analysis.AnalysisContext$AnalysisDriverImpl.analyze(AnalysisContext.java:576)
> at
> org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:496)
> at
> org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2938)
> at
> org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2505)
> at
> org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2374)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2063)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:176){noformat}
> Catalogd logs show that invalidateTable triggered by alterTableOrViewRename()
> is running concurrently with a catalog reset (i.e. global INVALIDATE
> METADATA):
> {noformat}
> I20250811 16:22:58.129398 3609170 JniUtil.java:167]
> f548d67b72570890:626780e200000000] execDdl request: ALTER_TABLE
> test_ddls_with_invalidate_metadata_9525e717.test_6_2 RENAME_TABLE issued by
> jenkins
> I20250811 16:22:58.144670 3608935 JniUtil.java:167]
> 1d4918bff853c0ec:c5646e4a00000000] resetMetadata request: INVALIDATE ALL
> issued by jenkins
> I20250811 16:22:58.173722 3608935 CatalogServiceCatalog.java:2339]
> 1d4918bff853c0ec:c5646e4a00000000] Loaded table list (12) and functions (0)
> for database: test_ddls_with_invalidate_metadata_9525e717. Fetch duration: 2
> ms.
> I20250811 16:22:58.177870 3609170 CatalogServiceCatalog.java:3283]
> f548d67b72570890:626780e200000000] Invalidating table metadata:
> test_ddls_with_invalidate_metadata_9525e717.test_6
> I20250811 16:22:58.178402 3608935 JniUtil.java:178]
> 1d4918bff853c0ec:c5646e4a00000000] Finished resetMetadata request: INVALIDATE
> ALL issued by jenkins. Time spent: 34ms
> I20250811 16:22:58.181602 3609170 Table.java:246]
> f548d67b72570890:626780e200000000] createEventId_ for table:
> test_ddls_with_invalidate_metadata_9525e717.test_6 set to: -1
> I20250811 16:22:58.181922 3609170 JniUtil.java:178]
> f548d67b72570890:626780e200000000] Finished execDdl request: ALTER_TABLE
> test_ddls_with_invalidate_metadata_9525e717.test_6_2 RENAME_TABLE issued by
> jenkins. Time spent: 52ms
> {noformat}
> Then the catalog update collector thread saw the table disappears and sent
> the deletion update:
> {noformat}
> I20250811 16:22:58.247798 3608258 JniUtil.java:167] getCatalogDelta request:
> Getting catalog delta from version 8517
> I20250811 16:22:58.250062 3608258 catalog-server.cc:1615] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_9525e717.test_6_2, version=8524,
> original size=106, compressed size=105
> I20250811 16:22:58.250487 3608258 catalog-server.cc:1615] Collected deletion:
> 1:TABLE:test_ddls_with_invalidate_metadata_9525e717.test_6, version=8523,
> original size=97, compressed size=95
> I20250811 16:22:58.250890 3608258 catalog-server.cc:1615] Collected update:
> 1:CATALOG_SERVICE_ID, version=8532, original size=60, compressed size=58
> I20250811 16:22:58.250959 3608258 JniUtil.java:178] Finished getCatalogDelta
> request: Getting catalog delta from version 8517. Time spent: 3ms{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
