[
https://issues.apache.org/jira/browse/IMPALA-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith resolved IMPALA-9135.
-----------------------------------
Resolution: Fixed
The exception message for this error was changed in 2021 as part of
IMPALA-6671. The verification in test_concurrent_ddls was not, so it no longer
would have helped. We haven't seen any failures with this case since then, so I
think it's been fixed.
> DDLs with sync_ddl may fail with concurrent INVALIDATE METADATA
> ---------------------------------------------------------------
>
> Key: IMPALA-9135
> URL: https://issues.apache.org/jira/browse/IMPALA-9135
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Priority: Major
> Labels: concurrency
>
> This can be revealed by tests/custom_cluster/test_concurrent_ddls.py added in
> [https://gerrit.cloudera.org/c/14307]
> If running with INVALIDATE METADATA concurrently, the DDLs may run out of
> attemps in CatalogServiceCatalog.waitForSyncDdlVersion() to wait for the
> target update being sent, no matter how large we increase the maxNumAttempts.
> The error logs:
> {code:java}
> E1107 17:34:25.092439 7353 CatalogServiceCatalog.java:2626] Couldn't
> retrieve the covering topic version for catalog objects. Updated objects:
> [TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_9_part
> version: 349], deleted objects: []
> I1107 17:34:25.093451 7353 jni-util.cc:288]
> org.apache.impala.catalog.CatalogException: Couldn't retrieve the catalog
> topic version for the SYNC_DDL operation after 5 attempts.The operation has
> been successfully executed but its effects may have not been broadcast to all
> the coordinators.
> at
> org.apache.impala.catalog.CatalogServiceCatalog.waitForSyncDdlVersion(CatalogServiceCatalog.java:2630)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:414)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167)
> I1107 17:34:25.142006 6389 catalog-server.cc:337] A catalog update with 2
> entries is assembled. Catalog version: 356 Last sent catalog version: 355
> I1107 17:34:25.142168 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_15_part,
> version=357, original size=101, compressed size=98
> I1107 17:34:25.142215 6381 catalog-server.cc:641] Collected update:
> 1:CATALOG_SERVICE_ID, version=357, original size=49, compressed size=52
> I1107 17:34:25.142287 7356 CatalogServiceCatalog.java:2642] Operation using
> SYNC_DDL is waiting for catalog topic version: 357. Time to identify topic
> version (msec): 19
> I1107 17:34:25.192239 6389 catalog-server.cc:337] A catalog update with 2
> entries is assembled. Catalog version: 357 Last sent catalog version: 356
> I1107 17:34:25.192428 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_16_part,
> version=358, original size=101, compressed size=98
> I1107 17:34:25.192462 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_11_part,
> version=359, original size=101, compressed size=98
> I1107 17:34:25.192484 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_12_part,
> version=360, original size=101, compressed size=98
> I1107 17:34:25.192535 6381 catalog-server.cc:641] Collected update:
> 1:CATALOG_SERVICE_ID, version=360, original size=49, compressed size=52
> I1107 17:34:25.192613 7355 CatalogServiceCatalog.java:2642] Operation using
> SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic
> version (msec): 13
> I1107 17:34:25.192695 7351 CatalogServiceCatalog.java:2642] Operation using
> SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic
> version (msec): 45
> I1107 17:34:25.192734 7350 CatalogServiceCatalog.java:2642] Operation using
> SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic
> version (msec): 29
> I1107 17:34:25.222911 7353 status.cc:126] CatalogException: Couldn't
> retrieve the catalog topic version for the SYNC_DDL operation after 5
> attempts.The operation has been successfully executed but its effects may
> have not been broadcast to all the coordinators.
> @ 0x1c5ae50 impala::Status::Status()
> @ 0x24f7ad2 impala::JniUtil::GetJniExceptionMsg()
> @ 0x1c41987 impala::JniCall::Call<>()
> @ 0x1c3fec9 impala::JniUtil::CallJniMethod<>()
> @ 0x1c3e0e6 impala::Catalog::ExecDdl()
> @ 0x1c1ed17 CatalogServiceThriftIf::ExecDdl()
> @ 0x1cb3047 impala::CatalogServiceProcessor::process_ExecDdl()
> @ 0x1cb2d95 impala::CatalogServiceProcessor::dispatchCall()
> @ 0x1c08d65 apache::thrift::TDispatchProcessor::process()
> @ 0x20e8a0d
> apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0x20de040 impala::ThriftThread::RunRunnable()
> @ 0x20df766 boost::_mfi::mf2<>::operator()()
> @ 0x20df5fc boost::_bi::list3<>::operator()<>()
> @ 0x20df348 boost::_bi::bind_t<>::operator()()
> @ 0x20df25b
> boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x1ffb6e9 boost::function0<>::operator()()
> @ 0x2573dea impala::Thread::SuperviseThread()
> @ 0x257c16e boost::_bi::list5<>::operator()<>()
> @ 0x257c092 boost::_bi::bind_t<>::operator()()
> @ 0x257c055 boost::detail::thread_data<>::run()
> @ 0x3d61599 thread_proxy
> @ 0x7f1ce6ca46b9 start_thread
> @ 0x7f1ce343f41c clone
> E1107 17:34:25.222932 7353 catalog-server.cc:112] CatalogException: Couldn't
> retrieve the catalog topic version for the SYNC_DDL operation after 5
> attempts.The operation has been successfully executed but its effects may
> have not been broadcast to all the coordinators.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
