Quanlong Huang created IMPALA-14447:
---------------------------------------
Summary: Metadata loading is not triggered in parallel in local
catalog mode
Key: IMPALA-14447
URL: https://issues.apache.org/jira/browse/IMPALA-14447
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Quanlong Huang
When a query accesses multiple tables that are unloaded, metadata loading of
them is triggered sequentially in local catalog mode. The stacktrace of
coordinator thread:
{noformat}
"Thread-20 [LoadWithCaching for table metadata for default.part_900k_parq1]"
#112 prio=5 os_prio=0 tid=0x000000000aa08000 nid=0x27f7 runnable
[0x00007fcb9afe5000]
java.lang.Thread.State: RUNNABLE
at
org.apache.impala.service.FeSupport.NativeGetPartialCatalogObject(Native Method)
at
org.apache.impala.service.FeSupport.GetPartialCatalogObject(FeSupport.java:472)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:463)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.access$100(CatalogdMetaProvider.java:209)
at
org.apache.impala.catalog.local.CatalogdMetaProvider$4.call(CatalogdMetaProvider.java:815)
at
org.apache.impala.catalog.local.CatalogdMetaProvider$4.call(CatalogdMetaProvider.java:807)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:601)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadTable(CatalogdMetaProvider.java:803)
at
org.apache.impala.catalog.local.LocalTable.loadTableMetadata(LocalTable.java:164)
at org.apache.impala.catalog.local.LocalTable.load(LocalTable.java:114)
at org.apache.impala.catalog.local.LocalDb.getTable(LocalDb.java:148)
at
org.apache.impala.analysis.StmtMetadataLoader.getMissingTables(StmtMetadataLoader.java:323)
at
org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:176)
at
org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:145)
at
org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2600)
at
org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2295)
at
org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2032)
at
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:171){noformat}
For an unloaded table, metadata loading is triggered by the first call on
LocalDb.getTable(). The metadata loading of the second unloaded table is
triggered after this is done.
This is a performance regression comparing to the legacy catalog mode where
metadata loading on all tables accessed by a query are triggered in parallel.
CC [~rizaon]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
