Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24049
Change subject: POC: LocalIcebergTable loads files in coordinator ...................................................................... POC: LocalIcebergTable loads files in coordinator If load_iceberg_files_in_coordinator=true, load files on the coordintor for Iceberg tables instead of getting from the catalog. This is inefficient at the moment as catalogd still loads the files but not really uses them. The long term goal is to load only minimal info for the table on catalogd side. Instead of caching TPartialTableInfo this solution caches IcebergFileContentStore + hostIndex pair. This would be suitable for REST catalog too if the key contained snapshot ID instead of catalog version. Pros: - File descs are not transferred in getPartialCatalogObject RPC - Size of cache objects seem to decrease: ~543->~431 MB for 1M file table - Plans look faster due to skipping construction of IcebergFileContentStore (DESCRIBE 1M file table 0.5s->1ms) Cons: - Initial table loading is ~doubled as both the catalog and the coordinator need to load the files. - No incremental loading, any catalog version change leads the reloaiding the files from scratch on coordinator side. Change-Id: I6732af76a2e040fa57e39260302951466037b934 --- M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java 13 files changed, 161 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/1 -- To view, visit http://gerrit.cloudera.org:8080/24049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934 Gerrit-Change-Number: 24049 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer <[email protected]>
