Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22640 )
Change subject: IMPALA-13850 (part 2): Implement in-place reset for CatalogD ...................................................................... Patch Set 15: (2 comments) http://gerrit.cloudera.org:8080/#/c/22640/15/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/22640/15/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2408 PS15, Line 2408: .forEachOrdered(resettingDbs_::add); > I think we should remove non-existing dbs (i.e. dbs in oldDbNamesList but n non-existing dbs is removed by polling oldDbNames in L2421 and L2473. That should cover all dbs dropped by external systems? http://gerrit.cloudera.org:8080/#/c/22640/15/tests/custom_cluster/test_local_catalog.py File tests/custom_cluster/test_local_catalog.py: http://gerrit.cloudera.org:8080/#/c/22640/15/tests/custom_cluster/test_local_catalog.py@111 PS15, Line 111: # Make sure to wait for HMS events for the next query. : query_options["sync_hms_events_wait_time_s"] = 60 : query_options["sync_hms_events_strict_mode"] = True > Any stale metadata that causes global INVALIDATE METADATA to fail without t Test that use_local_catalog=True failed assertion in L115 (no failure happen). I thought this is because it takes awhile until second impalad receive db invalidation triggered by "invalidate metadata" initiated through the first impalad. But that does not seem to be true if I revert this change and run test_minimal_topic_updates_sync_ddl. I see these log lines from second impalad: I0410 21:19:57.935956 577857 CatalogdMetaProvider.java:1684] Invalidated objects in cache: [list of database names, TABLE_LIST for DB default] I0410 21:19:57.959349 577857 impala-server.cc:2280] Catalog topic update applied with version: 4351 new min catalog object version: 2167 ... I0410 21:19:58.109306 578419 Frontend.java:2369] 5248d4d70f6b72e9:311b0fbb00000000] Analyzing query: describe database test_minimal_topic_updates_sync_ddl_13c90bd2_new db: default Topic update version 4351 comes from the global INVALIDATE METADATA triggered through first impalad. I see in catalogd log too that database removal happens when collecting topic update version 4351. I0410 21:19:57.866428 578412 CatalogServiceCatalog.java:2424] 7e4a9171b72f8882:05cfb06700000000] Removing database: test_minimal_topic_updates_sync_ddl_13c90bd2_new ... I0410 21:19:57.909670 576940 JniUtil.java:178] Finished getCatalogDelta request: Getting catalog delta from version 2166. Time spent: 24ms I0410 21:19:57.924968 576949 catalog-server.cc:694] A catalog update with 2159 entries is assembled. Catalog version: 4351 Last sent catalog version: 2166 But somehow, invalidation at CatalogdMetaProvider.java:1684 does not fail "describe" query later on. Do you know what happen? -- To view, visit http://gerrit.cloudera.org:8080/22640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4ae2154612746b34484391c5950e74b61f85c9d Gerrit-Change-Number: 22640 Gerrit-PatchSet: 15 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Comment-Date: Fri, 11 Apr 2025 04:47:45 +0000 Gerrit-HasComments: Yes
