Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22634 )
Change subject: IMPALA-13850: Wait until CatalogD active before resetting metadata ...................................................................... Patch Set 8: (5 comments) http://gerrit.cloudera.org:8080/#/c/22634/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/22634/5//COMMIT_MSG@23 PS5, Line 23: TriggerResetMetadata that monitor and trigger this initial reset > It appears to be the case. > https://github.com/apache/impala/commit/1d63348b933b266f63d76b06eecbdf636cb45770 We don't track the db/table numbers (in local-catalog mode coordinators) after https://github.com/apache/impala/commit/6af8154ec So maybe that case is not an issue now. I'll check the other patch. http://gerrit.cloudera.org:8080/#/c/22634/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/22634/7//COMMIT_MSG@20 PS7, Line 20: The later : prerequisite is to ensure that all coordinators are not blocked waiting : for full topic update. > I meant use_local_catalog=true. I think local catalog mode is fine since queries will be blocked in fetching the db list from catalogd. This will then get blocked in CatalogServiceThriftIf.AcceptRequest(): https://github.com/apache/impala/blob/e73e2d40da115ed3804ffaecc0850c853b0e6330/be/src/catalog/catalog-server.cc#L502 For the legacy catalog mode, queries will failed by "Database does not exist" errors until the update of db/table names arrive. We might need to change the IsReady() check here: https://github.com/apache/impala/blob/b410a90b4707be2e08dcb6c02b9473aa9ade8d96/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L667 Note that this will add into the startup time of coordinators and they are "unhealthy" before that. Hopefully we don't use the legacy catalog mode in k8s. So this might be ok. BTW, maybe it's time to deprecate the legacy catalog mode. http://gerrit.cloudera.org:8080/#/c/22634/8/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/22634/8/be/src/catalog/catalog-server.cc@766 PS8, Line 766: // Run ResetMetadata without holding 'catalog_lock_' so that it can interleave with : // gathering thread. I think they will run sequentially due to the versionLock_ in CatalogServiceCatalog. http://gerrit.cloudera.org:8080/#/c/22634/8/be/src/catalog/catalog-server.cc@768 PS8, Line 768: 'invalidate metadata' metadata operation This sounds like a user triggered command. Let's use "catalog invalidation"? https://github.com/apache/impala/blob/e73e2d40da115ed3804ffaecc0850c853b0e6330/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L279 http://gerrit.cloudera.org:8080/#/c/22634/8/tests/common/impala_cluster.py File tests/common/impala_cluster.py: http://gerrit.cloudera.org:8080/#/c/22634/8/tests/common/impala_cluster.py@249 PS8, Line 249: if impalad._get_arg_value("use_local_catalog", default="false") != "false": I used to writing "--use_local_catalog" instead of "--use_local_catalog=true" for my laziness. E.g. bin/start-impala-cluster.py --catalogd_args=--catalog_topic_mode=minimal --impalad_args=--use_local_catalog It doesn't work after this. But it's ok.. -- To view, visit http://gerrit.cloudera.org:8080/22634 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cc66dcccedb306ff11893f2916ee5ee6a3efc1 Gerrit-Change-Number: 22634 Gerrit-PatchSet: 8 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Comment-Date: Wed, 26 Mar 2025 03:30:13 +0000 Gerrit-HasComments: Yes
