Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23174 )
Change subject: IMPALA-14227: In HA failover, passive catalogd should apply pending HMS events before being active ...................................................................... Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/23174/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/23174/4//COMMIT_MSG@11 PS4, Line 11: However, it could still use : a stale metadata cache when some pending HMS events generated by the : previous active catalogd are not applied yet. > Out of scope for this patch, but I was thinking if it could be still possib That's a good idea. We can add such a "loaded but might be stale" state and let catalogd don't propagate metadata of them so they are not used in queries. When the table is first used in queries, reload its metadata and turn it into the loaded state. EventProcessor can just mark tables in this state instead of aggressively reloading metadata for each event. But I'm not sure how much time we can save in reusing file descriptors. It might only help when the storage is HDFS that we can save the calls of fetching block locations. For storages like S3 that don't have block locations, reload might have the same performance as the initial loading since both time is dominated in file listing. Anyway, I do think that we are loading metadata too aggresively and need improvement in this area. Feel free to file a JIRA for more discussions. -- To view, visit http://gerrit.cloudera.org:8080/23174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4fcb0e27c14197f79625749949b47c033a5f31 Gerrit-Change-Number: 23174 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Comment-Date: Thu, 17 Jul 2025 10:15:56 +0000 Gerrit-HasComments: Yes
