Hi, which version of hms are you using now? I have checked the master branch and beta-1 branch source code, but I can't find the place where this method get_materialized_views_for_rewriting is called by mistake.
Thanks, Butao Zhang ---- Replied Message ---- | From | Eugene Miretsky<eug...@badal.io> | | Date | 11/16/2023 02:21 | | To | <user@hive.apache.org> | | Subject | Slow Hive query with a lot of 'get_materialized_views_for_rewriting' | Hey! We have a catalog with fairly a lot of databases and tables. Where we do a simple query (select * from table limit 5;) on an ideal cluster, it takes around 20seconds, sometimes longer (usually first run takes 40s+) Looking at the hive-metastore logs during most of the query time the logs show "metastore.Hivemetastore: 13: get_materialized_views_for_rewriting: db = <Database name>" for each database. When these calls finish, the query executes pretty quickly. My interpretation of this is that most of the time is spent on analyzing the metastore and building a query plan, perhaps some sort of Metastore in-memory cache is building built (but it happens on every call). But I am not really sure how to debug it further, nor could find in the code where this is happening. Any advice on what's causing it or how to troubleshoot? p.s The metadata (i.e original tables and databases) is actually coming from a very old Hive version (1.1), and we migrated it to the newest version of the Metastore using the upgrade tool. In the original Hive version materialized views were not even supported. Cheers, Eugene -- | | | | | Eugene Miretsky | | Managing Partner | Badal.io | Book a meeting /w me! | | mobile: 416-568-9245 | | email: eug...@badal.io | | | | |
