Re: CachingCatalog question

2021-02-25 Thread Peter Vary
Hi Nicolas, Sadly, no profiling info yet. At the time of the run we were checking correctness, and only "took note" of the perf issue. Still there are some fixes and a few more important base features we would like to tackle (CTAS, IOW), before we start to focus on performance. Still in the meanti

Re: CachingCatalog question

2021-02-25 Thread Grant Nicholas
Were you able to confirm why the loadTable() call was slow through profiling? >From personal experience, I've seen similar calls behave slowly when the connection to the hive metastore was unstable and retries to the hive metastore(with exponential backoff) occurred. On Thu, Feb 25, 2021 at 4:45

CachingCatalog question

2021-02-25 Thread Peter Vary
Hi Team, Recently we have been playing around 100GB TCP-DS queries above Iceberg backed Hive tables. We have found that for queries accessing big partitioned tables had very-very slow compilation time. One example is query77 the compilation took ~20min: INFO : Completed compiling command(quer