FileSystem.getContentSummary for total size stats in DetermineTableStats VS CommandUtils?

2018-01-02 Thread Jacek Laskowski
Hi, I was wondering what's wrong with FileSystem.getContentSummary in CommandUtils.calculateLocationSize as "expressed" in the comment [1]: // This method is mainly based on // org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForTable(HiveConf, Table) // in Hive 0.13 (except that

SQL Visualization for cached Dataset

2018-01-02 Thread Tomasz Gawęda
Hi, Recently I had to optimize few Apache Spark SQL queries. Some of the Datasets were reused, so they were cached. However after caching I don't see SQL Visualization for the cached Dataset in Spark UI - I see only InMemoryRelation node. Explain result at the bottom of the page still has full