It is specific to HoS 发件人: Mich Talebzadeh [mailto:mich.talebza...@gmail.com] 发送时间: 2016年7月14日 11:55 收件人: user 主题: Re: 答复: Using Spark on Hive with Hive also using Spark as its execution engine
Hi Wenli, You mentioned: Coming to HoS, I think the main problem now is many optimization should be done , but seems no progress. Like conditional task , union sql cann’t convert to mapjoin(hive-9044) etc, so many optimize feature is pending, no one working on them. Is this issue specific to Hive on Spark or they apply equally to Hive on MapReduce as well. In other words a general issue with Hive optimizer case hive-9044? Thanks Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 14 July 2016 at 01:56, Wangwenli <wangwe...@huawei.com<mailto:wangwe...@huawei.com>> wrote: Seems LLAP like tachyon, which purpose is also cache data between applications. Coming to HoS, I think the main problem now is many optimization should be done , but seems no progress. Like conditional task , union sql cann’t convert to mapjoin(hive-9044) etc, so many optimize feature is pending, no one working on them. On contrast, sparksql is improve very fast Regards wenli 发件人: Mich Talebzadeh [mailto:mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>] 发送时间: 2016年7月13日 7:21 收件人: user 主题: Re: Using Spark on Hive with Hive also using Spark as its execution engine I just read further notes on LLAP. As Gopal explained LLAP has more to do that just in-memory and I quote Gopal: "... LLAP is designed to be hammered by multiple user sessions running different queries, designed to automate the cache eviction & selection process. There's no user visible explicit .cache() to remember - it's automatic and concurrent. ..." Sounds like what Oracle classic or SAP ASE do in terms of buffer management strategy. As I understand Spark does not have this concept of hot area (MRU/LRU chain). It loads data into its memory if needed and gets rid of it. if ten users read the same table those blocks from that table will be loaded 10 times which is not efficient. LLAP is more intelligent in this respect. So somehow it maintains a Most Recently Used (MRU), Least Recently Used (LRU) chain. It maintains this buffer management strategy throughout the cluster. It must be using some clever algorithm to do so. Cheers . Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 12 July 2016 at 15:59, Mich Talebzadeh <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>> wrote: Thanks Alan. Point taken. In mitigation, here are members in Spark forum who have shown (interest) in using Hive directly and I quote one: "Did you have any benchmark for using Spark as backend engine for Hive vs using Spark thrift server (and run spark code for hive queries)? We are using later but it will be very useful to remove thriftserver, if we can. " Cheers, Mich Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 12 July 2016 at 15:39, Alan Gates <alanfga...@gmail.com<mailto:alanfga...@gmail.com>> wrote: > On Jul 11, 2016, at 16:22, Mich Talebzadeh > <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>> wrote: > > <snip> > • If I add LLAP, will that be more efficient in terms of memory usage > compared to Hive or not? Will it keep the data in memory for reuse or not. > Yes, this is exactly what LLAP does. It keeps a cache of hot data (hot columns of hot partitions) and shares that across queries. Unlike many MPP caches it will cache the same data on multiple nodes if it has more workers that want to access the data than can be run on a single node. As a side note, it is considered bad form in Apache to send a message to two lists. It causes a lot of background noise for people on the Spark list who probably aren’t interested in Hive performance. Alan.
