Re: Should we consider Spark3 support for Hive on Spark

Yes exactly. This is what is recommended, cause hive on Spark has little interest. However there is nothing enforcing not todo it. Important to me cause i sit here and work on grassroots marrieng hive on kafka-streams. Owen O'Malley schrieb am Mi., 24. Aug. 2022, 18:51: > Hive on Spar

Re: Should we consider Spark3 support for Hive on Spark

Hive on Spark is not recommended. The recommended path is to use either Tez or LLAP. If you already are using Spark 3, it would be far easier to use Spark SQL. .. Owen On Wed, Aug 24, 2022 at 3:46 AM Fred Bai wrote: > Hi everyone: > > Do we have any support for Hive on Spark? I nee

Re: Should we consider Spark3 support for Hive on Spark

s can do what you are trying to do. > Use canned solutions such as confluence or AWS EMR and save yourself weeks of > wasted effort. > > Hernán > On Tuesday, August 23, 2022 at 08:46:30 PM PDT, Fred Bai > wrote: > > > Hi everyone: > > Do we have any support

Re: Should we consider Spark3 support for Hive on Spark

s can do what you are trying to do. > Use canned solutions such as confluence or AWS EMR and save yourself weeks of > wasted effort. > > Hernán > On Tuesday, August 23, 2022 at 08:46:30 PM PDT, Fred Bai > wrote: > > > Hi everyone: > > Do we have any support for Hive o

Re: Should we consider Spark3 support for Hive on Spark

, 2022 at 08:46:30 PM PDT, Fred Bai wrote: Hi everyone: Do we have any support for Hive on Spark? I need Hive on Spark, but my Spark version is 3.X. I found Hive incompatible with Spark3, I modify a lot of code to be compatible. Hive on Spark has deprecated?  And. Hive on Spark is very slow

Should we consider Spark3 support for Hive on Spark

Hi everyone: Do we have any support for Hive on Spark? I need Hive on Spark, but my Spark version is 3.X. I found Hive incompatible with Spark3, I modify a lot of code to be compatible. Hive on Spark has deprecated? And. Hive on Spark is very slow when the job executes.

Re: Time to Remove Hive-on-Spark

rep -v xml|grep -v > properties|grep -v q.out` > > > On 1/28/22 2:32 PM, Stamatis Zampetakis wrote: > > Hi team, > > > > Almost one year has passed since the last exchange in this discussion and > > if I am not wrong there has been no effort to revive Hive-on-Spar

Re: Time to Remove Hive-on-Spark

st one year has passed since the last exchange in this discussion and if I am not wrong there has been no effort to revive Hive-on-Spark. To be more precise, I don't think I have seen any Spark related JIRA for quite some time now and although I don't want to rush into conclusions, ther

Re: Time to Remove Hive-on-Spark

Hi team, Almost one year has passed since the last exchange in this discussion and if I am not wrong there has been no effort to revive Hive-on-Spark. To be more precise, I don't think I have seen any Spark related JIRA for quite some time now and although I don't want to rush into c

hive on spark submit to yarn pools?

hive on spark + sentry jdbc:hive2://hiveser:1/;user=ajxtj;password=123456;hive.server2.proxy.user=jztwk pro.put("hiveconf:spark.yarn.queue","root.jzyc"); I use yarn pool root.jzyc but root.jzc only hive and ajxtj can use. so I want to use jztkw to submit root.jzyc

Re: Removing Hive-on-Spark

; Hello Hive Users. >> >> I am interested in gathering some feedback on the adoption of >> Hive-on-Spark. >> >> Does anyone care to volunteer their usage information and would you be >> open to removing it in favor of Hive-on-Tez in subsequent releases of Hive? >

Re: Removing Hive-on-Spark

Why would it be this way instead of the other way around? On Mon, 27 Jul 2020 at 12:27, David wrote: > Hello Hive Users. > > I am interested in gathering some feedback on the adoption of > Hive-on-Spark. > > Does anyone care to volunteer their usage information and woul

Removing Hive-on-Spark

Hello Hive Users. I am interested in gathering some feedback on the adoption of Hive-on-Spark. Does anyone care to volunteer their usage information and would you be open to removing it in favor of Hive-on-Tez in subsequent releases of Hive? If you are on MapReduce still, would you be open to

About the Hive on Spark 3.x upgrade plan

Hello, May I ask about the Hive on Spark 3.x upgrade plan? I found the newest dependent Spark version is 2.4.5 on Master Branch. Thanks, --- wjl

Re: Running Hive on Spark

"Hive on Spark" uses Spark purely as execution engine. It would not get the benefits of codegen and other optimizations of Spark. If it is mainly for testing, OOTB parameters should work without issues. However, Tez has lot better edge than Hive on Spark. Some of the areas where Hiv

Re: Running Hive on Spark

engine sending queries to Hive using HiveServer2 (Thrift), >> and using the SparkThriftServer (I thought it was built on top of >> HiveServer2) ? Could I read more about the differences somewhere ? >> >> I followed the following docs: >> https://cwiki.apache.org/confluence/dis

Re: Running Hive on Spark

ift), > and using the SparkThriftServer (I thought it was built on top of > HiveServer2) ? Could I read more about the differences somewhere ? > > I followed the following docs: > https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started > and after changing the e

Running Hive on Spark

2 (Thrift), and using the SparkThriftServer (I thought it was built on top of HiveServer2) ? Could I read more about the differences somewhere ? I followed the following docs: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started and after changing the execution engi

Re: Is hive on spark works with spark 2.3.0

trying >>>> all this it shows some compatibility issues. >>>> >>>> >>>> Regards, >>>> Sachin Janani >>>> >>>> On Tue, Jun 19, 2018 at 5:02 PM, Sahil Takiar >>>> wrote: >>>> >

Re: Is hive on spark works with spark 2.3.0

compatibility issues. >>> >>> >>> Regards, >>> Sachin Janani >>> >>> On Tue, Jun 19, 2018 at 5:02 PM, Sahil Takiar >>> wrote: >>> > Yes, Hive 3.0.0 works with Spark 2.3.0 - this section of the wiki has >>> &g

Re: Is hive on spark works with spark 2.3.0

gt; >> On Tue, Jun 19, 2018 at 5:02 PM, Sahil Takiar >> wrote: >> > Yes, Hive 3.0.0 works with Spark 2.3.0 - this section of the wiki has >> > details on which Hive releases support which Spark versions. >> > >> > On Tue, Jun 19, 2018 at 5:59 AM, Sach

Re: Is hive on spark works with spark 2.3.0

the wiki has > > details on which Hive releases support which Spark versions. > > > > On Tue, Jun 19, 2018 at 5:59 AM, Sachin janani < > sachin.janani...@gmail.com> > > wrote: > >> > >> Hi, > >> I am trying to run hive on spark by follow

Re: Is hive on spark works with spark 2.3.0

:02 PM, Sahil Takiar wrote: > Yes, Hive 3.0.0 works with Spark 2.3.0 - this section of the wiki has > details on which Hive releases support which Spark versions. > > On Tue, Jun 19, 2018 at 5:59 AM, Sachin janani > wrote: >> >> Hi, >> I am trying to run hiv

Re: Is hive on spark works with spark 2.3.0

Yes, Hive 3.0.0 works with Spark 2.3.0 - this <https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-VersionCompatibility> section of the wiki has details on which Hive releases support which Spark versions. On Tue, Jun 19, 2018 at 5

Is hive on spark works with spark 2.3.0

Hi, I am trying to run hive on spark by following the steps mentioned here- https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started , but getting many compatibility issues like NoSuchMethodError, NoSuchFieldException etc. So just need to know if it works and whether

Re: hive on spark - why is it so hard?

made some progress after much copying of jar files around (as > alluded to by Gopal previously on this thread). > > > following the instructions here: > https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started > > and doing this as instructed will lea

Re: hive on spark - why is it so hard?

so... i made some progress after much copying of jar files around (as alluded to by Gopal previously on this thread). following the instructions here: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started and doing this as instructed will leave off about a dozen or

Re: hive on spark - why is it so hard?

at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSpark >>> Session(SparkUtilities.java:126) [hive-exec-2.3.0.jar:2.3.0] >>> at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerPar >>> allelism.getSparkMemoryAndCores(SetSparkReducerParallelism.java:236) >>> [hive-exec-2.3.0.

Re: hive on spark - why is it so hard?

erParallelism.java:236) >> [hive-exec-2.3.0.jar:2.3.0] >> >> >> i'll dig some more tomorrow. >> >> On Tue, Sep 26, 2017 at 8:23 PM, Stephen Sprague >> wrote: >> >>> oh. i missed Gopal's reply. oy... that sounds foreboding. I'

Re: hive on spark - why is it so hard?

23 PM, Stephen Sprague > wrote: > >> oh. i missed Gopal's reply. oy... that sounds foreboding. I'll keep you >> posted on my progress. >> >> On Tue, Sep 26, 2017 at 4:40 PM, Gopal Vijayaraghavan >> wrote: >> >>> Hi, >>> >>> > org.apache.hadoop.hive

Re: hive on spark - why is it so hard?

g. I'll keep you > posted on my progress. > > On Tue, Sep 26, 2017 at 4:40 PM, Gopal Vijayaraghavan > wrote: > >> Hi, >> >> > org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a >> spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed >> to create spark cl

Re: hive on spark - why is it so hard?

get a > spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed > to create spark client. > > I get inexplicable errors with Hive-on-Spark unless I do a three step > build. > > Build Hive first, use that version to build Spark, use that Spark version > to rebuild Hive. > > I have to do this

Re: hive on spark - why is it so hard?

pache.hadoop.util.RunJar.run(RunJar.java:221) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:136) >> >> >> I bugs me that that class is in spark-core_2.11-2.2.0.jar yet so >> seemingly out of reach. :( >> >> >> >> On Tue, Sep 26

Re: hive on spark - why is it so hard?

Hi, > org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a spark > session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create > spark client. I get inexplicable errors with Hive-on-Spark unless I do a three step build. Build Hive first, use that v

Re: hive on spark - why is it so hard?

r yet so seemingly > out of reach. :( > > > > On Tue, Sep 26, 2017 at 2:44 PM, Sahil Takiar > wrote: > >> Hey Stephen, >> >> Can you send the full stack trace for the NoClassDefFoundError? For Hive >> 2.3.0, we only support Spark 2.0.0. Hive may work with

Re: hive on spark - why is it so hard?

h more recent versions > of Spark, but we only test with Spark 2.0.0. > > --Sahil > > On Tue, Sep 26, 2017 at 2:35 PM, Stephen Sprague > wrote: > >> * i've installed hive 2.3 and spark 2.2 >> >> * i've read this doc plenty of times -> https://cwi

Re: hive on spark - why is it so hard?

nstalled hive 2.3 and spark 2.2 > > * i've read this doc plenty of times -> https://cwiki.apache.org/ > confluence/display/Hive/Hive+on+Spark%3A+Getting+Started > > * i run this query: > >hive --hiveconf hive.root.logger=DEBUG,console -e 'set > hive.execution

hive on spark - why is it so hard?

* i've installed hive 2.3 and spark 2.2 * i've read this doc plenty of times -> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started * i run this query: hive --hiveconf hive.root.logger=DEBUG,console -e 'set hive.execution.engine=spark; sele

Re: Hive on Spark

Xuefu is planning to give a talk on Hive-on-Spark @Uber the user meetup this week. We can check if can share the presentation on this list for folks who can't attend the meetup. https://www.meetup.com/Hive-User-Group-Meeting/events/242210487/ On Mon, Aug 21, 2017 at 11:44 PM, peter zhang

Hive on Spark

Hi All, Has anybody used hive on spark in your production environment? How does it's the stability and performance compared with spark sql? Hope anybody can share your experience. Thanks in advance!

?????? hive on spark - version question

3 ??: "user@hive.apache.org"; : Re: hive on spark - version question :( gettin' no love on this one. any SME's know if Spark 2.1.0 will work with Hive 2.1.0 ? That JavaSparkListener class looks like a deal breaker to me, alas. thanks in advance. Cheers, Stephen.

Re: hive on spark - version question

hose who promote and DOCUMENT > spark on hive is that, seemingly, there is absolutely no evidence seen that > says that hive on spark WORKS. > As a matter of fact, after a lot of pain, I noticed it is not supported by > just about anybody. > > If someone dares to do

Re: hive on spark - version question

Stephan, Thanks for the response. The one thing that I don't appreciate from those who promote and DOCUMENT spark on hive is that, seemingly, there is absolutely no evidence seen that says that hive on spark WORKS. As a matter of fact, after a lot of pain, I noticed it is not supported by

Re: hive on spark - version question

sion, don't try it. Or try using Tez/Hive instead of Spark/Hive if >> you are querying large files. >> >> >> >> On Friday, March 17, 2017 11:33 AM, Stephen Sprague >> wrote: >> >> >> :( gettin' no love on this one. any SME&#x

Re: hive on spark - version question

; > thanks in advance. > > Cheers, > Stephen. > > On Mon, Mar 13, 2017 at 10:32 PM, Stephen Sprague > wrote: > > hi guys, > wondering where we stand with Hive On Spark these days? > > i'm trying to run Spark 2.1.0 with Hive 2.1.0 (purely coincidental > ve

Re: hive on spark - version question

SparkListener class looks like a deal breaker to me, alas. thanks in advance. Cheers, Stephen. On Mon, Mar 13, 2017 at 10:32 PM, Stephen Sprague wrote: hi guys, wondering where we stand with Hive On Spark these days? i'm trying to run Spark 2.1.0 with Hive 2.1.0 (purely coincidental ver

Re: hive on spark - version question

g where we stand with Hive On Spark these days? > > i'm trying to run Spark 2.1.0 with Hive 2.1.0 (purely coincidental > versions) and running up against this class not found: > > java.lang.NoClassDefFoundError: org/apache/spark/JavaSparkListener > > > s

hive on spark - version question

hi guys, wondering where we stand with Hive On Spark these days? i'm trying to run Spark 2.1.0 with Hive 2.1.0 (purely coincidental versions) and running up against this class not found: java.lang.NoClassDefFoundError: org/apache/spark/JavaSparkListener searching the Cyber i find this:

Re: Need inputs on configuring hive timeout + hive on spark : Job hasn't been submitted after 61s. Aborting it.

Naresh, The properties hive.spark.job.monitor.timeout and hive.spark.client.server. connect.timeout in hive-site.xml control Hive on Spark timeouts. Details at https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Spark Ian Cook Cloudera On Thu, Feb

Need inputs on configuring hive timeout + hive on spark : Job hasn't been submitted after 61s. Aborting it.

Hello, i am facing this issue "Job hasn't been submitted after 61s. Aborting it." when i am running multiple hive queries. Details: (Hive on Spark) I am using spark dynamic allocation and external shuffle service (yarn) Assume one queries is using all of the resources in the c

hive on spark ，three tables(one is small, others are big)，cannot go mapjoin

il" cannot go mapjoin...... and I DEBUG the code,: in hive-on-spark: (1)(select a.src_ip,b.appid from small_tbl a join im b on a.src_ip=b.src_ip) ->>> MapWork.getMapredLocalWork() is OK,there is one MapRedLocalWork Object (2) the result of the previous stage named ‘

Re: please give me the permission to update the wiki of hive on spark

Done. Welcome to the Hive wiki team, Kelly, and happy new year! -- Lefty On Mon, Jan 2, 2017 at 5:40 PM, Zhang, Liyun wrote: > Hi > > I want to update wiki<https://cwiki.apache.org/ > confluence/display/Hive/Hive+on+Spark%3A+Getting+Started> of hive on > spark

please give me the permission to update the wiki of hive on spark

Hi I want to update wiki<https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started> of hive on spark because HIVE-8373, my Confluence<https://cwiki.apache.org/confluence/signup.action> username is kellyzly, please provide the privilege to me to update

please give me the permission to update the wiki of hive on spark

Hi I want to update wiki<https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started> of hive on spark because HIVE-8373, my Confluence<https://cwiki.apache.org/confluence/signup.action> username is kellyzly, please provide the privilege to me to update

RE: When Hive on Spark will support Spark 2.0?

The version that will support Spark2.0 is Hive2.2 No not know yet when this is going to be release. -Original Message- From: baipeng [mailto:b...@meitu.com] Sent: 07 December 2016 08:04 To: user@hive.apache.org Subject: When Hive on Spark will support Spark 2.0? Does Anyone know when

When Hive on Spark will support Spark 2.0?

Does Anyone know when Hive will release version to support Spark 2.0? Now hive 2.1.0 only supports spark 1.6.

RE: Hive on Spark not working

Being unable to integrate separately Hive with Spark I just started directly on Spark the thrift server. Now it is working as expected. From: Mich Talebzadeh [mailto:mich.talebza...@gmail.com] Sent: 29 November 2016 11:12 To: user Subject: Re: Hive on Spark not working Hive on Spark engine

RE: Hive on Spark not working

gmail.com] Sent: 29 November 2016 11:12 To: user Subject: Re: Hive on Spark not working Hive on Spark engine only works with Spark 1.3.1. Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com Dis

Re: Hive on Spark not working

Hive on Spark engine only works with Spark 1.3.1. Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpre

Re: Hive on Spark not working

ClassNotFoundException generally means that jars are missing from your class path. You probably need to link the spark jar to $HIVE_HOME/lib https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive On Tue, Nov 29, 2016 at 2:03

Hive on Spark not working

Hi Guys No matter what I do that when I execute "select count(*) from employee" I get the following output on the logs: It is quiet funny because if I put hive.execution.engine=mr the output is correct. If I put hive.execution.engine=spark then I get the bellow errors. If I do the search directl

Re: Hive on Spark - Mesos

t; > John > > > On Thu, Sep 15, 2016 at 4:20 PM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> Yes you can. Hive on Spark meaning Hive using Spark as its execution >> engine works fine. The version that I managed to make it work is any Hive >> v

Re: Hive on Spark - Mesos

Did you run it on Mesos? Your presentation doesn't mention Mesos at all... John On Thu, Sep 15, 2016 at 4:20 PM, Mich Talebzadeh wrote: > Yes you can. Hive on Spark meaning Hive using Spark as its execution > engine works fine. The version that I managed to make it work is any Hiv

Hive on Spark - Mesos

Hey all, I was experimenting with some bleeding edge Hive. (2.1) and trying to get it to run on bleeding edge Spark (2.0). Spark is working fine, I can query the data all is setup, however, I can't get Hive on Spark to work. I understand it's not really a thing (Hive on Spark on Mesos

Re: Hive On Spark - ORC Table - Hive Streaming Mutation API

r hive-site.xml so I cannot figure it out. > > BTW you are using Spark standalone as the mode. I tend to use yarn-client. > > Now back to the above issue. Do other queries work OK with Hive on Spark? > > Some of those perf parameters can be set up in Hive session itself or > thr

Re: Hive On Spark - ORC Table - Hive Streaming Mutation API

Hive on Spark? Some of those perf parameters can be set up in Hive session itself or through init file set spark.home=/usr/lib/spark-1.6.2-bin-hadoop2.6; set spark.master=yarn; set spark.deploy.mode=client; set spark.executor.memory=8g; set spark.driver.memory=8g; set spark.executor.instances=6

Hive On Spark - ORC Table - Hive Streaming Mutation API

Hi, After several days trying to figure out the problem I'm stuck with a class cast exception when running a query with hive on spark on orc tables that I updated with the streaming mutation api of hive 2.0. The context is the following: For hive: The version is the latest available fro

Re: hive on spark job not start enough executors

All the parameters except spark.executor.instances are specified in spark-default.conf located in hive's conf folder. So I think it's a yes. I also checked on spark's web page when a hive on spark job is running, the parameters shown on the web page are exactly what I specifie

Re: hive on spark job not start enough executors

when you start hive on spark do you set any parameters for the submitted job (or read them from init file)? set spark.master=yarn; set spark.deploy.mode=client; set spark.executor.memory=3g; set spark.driver.memory=3g; set spark.executor.instances=2; set spark.ui.port=; Dr Mich Talebzadeh

hive on spark job not start enough executors

Hi there, I encountered a problem that makes hive on spark with a very low performance. I'm using spark 1.6.2 and hive 2.1.0, I specified spark.shuffle.service.enabledtrue spark.dynamicAllocation.enabled true in my spark-default.conf file (the file is in both spark and hive

Re: Hive on spark

s, damage or destruction. > > > > On 28 July 2016 at 04:24, Mudit Kumar wrote: > >> Yes Mich,exactly. >> >> Thanks, >> Mudit >> >> From: Mich Talebzadeh >> Reply-To: >> Date: Thursday, July 28, 2016 at 1:08 AM >> To: user &g

Re: Hive on spark

; >> Thanks, >> Mudit >> >> From: Mich Talebzadeh >> Reply-To: >> Date: Thursday, July 28, 2016 at 1:08 AM >> To: user >> Subject: Re: Hive on spark >> >> You mean you want to run Hive using Spark as the execution engine wh

Re: Hive on spark

Thanks Guys for the help! Thanks, Mudit From: Mich Talebzadeh Reply-To: Date: Thursday, July 28, 2016 at 9:43 AM To: user Subject: Re: Hive on spark Hi, I made a presentation in London on 20th July on this subject:. In that I explained how to make Spark work as an execution engine for

Re: Hive on spark

ruction. On 28 July 2016 at 04:24, Mudit Kumar wrote: > Yes Mich,exactly. > > Thanks, > Mudit > > From: Mich Talebzadeh > Reply-To: > Date: Thursday, July 28, 2016 at 1:08 AM > To: user > Subject: Re: Hive on spark > > You mean you want to run Hive usi

Re: Hive on spark

mudit, this link can guide you - https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started Thanks, Karthik On Thu, Jul 28, 2016 at 8:54 AM, Mudit Kumar wrote: > Yes Mich,exactly. > > Thanks, > Mudit > > From: Mich Talebzadeh > Reply-To: >

Re: Hive on spark

Yes Mich,exactly. Thanks, Mudit From: Mich Talebzadeh Reply-To: Date: Thursday, July 28, 2016 at 1:08 AM To: user Subject: Re: Hive on spark You mean you want to run Hive using Spark as the execution engine which uses Yarn by default? Something like below hive> select max(id) f

Re: Hive on spark

You mean you want to run Hive using Spark as the execution engine which uses Yarn by default? Something like below hive> select max(id) from oraclehadoop.dummy_parquet; Starting Spark Job = 8218859d-1d7c-419c-adc7-4de175c3ca6d Query Hive on Spark job[1] stages: 2 3 Status: Running (Hive

Hive on spark

Hi All, I need to configure hive cluster based on spark engine (yarn). I already have a running hadoop cluster. Can someone point me to relevant documentation? TIA. Thanks, Mudit

Re: Presentation in London: Running Spark on Hive or Hive on Spark

ve or Hive on Spark Dr Mich  do you have any slides or videos available for the presentation you did @Canary Wharf?kindest regards marco On Wed, Jul 6, 2016 at 10:37 PM, Mich Talebzadeh wrote: Dear forum members I will be presenting on the topic of "Running Spark on Hive or Hive on Spark, yo

Re: Presentation in London: Running Spark on Hive or Hive on Spark

.@gmail.com] > *Sent:* 15 July 2016 11:04 > *To:* Mich Talebzadeh > *Cc:* user @spark ; user > *Subject:* Re: Presentation in London: Running Spark on Hive or Hive on > Spark > > > > Dr Mich > > do you have any slides or videos available for the presentation you

Re: Presentation in London: Running Spark on Hive or Hive on Spark

Hi Mich, Would it be on YouTube , post session ? - Harmeet On Jul 7, 2016, at 3:07, Mich Talebzadeh wrote: > Dear forum members > > I will be presenting on the topic of "Running Spark on Hive or Hive on Spark, > your mileage varies" in Future of Data: London >

Re: Presentation in London: Running Spark on Hive or Hive on Spark

Thanks. Will this presentation recorded as well? Regards On Wednesday, 6 July 2016, 22:38, Mich Talebzadeh wrote: Dear forum members I will be presenting on the topic of "Running Spark on Hive or Hive on Spark, your mileage varies" in Future of Data: London DetailsOr

Presentation in London: Running Spark on Hive or Hive on Spark

Dear forum members I will be presenting on the topic of "Running Spark on Hive or Hive on Spark, your mileage varies" in Future of Data: London <http://www.meetup.com/futureofdata-london/events/232423292/> *Details* *Organized by: Hortonworks <http://hortonworks.com/>*

Hive on Spark issues with Hive-XML-Serde

Hi we are using Cloudera 5.7.0 there's a use case to process XML data, we are using the https://github.com/dvasilen/Hive-XML-SerDe XML serde is working with Hive execution engine as Map-Reduce, we enabled Hive on Spark to test the performance, and we are facing following issue 16/06/

Re: Hive on Spark engine

Thanks Ted, More interested in general availability of Hive 2 on Spark 1.6 engine as opposed to Vendors specific custom built. Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Hive on Spark engine

According to: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/bk_HDP_RelNotes-20151221.pdf Spark 1.5.2 comes out of box. Suggest moving questions on HDP to Hortonworks forum. Cheers On Sat, Mar 26, 2016 at 3:32 PM, Mich Talebzadeh wrote: > Thanks Jorn. > > Just to be

Re: Hive on Spark engine

Thanks Jorn. Just to be clear they get Hive working with Spark 1.6 out of the box (binary download)? The usual work-around is to build your own package and get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib. Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/

Re: Hive on Spark engine

If you check the newest Hortonworks distribution then you see that it generally works. Maybe you can borrow some of their packages. Alternatively it should be also available in other distributions. > On 26 Mar 2016, at 22:47, Mich Talebzadeh wrote: > > Hi, > > I am running Hive 2 and now Spar

Hive on Spark engine

Hi, I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign that Hive can utilise a Spark engine higher than 1.3.1 My understanding was that there were miss-match on Hadoop assembly Jar files that cause Hive not being able to run on Spark using the binary downloads. I just tried H

Re: Error in Hive on Spark

r in > > > yarn cluster. Then application threw the exception: > "hive-exec-2.0.0.jar > > > does not exist ...". > > > > > > Can it be set property of hive-exec-2.0.0.jar path in application ? > > > Something like &

Re: Error in Hive on Spark

", > "hdfs://storm0:9000/tmp/spark-assembly-1.4.1-hadoop2.6.0.jar")'. > If not, is it possible to achieve in the future version? > > > > > 2016-03-10 23:51 GMT+08:00 Xuefu Zhang : > >> You can probably avoid the problem by se

Re: Hive on Spark performance

m_number NOT LIKE 'UG%' AND >> f.item_number NOT LIKE 'DEV%' >> group by >> f.description, >> f.item_number >> ### >> >> This query works fine in oracle but not Hive or Spark. >> So the problem is: "sum(f.df_a * (select c

Re: Hive on Spark performance

he problem is: "sum(f.df_a * (select count(1) from e.mv_A_h_a where > hb_h_name = r.h_id)) as df_a" field. > > > Thanks, > Wlodek > -- > > > On Sunday, March 13, 2016 7:36 PM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > > > Depending on th

Re: Hive on Spark performance

where hb_h_name = r.h_id)) as df_a" field. Thanks,Wlodek-- On Sunday, March 13, 2016 7:36 PM, Mich Talebzadeh wrote: Depending on the version of Hive on Spark engine. As far as I am aware the latest version of Hive that I am using (Hive 2) has improvements compared to the previou

Re: Hive on Spark performance

Depending on the version of Hive on Spark engine. As far as I am aware the latest version of Hive that I am using (Hive 2) has improvements compared to the previous versions of Hive (0.14,1.2.1) on Spark engine. As of today I have managed to use Hive 2.0 on Spark version 1.3.1. So it is not the

Hive on Spark performance

Just curious if you could share your experience on the performance of spark in your company? How much data do you process? And what's the latency you are getting with spark engine? Vidya

Re: Error in Hive on Spark

u Zhang : > You can probably avoid the problem by set environment variable SPARK_HOME > or JVM property spark.home that points to your spark installation. > > --Xuefu > > On Thu, Mar 10, 2016 at 3:11 AM, Stana wrote: > > > I am trying out Hive on Spark with hive 2.0.0 and sp

Error in Hive on Spark

I am trying out Hive on Spark with hive 2.0.0 and spark 1.4.1, and executing org.apache.hadoop.hive.ql.Driver with java application. Following are my situations: 1.Building spark 1.4.1 assembly jar without Hive . 2.Uploading the spark assembly jar to the hadoop cluster. 3.Executing the java

Re: Hive on Spark Engine versus Spark using Hive metastore

6:46 PM, Xuefu Zhang wrote: >>> >>>> When comparing the performance, you need to do it apple vs apple. In >>>> another thread, you mentioned that Hive on Spark is much slower than Spark >>>> SQL. However, you configured Hive such that only two tasks can

RE: Hive on Spark Engine versus Spark using Hive metastore

nt: 04 February 2016 17:41 To: user@hive.apache.org Subject: Re: Hive on Spark Engine versus Spark using Hive metastore Hive is not the correct tool for every problem. Use the tool that makes the most sense for your problem and your experience. Many people like hive because it is genera

  1   2   3   4   >