Hi Linlin,
have you got the solution for this issue, if yes then what are the thing
need to make correct,because I am also getting same error,when submitting
spark job in cluster mode getting error as under -
2015-04-14 18:16:43 DEBUG Transaction - Transaction rolled back in 0 ms
2015-04-14 18:16:4
es.
This worked for me, and I can have HiveContext API to talk to Hive metastore,
and vice versa. Thanks.
Date: Thu, 5 Feb 2015 16:59:12 -0800
From: lian.cs@gmail.com
To: linlin200...@gmail.com; huaiyin@gmail.com
CC: user@spark.apache.org
Subject: Re: Spark sql failed in yarn-cluste
Hi Jenny,
You may try to use |--files $SPARK_HOME/conf/hive-site.xml
--driver-class-path hive-site.xml| when submitting your application. The
problem is that when running in cluster mode, the driver is actually
running in a random container directory on a random executor node. By
using |--fil
cluster_table SELECT key,
value").collect().foreach(println)
}
}
From: huaiyin....@gmail.com
Date: Wed, 13 Aug 2014 16:56:13 -0400
Subject: Re: Spark sql failed in yarn-cluster mode when connecting to
non-default hive database
To: linlin200...@gmail.com
CC: lian.cs@gmail.com; user@spark.apache
built your JARs with -Phadoop-provided, and other common libraries that
are required.
From: alee...@hotmail.com
To: user@spark.apache.org
CC: lian.cs@gmail.com; linlin200...@gmail.com; huaiyin@gmail.com
Subject: RE: Spark sql failed in yarn-cluster mode when connecting to
non-default hive dat
ot;jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true")
From: huaiyin@gmail.com
Date: Wed, 13 Aug 2014 16:56:13 -0400
Subject: Re: Spark sql failed in yarn-cluster mode when connecting to
non-default hive database
To: linlin200...@gmail.com
CC: lian.cs@gmail.com; user@s
I think the problem is that when you are using yarn-cluster mode, because
the Spark driver runs inside the application master, the hive-conf is not
accessible by the driver. Can you try to set those confs by using
hiveContext.set(...)? Or, maybe you can copy hive-site.xml to spark/conf in
the node
Hi Yin,
hive-site.xml was copied to spark/conf and the same as the one under
$HIVE_HOME/conf.
through hive cli, I don't see any problem. but for spark on yarn-cluster
mode, I am not able to switch to a database other than the default one, for
Yarn-client mode, it works fine.
Thanks!
Jenny
On
Hi Jenny,
Have you copied hive-site.xml to spark/conf directory? If not, can you put
it in conf/ and try again?
Thanks,
Yin
On Mon, Aug 11, 2014 at 8:57 PM, Jenny Zhao wrote:
>
> Thanks Yin!
>
> here is my hive-site.xml, which I copied from $HIVE_HOME/conf, didn't
> experience problem conne
Thanks Yin!
here is my hive-site.xml, which I copied from $HIVE_HOME/conf, didn't
experience problem connecting to the metastore through hive. which uses DB2
as metastore database.
hive.hwi.listen.port
hive.querylog.location
/var/ibm/biginsights/hive/query/${user.name}
Hi Jenny,
How's your metastore configured for both Hive and Spark SQL? Which
metastore mode are you using (based on
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
)?
Thanks,
Yin
On Mon, Aug 11, 2014 at 6:15 PM, Jenny Zhao wrote:
>
>
> you can reproduce this issue
you can reproduce this issue with the following steps (assuming you have
Yarn cluster + Hive 12):
1) using hive shell, create a database, e.g: create database ttt
2) write a simple spark sql program
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql._
import org.apache
Since you were using hql(...), it’s probably not related to JDBC driver.
But I failed to reproduce this issue locally with a single node pseudo
distributed YARN cluster. Would you mind to elaborate more about steps to
reproduce this bug? Thanks
On Sun, Aug 10, 2014 at 9:36 PM, Cheng Lian wrote
Hi Jenny, does this issue only happen when running Spark SQL with YARN in
your environment?
On Sat, Aug 9, 2014 at 3:56 AM, Jenny Zhao wrote:
>
> Hi,
>
> I am able to run my hql query on yarn cluster mode when connecting to the
> default hive metastore defined in hive-site.xml.
>
> however, if
14 matches
Mail list logo
