The thing is that the class it is complaining about is part of the spark
assembly jar, not in my extra jar. The assembly jar was compiled with
-Phive which is proven by the fact that it works with the same SPARK_HOME
when run as shell.
On 23 July 2015 at 17:33, Akhil Das wrote:
> You can try add
You can try adding that jar in SPARK_CLASSPATH (its deprecated though) in
spark-env.sh file.
Thanks
Best Regards
On Tue, Jul 21, 2015 at 7:34 PM, Michal Haris
wrote:
> I have a spark program that uses dataframes to query hive and I run it
> both as a spark-shell for exploration and I have a run
nov.com | 617.299.6746
>
>
> From: Yin Huai
> Date: Monday, July 6, 2015 at 11:41 AM
> To: Denny Lee
> Cc: Simeon Simeonov , Andy Huang ,
> user
>
> Subject: Re: 1.4.0 regression: out-of-memory errors on small data
>
> Hi Sim,
>
> I
ang
mailto:andy.hu...@servian.com.au>>, user
mailto:user@spark.apache.org>>
Subject: Re: 1.4.0 regression: out-of-memory errors on small data
Hi Sim,
I think the right way to set the PermGen Size is through driver extra JVM
options, i.e.
--conf "spark.driver.extraJavaOptions=-XX:MaxPe
s> | blog.simeonov.com | 617.299.6746
>>
>>
>> From: Yin Huai
>> Date: Monday, July 6, 2015 at 12:59 AM
>> To: Simeon Simeonov
>> Cc: Denny Lee , Andy Huang <
>> andy.hu...@servian.com.au>, user
>>
>> Subject: Re: 1.4.0 regression:
4g
>
> /Sim
>
> Simeon Simeonov, Founder & CTO, Swoop <http://swoop.com/>
> @simeons <http://twitter.com/simeons> | blog.simeonov.com | 617.299.6746
>
>
> From: Yin Huai
> Date: Monday, July 6, 2015 at 12:59 AM
> To: Simeon Simeonov
> Cc:
12:59 AM
To: Simeon Simeonov mailto:s...@swoop.com>>
Cc: Denny Lee mailto:denny.g@gmail.com>>, Andy Huang
mailto:andy.hu...@servian.com.au>>, user
mailto:user@spark.apache.org>>
Subject: Re: 1.4.0 regression: out-of-memory errors on small data
I have never seen issue like thi
lt;http://swoop.com/>
> @simeons <http://twitter.com/simeons> | blog.simeonov.com | 617.299.6746
>
>
> From: Yin Huai
> Date: Sunday, July 5, 2015 at 11:04 PM
> To: Denny Lee
> Cc: Andy Huang , Simeon Simeonov ,
> user
> Subject: Re: 1.4.0 regression: out-of-m
to:yh...@databricks.com>>
Date: Sunday, July 5, 2015 at 11:04 PM
To: Denny Lee mailto:denny.g@gmail.com>>
Cc: Andy Huang mailto:andy.hu...@servian.com.au>>,
Simeon Simeonov mailto:s...@swoop.com>>, user
mailto:user@spark.apache.org>>
Subject: Re: 1.4.0 regression:
Sim,
Can you increase the PermGen size? Please let me know what is your setting
when the problem disappears.
Thanks,
Yin
On Sun, Jul 5, 2015 at 5:59 PM, Denny Lee wrote:
> I had run into the same problem where everything was working swimmingly
> with Spark 1.3.1. When I switched to Spark 1.4
I had run into the same problem where everything was working swimmingly
with Spark 1.3.1. When I switched to Spark 1.4, either by upgrading to
Java8 (from Java7) or by knocking up the PermGenSize had solved my issue.
HTH!
On Mon, Jul 6, 2015 at 8:31 AM Andy Huang wrote:
> We have hit the same
We have hit the same issue in spark shell when registering a temp table. We
observed it happening with those who had JDK 6. The problem went away after
installing jdk 8. This was only for the tutorial materials which was about
loading a parquet file.
Regards
Andy
On Sat, Jul 4, 2015 at 2:54 AM, s
@bipin, in my case the error happens immediately in a fresh shell in 1.4.0.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/1-4-0-regression-out-of-memory-errors-on-small-data-tp23595p23614.html
Sent from the Apache Spark User List mailing list archive at Na
I have a hunch I want to share: I feel that data is not being deallocated in
memory (at least like in 1.3). Once it goes in-memory it just stays there.
Spark SQL works fine, the same query when run on a new shell won't throw
that error, but when run on a shell which has been used for other queries
I will second this. I very rarely used to get out-of-memory errors in 1.3.
Now I get these errors all the time. I feel that I could work on 1.3
spark-shell for long periods of time without spark throwing that error,
whereas in 1.4 the shell needs to be restarted or gets killed frequently.
--
Vie
From: Yin Huai mailto:yh...@databricks.com>>
Date: Thursday, July 2, 2015 at 4:34 PM
To: Simeon Simeonov mailto:s...@swoop.com>>
Cc: user mailto:user@spark.apache.org>>
Subject: Re: 1.4.0 regression: out-of-memory errors on small data
Hi Sim,
Seems you already set the PermGe
Hi Sim,
Seems you already set the PermGen size to 256m, right? I notice that in
your the shell, you created a HiveContext (it further increased the memory
consumption on PermGen). But, spark shell has already created a HiveContext
for you (sqlContext. You can use asInstanceOf to access HiveContext
Hi Sim,
Spark 1.4.0's memory consumption on PermGen is higher then Spark 1.3
(explained in https://issues.apache.org/jira/browse/SPARK-8776). Can you
add --conf "spark.driver.extraJavaOptions=-XX:MaxPermSize=256m" in the
command you used to launch Spark shell? This will increase the PermGen size
f
I wonder if this could be a side effect of Spark-3928. Does ending the path
with *.parquet work?
Original message From: Exie
Date:06/30/2015 9:20 PM (GMT-05:00)
To: user@spark.apache.org Subject: 1.4.0
So I was delighted with Spark 1.3.1 using Parquet 1.6.0 which would
"par
19 matches
Mail list logo
