I’m running Spark 1.6.0 in a standalone cluster. Periodically I’ve seen
StackOverflowErrors when running queries. An example below.
In the past I’ve been able to avoid such situations by ensuring we don’t have
too many arguments in ‘in’ clauses or too many unioned queries both of which
seem to t
I’ve seen this when I specified “too many” where clauses in the SQL query. I
was able to adjust my query to use a single ‘in’ clause rather than many ‘=’
clauses but I realize that may not be an option in all cases.
Jeff
On 5/4/16, 2:04 PM, "BenD" wrote:
>I am getting a java.lang.StackOverflo
I upgraded our Spark standalone cluster from 1.4.1 to 1.6.0 yesterday. We are
now seeing regular timeouts between two of the workers when making connections.
These workers and the same driver code worked fine running on 1.4.1 and
finished in under a second. Any thoughts on what might have change
I’ve written an application that hosts the Spark driver in-process using
“local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve also
tried putting the following code prior to creating my SparkContext. These were
coupled together from various posts I’ve. None of these steps ha
:34 PM
To: Jeff Jones
Cc: "user@spark.apache.org<mailto:user@spark.apache.org>"
Subject: Re: How can I disable logging when running local[*]?
Did you try “--driver-java-options
'-Dlog4j.configuration=file:/'” and setting the
log4j.rootLogger=FATAL,console?
On Mon, Oc
ty, because it's a bit verbose
log4j.logger.org.eclipse.jetty=WARN
spark.log.threshold=OFF
spark.root.logger=OFF,DRFA
From: Alex Kozlov
Date: Tuesday, October 6, 2015 at 10:50 AM
To: Jeff Jones
Cc: "user@spark.apache.org<mailto:user@spark.apache.org>"
Subject: Re: Ho
I’ve got an a series of applications using a single standalone Spark cluster
(v1.4.1). The cluster has 1 master and 4 workers (4 CPUs per worker node). I
am using the start-slave.sh script to launch the worker process on each node
and I can see the nodes were successfully registered using the S
Regards
>JB
>
>On 11/02/2015 08:56 PM, Jeff Jones wrote:
>> I’ve got an a series of applications using a single standalone Spark
>> cluster (v1.4.1). The cluster has 1 master and 4 workers (4 CPUs per
>> worker node). I am using the start-slave.sh script to launch
I wrote a very simple Spark 1.4.1 app that I can run through a local driver
program just fine using setMaster("local[*]"). The app is as follows:
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
object
, 2015 11:22 PM
To: Jeff Jones
Cc: user@spark.apache.org
Subject: Re: All masters are unresponsive! Giving up.
There seems to be a version mismatch somewhere. You can try and find out the
cause with debug serialization information. I think the jvm flag
-Dsun.io.serialization.extendedDebugInfo=true
I've got a Spark application running on a host with > 64 character FQDN. When
running with Spark master "local[*]" I get the following error. Note, the host
name should be
ip-10-248-0-177.us-west-2.compute.internaldna.corp.adaptivebiotech.com but the
last 6 characters are missing. The same ap
We are using Scala 2.11 for a driver program that is running Spark SQL queries
in a standalone cluster. I’ve rebuilt Spark for Scala 2.11 using the
instructions at http://spark.apache.org/docs/latest/building-spark.html. I’ve
had to work through a few dependency conflict but all-in-all it seems
jars list. Unfortunately the actual error got masked
by the one I sent below.
Jeff
From: Shixiong Zhu
Date: Sunday, September 6, 2015 at 9:02 AM
To: Jeff Jones
Cc: "user@spark.apache.org<mailto:user@spark.apache.org>"
Subject: Re: ClassCastException in driver program
Looks there a
I’m trying to perform a Spark SQL (1.5) query containing a UDF in the select
and group by clauses. From what I’ve been able to find this should be
supported. A few examples include
https://github.com/spirom/LearningSpark/blob/master/src/main/scala/sql/UDF.scala,
https://issues.apache.org/jira/
14 matches
Mail list logo