Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
From: Khaled Hammouda [mailto:khaled.hammo...@kik.com]
Sent: Thursday, June 16, 2016 11:45 AM
To: Mohammed Guller
Cc: user
Subject: Re: Spark SQL driver memory keeps rising
I'm using pyspark and running in YARN client mode. I managed t
d GC
> pause. This should not happen on the machine running the driver program if
> all that you are doing is reading data from HDFS, perform a bunch of
> transformations and write result back into HDFS.
>
>
>
> Perhaps, the program is not actually using Spark in cluster mode, bu
d
>
> Author: Big Data Analytics with Spark
> <http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
>
>
>
> *From:* Khaled Hammouda [mailto:khaled.hammo...@kik.com]
> *Sent:* Tuesday, June 14, 2016 10:23 PM
> *To:* user
> *Subject:* Spark SQL driver memo
mmouda [mailto:khaled.hammo...@kik.com]
Sent: Tuesday, June 14, 2016 10:23 PM
To: user
Subject: Spark SQL driver memory keeps rising
I'm having trouble with a Spark SQL job in which I run a series of SQL
transformations on data loaded from HDFS.
The first two stages load data from hdfs input wit
I'm having trouble with a Spark SQL job in which I run a series of SQL
transformations on data loaded from HDFS.
The first two stages load data from hdfs input without issues, but later
stages that require shuffles cause the driver memory to keep rising until
it is exhausted, and then the driver s
ent_ip = None
98 self._client_port = None
TypeError: 'JavaPackage' object is not callable"
What does it mean?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/SQL-Driver-tp26800.html
Sent from the Apache S
