Hey Amnon,
So just to make sure I understand - you also saw the same issue with 1.0.2?
Just asking because whether or not this regresses the 1.0.2 behavior is
important for our own bug tracking.
- Patrick
On Mon, Aug 25, 2014 at 10:22 PM, Amnon Khen wrote:
> There were no failures nor excepti
There were no failures nor exceptions.
On Tue, Aug 26, 2014 at 1:31 AM, Matei Zaharia
wrote:
> Got it. Another thing that would help is if you spot any exceptions or
> failed tasks in the web UI (http://:4040).
>
> Matei
>
> On August 25, 2014 at 3:07:41 PM, amnonkhen (amnon...@gmail.com) wrote
Got it. Another thing that would help is if you spot any exceptions or failed
tasks in the web UI (http://:4040).
Matei
On August 25, 2014 at 3:07:41 PM, amnonkhen (amnon...@gmail.com) wrote:
Hi Matei,
The original issue happened on a spark-1.0.2-bin-hadoop2 installation.
I will try the synth
Hi Matei,
The original issue happened on a spark-1.0.2-bin-hadoop2 installation.
I will try the synthetic operation and see if I get the same results or not.
Amnon
On Mon, Aug 25, 2014 at 11:26 PM, Matei Zaharia [via Apache Spark
Developers List] wrote:
> Was the original issue with Spark 1.1 (
Hi Patrick,
Here's the process:
java -cp
/root/ephemeral-hdfs/conf/root/ephemeral-hdfs/conf:/root/spark/conf:/root/spark/assembly/target/scala-2.10/spark-assembly-1.1.1-SNAPSHOT-hadoop1.0.4.jar
-XX:MaxPermSize=128m -Djava.library.path=/root/ephemeral-hdfs/lib/native/
-Xms5g -Xmx10g -XX:MaxPermS
Hi Matei,
At least in my case, the s3 bucket is in the same region. Running count()
works and so does generating synthetic data. What I saw was that the job
would hang for over an hour with no progress but tasks would immediately
start finishing if I cached the data.
- jerry
On Mon, Aug 25, 2014
One other idea - when things freeze up, try to run jstack on the spark
shell process and on the executors and attach the results. It could be that
somehow you are encountering a deadlock somewhere.
On Mon, Aug 25, 2014 at 1:26 PM, Matei Zaharia
wrote:
> Was the original issue with Spark 1.1 (i.
Was the original issue with Spark 1.1 (i.e. master branch) or an earlier
release?
One possibility is that your S3 bucket is in a remote Amazon region, which
would make it very slow. In my experience though saveAsTextFile has worked even
for pretty large datasets in that situation, so maybe ther
Hi jerryye,
Maybe if you voted up my question on Stack Overflow it would get some
traction and we would get nearer to a solution.
Thanks,
Amnon
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/saveAsTextFile-to-s3-on-spark-does-not-work-just-hangs-tp7
bump.
I'm seeing the same issue with branch-1.1. Caching the RDD before running
saveAsTextFile gets things running but the job stalls 2/3 of the way by
using too much memory.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/saveAsTextFile-to-s3-on-spar
10 matches
Mail list logo
