spark.speculation was not set, any speculative execution on tachyon side?
tachyon-env.sh only changed following
export TACHYON_MASTER_ADDRESS=test01.zala
#export TACHYON_UNDERFS_ADDRESS=$TACHYON_HOME/underfs
export TACHYON_UNDERFS_ADDRESS=hdfs://test01.zala:8020
export TACHYON_WORKER_MEMORY_SIZE=16GB
test01.zala is master node for HDFS, tachyon, Spark, etc.
worker nodes are
test02.zala
test03.zala
test04.zala
spark-shell run on test02
after parquetFile.saveAsParquetFile("tachyon://test01.zala:19998/parquet_1")
i got FailedToCheckpointException with Failed to rename
but tfs lsr, there are some temporary files and metadata file
0.00 B 08-11-2014 16:19:28:054 /parquet_1
881.00 B 08-11-2014 16:19:28:054 In Memory /parquet_1/_metadata
0.00 B 08-11-2014 16:19:28:314 /parquet_1/_temporary
0.00 B 08-11-2014 16:19:28:314 /parquet_1/_temporary/0
0.00 B 08-11-2014 16:19:28:931
/parquet_1/_temporary/0/_temporary
0.00 B 08-11-2014 16:19:28:931
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_33
0.00 B 08-11-2014 16:19:28:931 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_33/part-r-1.parquet
0.00 B 08-11-2014 16:19:28:940
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_35
0.00 B 08-11-2014 16:19:28:940 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_35/part-r-2.parquet
0.00 B 08-11-2014 16:19:28:962
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_36
0.00 B 08-11-2014 16:19:28:962 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_36/part-r-4.parquet
0.00 B 08-11-2014 16:19:28:971
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_34
0.00 B 08-11-2014 16:19:28:971 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_34/part-r-3.parquet
0.00 B 08-11-2014 16:20:06:349
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_37
0.00 B 08-11-2014 16:20:06:349 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_37/part-r-2.parquet
0.00 B 08-11-2014 16:20:09:519
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_38
0.00 B 08-11-2014 16:20:09:519 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_38/part-r-4.parquet
0.00 B 08-11-2014 16:20:18:777
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_39
0.00 B 08-11-2014 16:20:18:777 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_39/part-r-3.parquet
0.00 B 08-11-2014 16:20:28:315
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_40
0.00 B 08-11-2014 16:20:28:315 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_40/part-r-1.parquet
0.00 B 08-11-2014 16:20:38:382
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_41
0.00 B 08-11-2014 16:20:38:382 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_41/part-r-2.parquet
0.00 B 08-11-2014 16:20:40:681
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_42
0.00 B 08-11-2014 16:20:40:681 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_42/part-r-4.parquet
0.00 B 08-11-2014 16:20:50:376
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_43
0.00 B 08-11-2014 16:20:50:376 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_43/part-r-3.parquet
0.00 B 08-11-2014 16:21:00:932
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_44
0.00 B 08-11-2014 16:21:00:932 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_44/part-r-1.parquet
0.00 B 08-11-2014 16:21:10:355
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_45
0.00 B 08-11-2014 16:21:10:355 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000001_45/part-r-2.parquet
0.00 B 08-11-2014 16:21:11:468
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_46
0.00 B 08-11-2014 16:21:11:468 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000003_46/part-r-4.parquet
0.00 B 08-11-2014 16:21:21:681
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_47
0.00 B 08-11-2014 16:21:21:681 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000002_47/part-r-3.parquet
0.00 B 08-11-2014 16:21:32:583
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_48
0.00 B 08-11-2014 16:21:32:583 In Memory
/parquet_1/_temporary/0/_temporary/attempt_201408111619_0017_r_000000_48/part-r-1.parquet
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/share-reuse-off-heap-persisted-tachyon-RDD-in-SparkContext-or-saveAsParquetFile-on-tachyon-in-SQLCont-tp11897p11977.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
