A way to share RDD directly using Tachyon?

But get a RDD directly from tachyon instead of a file can sometimes avoid parsing the same file repeatedly in different Apps, I think. What am I supposed to do in order to share RDDs to get a better performance? —  Best Regards! Yijie Shen

Re: Read parquet folders recursively

if (baseStatus.isDir) recurse(basePath) else Array(baseStatus)   } —  Best Regards! Yijie Shen On March 12, 2015 at 2:35:49 PM, Akhil Das (ak...@sigmoidanalytics.com) wrote: Hi We have a custom build to read directories recursively, Currently we use it with fileStream like: val lines = ssc.fileStr

Spark SQL saveAsParquet failed after a few waves

d.run(Thread.java:745) I have no idea what happened since jobs may fail or success without any reason. Thanks. Yijie Shen

Re: Spark SQL saveAsParquet failed after a few waves

! Yijie Shen On April 2, 2015 at 2:05:31 AM, Michael Armbrust (mich...@databricks.com) wrote: When few waves (1 or 2) are used in a job, LoadApp could finish after a few failures and retries. But when more waves (3) are involved in a job, the job would terminate abnormally. Can you clarify what

Submit & Kill Spark Application program programmatically from another application

ssBuilder` and query status about the query? Any Suggestions? —  Best Regards! Yijie Shen