,
Andrew Ash ,
"dev@spark.apache.org"
Subject
Re: Tachyo
IBM]
>
> BLD 28,ZGC Software Park
> No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193
> China
>
>
>
>
>
> *Reynold Xin >*
>
> 2014/12/12 10:22
> To
> Andrew Ash ,
> cc
> Jun Feng Liu/China/IBM@IBMCN, "dev@spark.apache.org" >
&
hina
Reynold Xin
2014/12/12 10:22
To
Andrew Ash ,
cc
Jun Feng Liu/China/IBM@IBMCN, "dev@spark.apache.org"
Subject
Re: Tachyon in Spark
Actually HY emailed me offline about this and this is supported in the
latest version of Tachyon. It is a hard problem to push this into storage;
Actually HY emailed me offline about this and this is supported in the
latest version of Tachyon. It is a hard problem to push this into storage;
need to think about how to handle isolation, resource allocation, etc.
https://github.com/amplab/tachyon/blob/master/core/src/main/java/tachyon/master/D
I don't think the lineage thing is even turned on in Tachyon - it was
mostly a research prototype, so I don't think it'd make sense for us to use
that.
On Thu, Dec 11, 2014 at 3:51 PM, Andrew Ash wrote:
> I'm interested in understanding this as well. One of the main ways Tachyon
> is supposed
I'm interested in understanding this as well. One of the main ways Tachyon
is supposed to realize performance gains without sacrificing durability is
by storing the lineage of data rather than full copies of it (similar to
Spark). But if Spark isn't sending lineage information into Tachyon, then
Dose Spark today really leverage Tachyon linage to process data? It seems
like the application should call createDependency function in TachyonFS to
create a new linage node. But I did not find any place call that in Spark
code. Did I missed anything?
Best Regards
Jun Feng Liu
IBM China System
