yes, Spark needs to create the RDD first(loads all the data) to create the
sample. You can split the files into two sets outside of spark in order to
load only the sample set. Thank youDhiraj
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Behaviour-of-RDD-
Spark will load the whole dataset.
The sampling action can be viewed as an filter. The real implementation can
be more complicate, but I give you the idea by simple implementation.
val rand = new Random();
val subRdd = rdd.filter(x => rand.nextDouble() <= 0.3)
To prevent recomputing data, you c
I would assume that the driver has to count the number of lines in the json
file anyway.
Otherwise, how could it tell the workers which lines they should work on?
2016-05-31 10:03 GMT+02:00 Gavin Yue :
> If not reading the whole dataset, how do you know the total number of
> records? If not kno
If not reading the whole dataset, how do you know the total number of records?
If not knowing total number, how do you choose 30%?
> On May 31, 2016, at 00:45, pbaier wrote:
>
> Hi all,
>
> I have to following use case:
> I have around 10k of jsons that I want to use for learning.
> The json
