Yup, i meant union only.
On Wed, Jan 7, 2015, 16:19 Sean Owen wrote:
> I think you mean union(). Yes, you could also simply make an RDD for each
> file, and use SparkContext.union() to put them together.
>
> On Wed, Jan 7, 2015 at 9:51 AM, Raghavendra Pandey <
> raghavendra.pan...@gmail.com> wro
I think you mean union(). Yes, you could also simply make an RDD for each
file, and use SparkContext.union() to put them together.
On Wed, Jan 7, 2015 at 9:51 AM, Raghavendra Pandey <
raghavendra.pan...@gmail.com> wrote:
> You can also use join function of rdd. This is actually kind of append
> f
You can also use join function of rdd. This is actually kind of append
funtion that add up all the rdds and create one uber rdd.
On Wed, Jan 7, 2015, 14:30 rkgurram wrote:
> Thank you for the response, sure will try that out.
>
> Currently I changed my code such that the first map "files.map" to
Thank you for the response, sure will try that out.
Currently I changed my code such that the first map "files.map" to
"files.flatMap", which I guess will do similar what you are saying, it gives
me a List[] of elements (in this case LabeledPoints, I could also do RDDs)
which I then turned into a
an RDD cannot contain elements of type RDD. (i.e. you can't nest RDDs within
RDDs, in fact, I don't think it makes any sense)
I suggest rather than having an RDD of file names, collect those file name
strings back on to the driver as a Scala array of file names, and then from
there, make an array
