hey u can use repartition and set it to 1
as in this example
unionDStream.foreachRDD((rdd, time) => {
val count = rdd.count()
println("count" + count)
if (count > 0) {
print("rdd partition=" + rdd.partitions.length)
val outputRDD = rdd.repartition(numFilesPerPartition)
outputRDD.saveAsTextFile(
outputDirectory + "/" + rdd.id)
}
}
})
On Tue, Apr 12, 2016 at 3:59 PM, Priya Ch <[email protected]>
wrote:
> Hi All,
>
> I am working with Kafka, Spark Streaming and I want to write the
> streaming output to a single file. dstream.saveAsTexFiles() is creating
> files in different folders. Is there a way to write to a single folder ? or
> else if written to different folders, how do I merge them ?
> Thanks,
> Padma Ch
>
--
Thanks & Regards
Sachin Aggarwal
7760502772
