Re: Need Streaming output to single HDFS File

[Sachin Aggarwal]

hey u can use repartition and set it to 1
as in this example

unionDStream.foreachRDD((rdd, time) => {
      val count = rdd.count()
      println("count" + count)
      if (count > 0) {
        print("rdd partition=" + rdd.partitions.length)
        val outputRDD = rdd.repartition(numFilesPerPartition)
        outputRDD.saveAsTextFile(
          outputDirectory + "/" + rdd.id)
        }
      }
    })

On Tue, Apr 12, 2016 at 3:59 PM, Priya Ch <learnings.chitt...@gmail.com>
wrote:

> Hi All,
>
>   I am working with Kafka, Spark Streaming and I want to write the
> streaming output to a single file. dstream.saveAsTexFiles() is creating
> files in different folders. Is there a way to write to a single folder ? or
> else if written to different folders, how do I merge them ?
> Thanks,
> Padma Ch
>



-- 

Thanks & Regards

Sachin Aggarwal
7760502772