Re: Is there a way to delete hdfs file/directory using spark API?

[Akhil Das]

There is no direct way of doing it, but you can do something like this:

val hadoopConf = ssc.sparkContext.hadoopConfiguration

var hdfs = org.apache.hadoop.fs.FileSystem.get(hadoopConf)


tmp_stream = ssc.textFileStream("/akhld/sigmoid/") // each line will have
hdfs location to be deleted.


tmp_stream.foreachRDD(path => {


    try {

      hdfs.delete(new org.apache.hadoop.fs.Path(path), true)

    } catch{ case e: Exception =>

      println("w00t!! Exception!!HDFS => " + e)

    }



  })


Thanks
Best Regards

On Thu, Jan 22, 2015 at 12:15 PM, LinQili <lin_q...@outlook.com> wrote:

> Hi, all
> I wonder how to delete hdfs file/directory using spark API?
>