SPARK STREAMING PROBLEM

[Animesh Baranawal]

Hi,

I am trying to extract the filenames from which a Dstream is generated by
parsing the toDebugString method on RDD
I am implementing the following code in spark-shell:

import org.apache.spark.streaming.{StreamingContext, Seconds}
val ssc = new StreamingContext(sc,Seconds(10))
val lines = ssc.textFileStream(// directory //)

def g : List[String] = {
   var res = List[String]()
   lines.foreachRDD{ rdd => {
      if(rdd.count > 0){
      val files = rdd.toDebugString.split("\n").filter(_.contains(":\"))
      files.foreach{ ms => {
         res = ms.split(" ")(2)::res
      }}   }
   }}
   res
}

g.foreach(x => {println(x); println("************")})

However when I run the code, nothing gets printed on the console apart from
the logs. Am I doing something wrong?
And is there any better way to extract the file names from DStream ?

Thanks in advance


Animesh