Hi,
I am trying simple file streaming example using
Sparkstreaming(spark-streaming_2.10,version:1.5.1)
public class DStreamExample {
public static void main(final String[] args) {
final SparkConf sparkConf = new SparkConf();
sparkConf.setAppName("SparkJob");
sparkConf.setMaster("local[4]"); // for local
final JavaSparkContext sc = new JavaSparkContext(sparkConf);
final JavaStreamingContext ssc = new JavaStreamingContext(sc,
new Duration(2000));
final JavaDStream<String> lines = ssc.textFileStream("/opt/test/");
lines.print();
ssc.start();
ssc.awaitTermination();
}
}
When I run this code on single file or director it does not print anything
from file, I see in logs its constantly polling but nothing is printed. I
tried moving file to directory when this program was running.
Is there something I am missing? I tried applying map function on lines RDD
that also does not work.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spak-filestreaming-issue-tp25380.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
