I moved them every interval to the monitored directory.
Patcharee
On 25. jan. 2016 22:30, Shixiong(Ryan) Zhu wrote:
Did you move the file into "hdfs://helmhdfs/user/patcharee/cerdata/",
or write into it directly? `textFileStream` requires that files must
be written to the monitored directory b
Any possibility that this file is still written by other application, so
what Spark Streaming processed is an incomplete file.
On Tue, Jan 26, 2016 at 5:30 AM, Shixiong(Ryan) Zhu wrote:
> Did you move the file into "hdfs://helmhdfs/user/patcharee/cerdata/", or
> write into it directly? `textFile
Did you move the file into "hdfs://helmhdfs/user/patcharee/cerdata/", or
write into it directly? `textFileStream` requires that files must be
written to the monitored directory by "moving" them from another location
within the same file system.
On Mon, Jan 25, 2016 at 6:30 AM, patcharee
wrote:
>
Hi,
My streaming application is receiving data from file system and just
prints the input count every 1 sec interval, as the code below:
val sparkConf = new SparkConf()
val ssc = new StreamingContext(sparkConf, Milliseconds(interval_ms))
val lines = ssc.textFileStream(args(0))
lines.count().pr
