but in fact the directories are not ready at the beginning to my task .
for example:
/user/root/2015/05/11/data.txt
/user/root/2015/05/12/data.txt
/user/root/2015/05/13/data.txt
like this.
and one new directory one day.
how to create the new DStream for tomorrow’s new
directory(/user/root/2015/05/13/) ??
> 在 2015年5月13日，下午4:59，Ankur Chauhan <achau...@brightcove.com> 写道：
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I would suggest creating one DStream per directory and then using
> StreamingContext#union(...) to get a union DStream.
>
> - -- Ankur
>
> On 13/05/2015 00:53, hotdog wrote:
>> I want to use use fileStream in spark streaming to monitor multi
>> hdfs directories, such as:
>>
>> val list_join_action_stream = ssc.fileStream[LongWritable, Text,
>> TextInputFormat]("/user/root/*/*", check_valid_file(_),
>> false).map(_._2.toString).print
>>
>>
>> Buy the way, i could not under the meaning of the three class :
>> LongWritable, Text, TextInputFormat
>>
>> but it doesn't work...
>>
>>
>>
>> -- View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-monitor-mul
> ti-directories-in-spark-streaming-task-tp22863.html
>>
>>
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>>
>>
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
> -----BEGIN PGP SIGNATURE-----
>
> iQEcBAEBAgAGBQJVUxJ1AAoJEOSJAMhvLp3L2f4IAKK+ouQ2VD7H6s/5w/YGbt2P
> uBGJPQ92Hb5REq3f4gK4YecygtAlSAwsqXGCoAaaoPAC7vUMs9RM+slqse1gmUPU
> pbORTIB9dv3iVxjPtZ6R8EX14BAlxcIOR6ni2RBHuQTL+dgIEUekmCg0IhFa5lVF
> Kt5in8rY5PSnX5l/dX9Yu8LI3uC4TLQ+eJXjjOGXoCHys+SaZWJckA3gVcF9GQdB
> dwdhv4UCIYVFj3QIVlLf0+B8FgA0DnRfBC+5ZfS88gcWMc4065sDdx5LkySy4oZB
> tB8IpC4yaY3Mqiu8jdvhcw+SevlYan5YkkkutSvKH7nL/0d1WIkEkHxPBjRqAmY=
> =U0oQ
> -----END PGP SIGNATURE-----
> [attachment]
>
> 0x6D461C4A.asc
> download: http://u.163.com/t0/fqZhSPbA
>
> preview: http://u.163.com/t0/2LRiaRy
>
>
> 0x6D461C4A.asc.sig
> download: http://u.163.com/t0/Ij1N9
>
> <0x6D461C4A.asc><0x6D461C4A.asc.sig>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
