@spark.apache.org
Subject: Re: hdfs streaming context
Yes but you can't follow three slashes with host:port. No host probably
defaults to whatever is found in your HDFS config.
On Mon, Dec 1, 2014 at 11:02 PM, Bui, Tri wrote:
> For the streaming example I am working on, Its accepted ("
Yes but you can't follow three slashes with host:port. No host
probably defaults to whatever is found in your HDFS config.
On Mon, Dec 1, 2014 at 11:02 PM, Bui, Tri wrote:
> For the streaming example I am working on, Its accepted ("hdfs:///user/data")
> without the localhost info.
>
> Let me dig
t
Cc: user@spark.apache.org
Subject: Re: hdfs streaming context
Yes, in fact, that's the only way it works. You need
"hdfs://localhost:8020/user/data", I believe.
(No it's not correct to write "hdfs:///...")
On Mon, Dec 1, 2014 at 10:41 PM, Benjamin Cuthbert
wrote:
>
Thanks Sean,
That worked just removing the /* and leaving it as /user/data
Seems to be streaming in.
> On 1 Dec 2014, at 22:50, Sean Owen wrote:
>
> Yes, in fact, that's the only way it works. You need
> "hdfs://localhost:8020/user/data", I believe.
>
> (No it's not correct to write "hdfs://
Yes, in fact, that's the only way it works. You need
"hdfs://localhost:8020/user/data", I believe.
(No it's not correct to write "hdfs:///...")
On Mon, Dec 1, 2014 at 10:41 PM, Benjamin Cuthbert
wrote:
> All,
>
> Is it possible to stream on HDFS directory and listen for multiple files?
>
> I hav
Have you tried just passing a path to ssc.textFileStream() ? It
monitors the path for new files by looking at mtime/atime ; all
new/touched files in the time window appear as an rdd in the dstream.
On 1 December 2014 at 14:41, Benjamin Cuthbert wrote:
> All,
>
> Is it possible to stream on HDFS d
Try
("hdfs:///localhost:8020/user/data/*")
With 3 "/".
Thx
tri
-Original Message-
From: Benjamin Cuthbert [mailto:cuthbert@gmail.com]
Sent: Monday, December 01, 2014 4:41 PM
To: user@spark.apache.org
Subject: hdfs streaming context
All,
Is it possible to stream on HDFS director
