I see, thanks a lot On Mon, Nov 16, 2015 at 6:29 PM, Bryan Cutler <cutl...@gmail.com> wrote:
> Hi Renyi, > > This is the intended behavior of the streaming HdfsWordCount example. It > makes use of a 'textFileStream' which will monitor a hdfs directory for any > newly created files and push them into a dstream. It is meant to be run > indefinitely, unless interrupted by ctrl-c, for example. > > -bryan > On Nov 13, 2015 10:52 AM, "Renyi Xiong" <renyixio...@gmail.com> wrote: > >> Hi, >> >> I try to run the following 1.4.1 sample by putting a words.txt under >> localdir >> >> bin\run-example org.apache.spark.examples.streaming.HdfsWordCount localdir >> >> 2 questions >> >> 1. it does not pick up words.txt because it's 'old' I guess - any option >> to let it picked up? >> 2. I managed to put a 'new' file on the fly which got picked up, but >> after processing, the program doesn't stop (keeps generating empty RDDs >> instead), any option to let it stop when no new files come in (otherwise it >> blocks others when I want to run multiple samples?) >> >> Thanks, >> Renyi. >> >
