Re: spark streaming doesn't pick new files from HDFS

2016-03-09 Thread srimugunthan dhandapani
I doubt if thats the problem. Thats how hdfs lists a directory. Output of few more commands below. *$ hadoop fs -ls /tmp/* Found 7 items drwxrwxrwx - hdfs supergroup 0 2016-03-10 11:09 /tmp/.cloudera_health_monitoring_canary_files -rw-r--r-- 3 ndsuser1 supergroup 447873024 2016-

Re: spark streaming doesn't pick new files from HDFS

2016-03-09 Thread Ted Yu
bq. drwxr-xr-x - tomcat7 supergroup 0 2016-03-09 23:16 /tmp/swg If I read the above line correctly, the size of the file was 0. On Wed, Mar 9, 2016 at 10:00 AM, srimugunthan dhandapani < srimugunthan.dhandap...@gmail.com> wrote: > Hi all > I am working in cloudera CDH5.6 and version

spark streaming doesn't pick new files from HDFS

2016-03-09 Thread srimugunthan dhandapani
Hi all I am working in cloudera CDH5.6 and version of spark is 1.5.0-cdh5.6.0 I have a strange problem that spark streaming works on a directory in local filesystem but doesnt work for hdfs. My spark streaming program: package com.oreilly.learningsparkexamples.java; import java.util.concurren