Re: spark streaming doesn't pick new files from HDFS

2016-03-09 Thread srimugunthan dhandapani
I doubt if thats the problem. Thats how hdfs lists a directory. Output of few more commands below. *$ hadoop fs -ls /tmp/* Found 7 items drwxrwxrwx - hdfs supergroup 0 2016-03-10 11:09 /tmp/.cloudera_health_monitoring_canary_files -rw-r--r-- 3 ndsuser1 supergroup 447873024 2016-

Re: spark streaming doesn't pick new files from HDFS

2016-03-09 Thread Ted Yu
bq. drwxr-xr-x - tomcat7 supergroup 0 2016-03-09 23:16 /tmp/swg If I read the above line correctly, the size of the file was 0. On Wed, Mar 9, 2016 at 10:00 AM, srimugunthan dhandapani < srimugunthan.dhandap...@gmail.com> wrote: > Hi all > I am working in cloudera CDH5.6 and version