Re: Recursive nested wildcard directory walking in Spark

2015-12-09 Thread James Ding
; Subject: Re: Recursive nested wildcard directory walking in Spark Have you seen this thread ? http://search-hadoop.com/m/q3RTt2uhMX1UhnCc1&subj=Re+Does+sc+newAPIHadoopFil e+support+multiple+directories+or+nested+directories+ <https://urldefense.proofpoint.com/v2/url?u=http-3A__search

Re: Recursive nested wildcard directory walking in Spark

2015-12-09 Thread Ted Yu
Have you seen this thread ? http://search-hadoop.com/m/q3RTt2uhMX1UhnCc1&subj=Re+Does+sc+newAPIHadoopFile+support+multiple+directories+or+nested+directories+ FYI On Wed, Dec 9, 2015 at 11:18 AM, James Ding wrote: > Hi! > > My name is James, and I’m working on a question there doesn’t seem to b

Recursive nested wildcard directory walking in Spark

2015-12-09 Thread James Ding
Hi! My name is James, and I’m working on a question there doesn’t seem to be a lot of answers about online. I was hoping spark/hadoop gurus could shed some light on this. I have a data feed on NFS that looks like /foobar/.gz Currently I have a spark scala job that calls sparkContext.textFile(