Re: JetS3T settings spark

2014-12-30 Thread durga katakam
Thanks Matei. -D On Tue, Dec 30, 2014 at 4:49 PM, Matei Zaharia wrote: > This file needs to be on your CLASSPATH actually, not just in a directory. > The best way to pass it in is probably to package it into your application > JAR. You can put it in src/main/resources in a Maven or SBT project,

Re: S3 files , Spark job hungsup

2014-12-22 Thread durga katakam
Yes . I am reading thousands of files every hours. Is there any way I can tell spark to timeout. Thanks for your help. -D On Mon, Dec 22, 2014 at 4:57 AM, Shuai Zheng wrote: > Is it possible too many connections open to read from s3 from one node? I > have this issue before because I open a few

Re: S3 globbing

2014-12-17 Thread durga katakam
Hi Akhil, Thanks for your time. I appreciate .I tried this approach , but either I am getting less files or more files not exact hour files. Is there any way I can tell the range (between this time to this time) Thanks, D On Tue, Dec 16, 2014 at 11:04 PM, Akhil Das wrote: > > Did you try somet