Re: IllegalArgumentException: Size exceeds Integer.MAX_VALUE

2015-10-05 Thread Jegan
S3? I am not sure I can do this now either. Please clarify me if I misunderstood what you meant. Thanks, Jegan On Mon, Oct 5, 2015 at 4:42 PM, Reynold Xin wrote: > You can write the data to local hdfs (or local disk) and just load it from > there. > > > On Mon, Oct 5, 2015 at 4

Re: IllegalArgumentException: Size exceeds Integer.MAX_VALUE

2015-10-05 Thread Jegan
-around? Thanks again. Regards, Jegan On Mon, Oct 5, 2015 at 3:53 PM, Ted Yu wrote: > As a workaround, can you set the number of partitions higher in the > sc.textFile method ? > > Cheers > > On Mon, Oct 5, 2015 at 3:31 PM, Jegan wrote: > >> Hi All, >> >> I

IllegalArgumentException: Size exceeds Integer.MAX_VALUE

2015-10-05 Thread Jegan
ethods } Please let me know if there is any other work-around for the same. Thanks for your time. Regards, Jegan