You will have to write your customReceiver to do that. I don’t think wholeTextFile is designed for that.
- Harjit > On Apr 26, 2016, at 7:19 PM, Mail.com <[email protected]> wrote: > > > Hi All, > I am reading entire directory of gz XML files with wholeTextFiles. > > I understand as it is gz and with wholeTextFiles the individual files are not > splittable but why the entire directory is read by one executor, single task. > I have provided number of executors as number of files in that directory. > > Is the only option here is to repartition after the xmls are read and parsed > with JaxB. > > Regards, > Pradeep > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > v/r, Harjit Singh Decipher Technology Studios email:[email protected] <mailto:email%[email protected]> mobile: 303- <tel:703-907-9311>870-0883 website: deciphertechstudios.com <http://deciphertechstudios.com/> <http://deciphertechstudios.com/ <http://deciphertechstudios.com/>> GPG: keyserver: hkps://hkps.pool.sks-keyservers.net <http://hkps.pool.sks-keyservers.net/> keyid: D814A2EF
signature.asc
Description: Message signed with OpenPGP using GPGMail
