Thanks everyone. I was able to resolve this. Here is what I did. Just passed conf file using —files option.
Mistake that I did was reading the json conf file before creating spark session . Reading if after creating spark session helped it. Thanks once again for your valuable suggestions Thanks, Asmath > On May 15, 2021, at 8:12 AM, Sean Owen <sro...@gmail.com> wrote: > > > If code running on the executors need some local file like a config file, > then it does have to be passed this way. That much is normal. > >> On Sat, May 15, 2021 at 1:41 AM Gourav Sengupta <gourav.sengu...@gmail.com> >> wrote: >> Hi, >> >> once again lets start with the requirement. Why are you trying to pass xml >> and json files to SPARK instead of reading them in SPARK? >> Generally when people pass on files they are python or jar files. >> >> Regards, >> Gourav
