I think spark checks the python path env variable. Need to provide that. Of course that works in local mode only
On Fri, Jan 8, 2021, 5:28 PM Sean Owen <sro...@gmail.com> wrote: > I don't see anywhere that you provide 'sparkstuff'? how would the Spark > app have this code otherwise? > > On Fri, Jan 8, 2021 at 10:20 AM Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Thanks Riccardo. >> >> I am well aware of the submission form >> >> However, my question relates to doing submission within PyCharm itself. >> >> This is what I do at Pycharm *terminal* to invoke the module python >> >> spark-submit --jars >> ..\lib\spark-bigquery-with-dependencies_2.12-0.18.0.jar \ >> --packages com.github.samelamin:spark-bigquery_2.11:0.2.6 >> analyze_house_prices_GCP.py >> >> However, at terminal run it does not pickup import dependencies in the >> code! >> >> Traceback (most recent call last): >> File >> "C:/Users/admin/PycharmProjects/pythonProject2/DS/src/analyze_house_prices_GCP.py", >> line 8, in <module> >> import sparkstuff as s >> ModuleNotFoundError: No module named 'sparkstuff' >> >> The python code is attached, pretty simple >> >> Thanks >> >> >> >>
