If you are running in the cluster mode, that zip file should exist in all the nodes! Is that the case?
HTH Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Wed, 9 Aug 2023 at 13:41, lnxpgn <lnx...@gmail.com> wrote: > Hi, > > I am using Spark 3.4.1, running on YARN. Hadoop runs on a single-node in > a pseudo-distributed mode. > > spark-submit --master yarn --deploy-mode cluster --py-files > /tmp/app-submodules.zip app.py > > The YARN application ran successfully, but have a warning log message: > > /opt/hadoop-tmp-dir/nm-local-dir/usercache/bigdata/appcache/application_1691548913900_0002/container_1691548913900_0002_01_000001/pyspark.zip/pyspark/context.py:350: > > RuntimeWarning: Failed to add file [file:///tmp/app-submodules.zip] > specified in 'spark.submit.pyFiles' to Python path: > > If I use HDFS file: > > spark-submit --master yarn --deploy-mode cluster --py-files > hdfs://hadoop-namenode:9000/tmp/app-submodules.zip app.py > > the warning message looks like this: > > /opt/hadoop-tmp-dir/nm-local-dir/usercache/bigdata/appcache/application_1691548913900_0002/container_1691548913900_0002_01_000001/pyspark.zip/pyspark/context.py:350: > > RuntimeWarning: Failed to add file > [hdfs://hadoop-namenode:9000/app-submodules.zip] specified in > 'spark.submit.pyFiles' to Python path: > > The part code of context.py: > > filepath = os.path.join(SparkFiles.getRootDirectory(), filename) > if not os.path.exists(filepath): > shutil.copyfile(path, filepath) > > Look like the submitted Python file has 'file:', 'hdfs:' URI schemes, > shutil.copyfile treats them as part of the file name. > > I searched, but didn't find useful information, didn't know why, this is > a bug or I did something wrong? > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
