I've hardly used Jupyter so can't comment on an equivalent for %run. But for Zeppelin you can put your python files on the local file system of your Spark driver node, or more commonly in HDFS, and then use sc.addPyFile() [1] to make each file available in the SparkContext. Then you can import your python packages as normal. The slightly annoying thing is that if you change your code you'll need to restart your Spark application to pick up the changes as there's no reliable way to reimport the updated modules in a running application. But you could put your importing of common files in a shared notebook so everyone can run it easily.
Once you're happy with your code and it's fairly stable then you can package it with a setup.py and install the packages on all the nodes of your cluster like any other python package. Then you can skip the sc.addPyFile() step. DataBricks have a great facility for allowing users to upload their own Python packages/libraries. It would be great if Zeppelin provided this feature as well (although maybe they do now as I'm on an older version...). Lucas. [1] https://spark.apache.org/docs/latest/api/python/pyspark.html?highlight=addpyfile#pyspark.SparkContext.addPyFile -----Original Message----- From: Dave Boyd <db...@incadencecorp.com> Sent: 09 January 2020 17:44 To: users@zeppelin.apache.org Subject: EXT: Equivalent of Jupyter %run I have googled this but don't see a solution. We are working on a project where we want to have some common python functions shared between notes. In Jupyter we would just do a %run. Is there an equivelent in Zeppelin? Is there a way to store files as .py files that zeppelin can find for import to work. Looking to see how folks may have solved this need. -- ========= mailto:db...@incadencecorp.com ============ David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 ============== http://www.incadencecorp.com/ ============ ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 Chair ANSI/INCITS TG Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
