We use Jupyter on Hadoop https://jupyterhub-on-hadoop.readthedocs.io/en/latest/
for developing spark jobs directly inside the Cluster it should run.
With that you have direct access to yarn and hdfs (fully secured) without any
migration steps.
You can control the size of your Jupyter yarn
Does anyone know where the data for this benchmark was stored?
Spark on YARN gets performance because of data locality via co-allocation of
YARN Nodemanager and HDFS Datanode, not because of the job scheduler, right?
Regards,
z0ltrix
\ Original-Nachric
