Re: running pyspark on kubernetes - no space left on device

2022-09-01 Thread Qian SUN
Hi Spark provides spark.local.dir configuration to specify work folder on the pod. You can specify spark.local.dir as your mount path. Best regards Manoj GEORGE 于2022年9月1日周四 21:16写道: > CONFIDENTIAL & RESTRICTED > > Hi Team, > > > > I am new to spark, so please excuse my ignorance. > > > > Curre

Re: running pyspark on kubernetes - no space left on device

2022-09-01 Thread Matt Proetsch
Hi George, You can try mounting a larger PersistentVolume to the work directory as described here instead of using localdir which might have site-specific size constraints: https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes -Matt > On Sep 1, 2022, at 09:1