Re: Spark-job error on writing result into hadoop w/ switch_user=false

Hi, For more details, I'm using mesos below and run a very simple command on spark-shell, scala> sc.textFile("/data/pickat/tsv/app/2014/07/31/*").map(_.split).groupBy(p => p(1)).saveAsTextFile("/user/1001079/pickat_test") "hdfs" is an account name run mesos, "1001079" is that of running script.

Spark-job error on writing result into hadoop w/ switch_user=false

Hi, I've used hdfs 2.3.0-cdh5.0.1, mesos 0.19.1 and spark 1.0.2 that is re-compiled. For a security reason, we run hdfs and mesos as hdfs, that is an account name and not in a root group, and non-root user submit a spark job on mesos. With no-switch_user, simple job, which only read data from hdf