You cannot pass your jobConf object inside any of the transformation function in spark (like map, mapPartitions, etc.) since org.apache.hadoop.mapreduce.Job is not Serializable. You can use KryoSerializer (See this doc http://spark.apache.org/docs/latest/tuning.html#data-serialization), We usually converts the JobConf into ByteArray and pass over the byteArray object inside the map and from there we creates the jobConf (new variable) with the data inside byteArray object.
> I should rephrase my question as follows: > > How to use the corresponding Hadoop Configuration of a HadoopRDD in > defining a function as an input parameter to the MapPartitions function? > > Thanks. > > Ey-Chih Chow -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/serialization-issue-with-mapPartitions-tp20858p20865.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
