Did you try SparkContext.getOrCreate() ? You don't need to pass the sparkContext to the map function, you can retrieve it from the SparkContext singleton.
Regards, Ricardo On Mon, Jan 18, 2016 at 6:29 PM, gpatcham [via Apache Spark User List] < ml-node+s1001560n25998...@n3.nabble.com> wrote: > Hi, > > I have a use case where I need to pass sparkcontext in map function > > reRDD.map(row =>method1(row,sc)).saveAsTextFile(outputDir) > > Method1 needs spark context to query cassandra. But I see below error > > java.io.NotSerializableException: org.apache.spark.SparkContext > > Is there a way we can fix this ? > > Thanks > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/using-spark-context-in-map-funciton-TASk-not-serilizable-error-tp25998.html > To start a new topic under Apache Spark User List, email > ml-node+s1001560n1...@n3.nabble.com > To unsubscribe from Apache Spark User List, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cmljYXJkby5wYWl2YUBjb3JwLmdsb2JvLmNvbXwxfDQ1MDcxMTc2Mw==> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- Ricardo Paiva Big Data *globo.com* <http://www.globo.com> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/using-spark-context-in-map-funciton-TASk-not-serilizable-error-tp25998p26006.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
