That isn't accurate, I think you're confused about foreach. Look at
http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd On Mon, Sep 21, 2015 at 7:36 AM, Romi Kuntsman <r...@totango.com> wrote: > foreach is something that runs on the driver, not the workers. > > if you want to perform some function on each record from cassandra, you > need to do cassandraRdd.map(func), which will run distributed on the spark > workers > > *Romi Kuntsman*, *Big Data Engineer* > http://www.totango.com > > On Mon, Sep 21, 2015 at 3:29 PM, Priya Ch <learnings.chitt...@gmail.com> > wrote: > >> Yes, but i need to read from cassandra db within a spark >> transformation..something like.. >> >> dstream.forachRDD{ >> >> rdd=> rdd.foreach { >> message => >> sc.cassandraTable() >> . >> . >> . >> } >> } >> >> Since rdd.foreach gets executed on workers, how can i make sparkContext >> available on workers ??? >> >> Regards, >> Padma Ch >> >> On Mon, Sep 21, 2015 at 5:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> You can use broadcast variable for passing connection information. >>> >>> Cheers >>> >>> On Sep 21, 2015, at 4:27 AM, Priya Ch <learnings.chitt...@gmail.com> >>> wrote: >>> >>> can i use this sparkContext on executors ?? >>> In my application, i have scenario of reading from db for certain >>> records in rdd. Hence I need sparkContext to read from DB (cassandra in our >>> case), >>> >>> If sparkContext couldn't be sent to executors , what is the workaround >>> for this ?????? >>> >>> On Mon, Sep 21, 2015 at 3:06 PM, Petr Novak <oss.mli...@gmail.com> >>> wrote: >>> >>>> add @transient? >>>> >>>> On Mon, Sep 21, 2015 at 11:27 AM, Priya Ch < >>>> learnings.chitt...@gmail.com> wrote: >>>> >>>>> Hello All, >>>>> >>>>> How can i pass sparkContext as a parameter to a method in an >>>>> object. Because passing sparkContext is giving me TaskNotSerializable >>>>> Exception. >>>>> >>>>> How can i achieve this ? >>>>> >>>>> Thanks, >>>>> Padma Ch >>>>> >>>> >>>> >>> >> >
