Yes, but i need to read from cassandra db within a spark
transformation..something like..
dstream.forachRDD{
rdd=> rdd.foreach {
message =>
sc.cassandraTable()
.
.
.
}
}
Since rdd.foreach gets executed on workers, how can i make sparkContext
available on workers ???
Regards,
Padma Ch
On Mon, Sep 21, 2015 at 5:10 PM, Ted Yu <[email protected]> wrote:
> You can use broadcast variable for passing connection information.
>
> Cheers
>
> On Sep 21, 2015, at 4:27 AM, Priya Ch <[email protected]>
> wrote:
>
> can i use this sparkContext on executors ??
> In my application, i have scenario of reading from db for certain records
> in rdd. Hence I need sparkContext to read from DB (cassandra in our case),
>
> If sparkContext couldn't be sent to executors , what is the workaround for
> this ??????
>
> On Mon, Sep 21, 2015 at 3:06 PM, Petr Novak <[email protected]> wrote:
>
>> add @transient?
>>
>> On Mon, Sep 21, 2015 at 11:27 AM, Priya Ch <[email protected]>
>> wrote:
>>
>>> Hello All,
>>>
>>> How can i pass sparkContext as a parameter to a method in an object.
>>> Because passing sparkContext is giving me TaskNotSerializable Exception.
>>>
>>> How can i achieve this ?
>>>
>>> Thanks,
>>> Padma Ch
>>>
>>
>>
>
