Sorry for the mistake, I actually have it this way:
val myObject = new MyObject();
val myObjectBroadcasted = sc.broadcast(myObject);
val rdd1 = sc.textFile("/file1").map(e =>
{
myObjectBroadcasted.value.insert(e._1);
(e._1,1)
});
rdd.cache.count(); //to make sure it is transformed.
val rdd2 = sc.textFile("/file2").map(e =>
{
val lookedUp = myObjectBroadcasted.value.lookup(e._1);
(e._1, lookedUp)
});
On 24 February 2015 at 17:36, Ganelin, Ilya <ilya.gane...@capitalone.com>
wrote:
> You're not using the broadcasted variable within your map operations.
> You're attempting to modify myObjrct directly which won't work because you
> are modifying the serialized copy on the executor. You want to do
> myObjectBroadcasted.value.insert and myObjectBroadcasted.value.lookup.
>
>
>
> Sent with Good (www.good.com)
>
>
>
> -----Original Message-----
> *From: *Yiannis Gkoufas [johngou...@gmail.com]
> *Sent: *Tuesday, February 24, 2015 12:12 PM Eastern Standard Time
> *To: *user@spark.apache.org
> *Subject: *Brodcast Variable updated from one transformation and used
> from another
>
> Hi all,
>
> I am trying to do the following.
>
> val myObject = new MyObject();
> val myObjectBroadcasted = sc.broadcast(myObject);
>
> val rdd1 = sc.textFile("/file1").map(e =>
> {
> myObject.insert(e._1);
> (e._1,1)
> });
> rdd.cache.count(); //to make sure it is transformed.
>
> val rdd2 = sc.textFile("/file2").map(e =>
> {
> val lookedUp = myObject.lookup(e._1);
> (e._1, lookedUp)
> });
>
> When I check the contents of myObject within the map of rdd1 everything
> seems ok.
> On the other hand when I check the contents of myObject within the map of
> rdd2 it seems to be empty.
> I am doing something wrong?
>
> Thanks a lot!
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the
> intended recipient, you are hereby notified that any review,
> retransmission, dissemination, distribution, copying or other use of, or
> taking of any action in reliance upon this information is strictly
> prohibited. If you have received this communication in error, please
> contact the sender and delete the material from your computer.
>
