"object" in Scala is similar to a class with only static fields /
methods in Java. So when you set its fields in the driver, the
"object" does not get serialized and sent to the executors; they have
their own copy of the class and its static fields, which haven't been
initialized.
Use a proper class, instantiate it, and then use it in the executors. e.g.
class Foo extends Serializable {
...
}
val foo = new Foo()
foo.field1 = "blah"
lines.map(line => { println(foo) }) // now you should see the field
values you set.
On Mon, Jun 23, 2014 at 7:44 AM, Yana Kadiyska <yana.kadiy...@gmail.com> wrote:
> Hi folks, hoping someone can explain to me what's going on:
>
> I have the following code, largely based on RecoverableNetworkWordCount
> example
> (https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/RecoverableNetworkWordCount.scala):
>
> I am setting fields on an object that gets accessed within the map function.
> But the workers do not see the set values. Could someone help me understand
> what is going on? I suspect the serialization of the object happens not when
> I think it does....
>
> object Foo{
> var field1 = -1
> var field2 = -2
> }
>
> def main(args: Array[String]) {
> //open a yaml file
> Foo.field1 = value_from_yaml
> Foo.field2 = value from_yaml
>
> val ssc = StreamingContext.getOrCreate(checkpointDirectory,
> () => { createContext() })
> ssc.start()
> }
>
> def createContext(){
> // create streaming context
> println(Foo) <--
> foo's fields are set
> lines.map(line =>{
> println(Foo) <--
> foo's fields are not set
> })
> }
--
Marcelo
