Technically you only need to change the build file, and change part of a line in SparkEnv so you don't have to break your oath :)
On Sun, May 4, 2014 at 10:22 PM, Soren Macbeth <so...@yieldbot.com> wrote: > that would violate my personal oath of never writing a single line of > scala, but I might be able to do that if I can get past the issue this > issue I'm struggling with in this thread. > > On Sunday, May 4, 2014, Reynold Xin <r...@databricks.com> wrote: > > > Thanks. Do you mind playing with chill-scala a little bit and see if it > > actually works well for closures? One way to try is to hard code the > > serializer to use Kryo with chill-scala, and then run through all the > unit > > tests. > > > > If it works well, we can incorporate that in the next release (probably > not > > 1.0, but after that). > > > > > > On Sun, May 4, 2014 at 9:08 PM, Soren Macbeth <so...@yieldbot.com> > wrote: > > > > > fwiw, it seems like it wouldn't be very difficult to integrate > > chill-scala, > > > since you're already chill-java and probably get kryo serialization of > > > closures and all sorts of other scala stuff for free. All that would be > > > needed would be to include the dependency and then update > KryoSerializer > > to > > > register the stuff in chill-scala. > > > > > > In that case, you could probably safely make kryo the default > serializer, > > > which I think would be desirable in general. > > > > > > > > > On Sun, May 4, 2014 at 8:48 PM, Reynold Xin <r...@databricks.com> > wrote: > > > > > > > Good idea. I submitted a pull request for the doc update here: > > > > https://github.com/apache/spark/pull/642 > > > > > > > > > > > > On Sun, May 4, 2014 at 3:54 PM, Soren Macbeth <so...@yieldbot.com> > > > wrote: > > > > > > > > > Thanks for the reply! > > > > > > > > > > Ok, if that's the case, I'd recommend a note to that affect in the > > docs > > > > at > > > > > least. > > > > > > > > > > Just to give some more context here, I'm working on a Clojure DSL > for > > > > Spark > > > > > called Flambo, which I plan to open source shortly. If I could I'd > > like > > > > to > > > > > focus on the initial bug that I hit. > > > > > > > > > > Exception in thread "main" org.apache.spark.SparkException: Job > > > aborted: > > > > > Exception while deserializing and fetching task: > > > > > com.esotericsoftware.kryo.KryoException: > > > > > java.lang.IllegalArgumentException: Can not set final > > > > > scala.collection.convert.Wrappers field > > > > > scala.collection.convert.Wrappers$SeqWrapper.$outer to > > > > > clojure.lang.PersistentVector > > > > > Serialization trace: > > > > > $outer (scala.collection.convert.Wrappers$SeqWrapper) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018) > > > > > at > > > > > > > > > > > > > > > > > > > > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > > > > > at > > > > > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > > > > > at org.apache.spark.scheduler.DAGScheduler.org > > > > > > > > > $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604) > > > > > at scala.Option.foreach(Option.scala:236) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190) > > > > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) > > > > > at akka.actor.ActorCell.invoke(ActorCell.scala:456) > > > > > >
