Yeah, this is related. From https://groups.google.com/forum/#!msg/spark-users/bwAmbUgxWrA/HwP4Nv4adfEJ: "This is a limitation that will hopefully go away in Scala 2.10 or 2.10 .1, when we'll use macros to remove the need to do this. (Or more generally if we get some changes in the Scala interpreter to do something smarter in this case.) "
We're using Spark 0.9.0, Scala 2.10.3 and the limitation is there. Any ideas when it is going to be fixed? The workaround with embedding everything inside a singleton object does not work for me, because nested classes defined there are still inner and require additional argument to the constructor (when invoked by reflection). If I only had some reliable way to obtain a reference to that outer object by reflection, we could somehow workaround it. E.g. saving it in some singleton object, etc. However, a proper fix would be to make non-inner classes properly non-inner. Thanks, Piotr 2014-04-25 0:13 GMT+02:00 Michael Armbrust <mich...@databricks.com>: > The Spark REPL is slightly modified from the normal Scala REPL to prevent > work from being done twice when closures are deserialized on the workers. > I'm not sure exactly why this causes your problem, but its probably worth > filing a JIRA about it. > > Here is another issues with classes defined in the REPL. Not sure if it is > related, but I'd be curious if the workaround helps you: > https://issues.apache.org/jira/browse/SPARK-1199 > > Michael > > > On Thu, Apr 24, 2014 at 3:14 AM, Piotr Kołaczkowski > <pkola...@datastax.com>wrote: > > > Hi, > > > > I'm working on Cassandra-Spark integration and I hit a pretty severe > > problem. One of the provided functionality is mapping Cassandra rows into > > objects of user-defined classes. E.g. like this: > > > > class MyRow(val key: String, val data: Int) > > sc.cassandraTable("keyspace", "table").select("key", "data").as[MyRow] > // > > returns CassandraRDD[MyRow] > > > > In this example CassandraRDD creates MyRow instances by reflection, i.e. > > matches selected fields from Cassandra table and passes them to the > > constructor. > > > > Unfortunately this does not work in Spark REPL. > > Turns out any class declared on the REPL is an inner classes, and to be > > successfully created, it needs a reference to the outer object, even > though > > it doesn't really use anything from the outer context. > > > > scala> class SomeClass > > defined class SomeClass > > > > scala> classOf[SomeClass].getConstructors()(0) > > res11: java.lang.reflect.Constructor[_] = public > > $iwC$$iwC$SomeClass($iwC$$iwC) > > > > I tried passing a null as a temporary workaround, and it also doesn't > work > > - I get NPE. > > How can I get a reference to the current outer object representing the > > context of the current line? > > > > Also, plain non-spark Scala REPL doesn't exhibit this behaviour - and > > classes declared on the REPL are proper top-most classes, not inner ones. > > Why? > > > > Thanks, > > Piotr > > > > > > > > > > > > > > > > -- > > Piotr Kolaczkowski, Lead Software Engineer > > pkola...@datastax.com > > > > 777 Mariners Island Blvd., Suite 510 > > San Mateo, CA 94404 > > > -- Piotr Kolaczkowski, Lead Software Engineer pkola...@datastax.com 777 Mariners Island Blvd., Suite 510 San Mateo, CA 94404
