This is a false error message actually - the Maven build no longer
requires SCALA_HOME but the message/check was still there. This was
fixed recently in master:
https://github.com/apache/spark/commit/d8c005d5371f81a2a06c5d27c7021e1ae43d7193
I can back port that fix into branch-1.0 so it will be i
I went ahead and created a JIRA for this and back ported the
improvement into branch-1.0. This wasn't a regression per-se because
the behavior existed in all previous versions, but it's annoying
behavior so best to fix it.
https://issues.apache.org/jira/browse/SPARK-1984
- Patrick
On Sun, Jun 1,
Cheers, I didn't think it was needed, but just wanted to point it out.
On Sun, Jun 1, 2014 at 11:21 AM, Patrick Wendell wrote:
> I went ahead and created a JIRA for this and back ported the
> improvement into branch-1.0. This wasn't a regression per-se because
> the behavior existed in all prev
https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/serializer/Serializer.scala#L64-L66
These changes to the SerializerInstance make it really gross to call
serialize and deserialize from non-scala languages. I'm not sure what the
purpose of a ClassTag is, but if we co
Why do you need to call Serializer from your own program? It’s an internal
developer API so ideally it would only be called to extend Spark. Are you
looking to implement a custom Serializer?
Matei
On Jun 1, 2014, at 3:40 PM, Soren Macbeth wrote:
> https://github.com/apache/spark/blob/v1.0.0/c
BTW passing a ClassTag tells the Serializer what the type of object being
serialized is when you compile your program, which will allow for more
efficient serializers (especially on streams).
Matei
On Jun 1, 2014, at 4:24 PM, Matei Zaharia wrote:
> Why do you need to call Serializer from your
I'm writing a Clojure DSL for Spark. I use kryo to serialize my clojure
functions and for efficiency I hook into Spark's kryo serializer. In order
to do that I get a SerializerInstance from SparkEnv and call the serialize
and deserialize methods. I was able to workaround it by making ClassTag
objec
Ah, got it. In general it will always be safe to pass the ClassTag for
java.lang.Object here — this is what our Java API does to say that type info is
not known. So you can always pass that. Look at the Java code for how to get
this ClassTag.
Matei
On Jun 1, 2014, at 4:33 PM, Soren Macbeth wr
Yep, that's what I'm doing.
(def OBJECT-CLASS-TAG (.apply ClassTag$/MODULE$ java.lang.Object))
ps - I'm planning to open source this Clojure DSL soon as well
On Sun, Jun 1, 2014 at 5:10 PM, Matei Zaharia
wrote:
> Ah, got it. In general it will always be safe to pass the ClassTag for
> java.la
Very cool, looking forward to it!
Matei
On Jun 1, 2014, at 5:42 PM, Soren Macbeth wrote:
> Yep, that's what I'm doing.
>
> (def OBJECT-CLASS-TAG (.apply ClassTag$/MODULE$ java.lang.Object))
>
> ps - I'm planning to open source this Clojure DSL soon as well
>
>
> On Sun, Jun 1, 2014 at 5:10
10 matches
Mail list logo
