Re: Can I add a new method to RDD class?

Not sure if that's documented in terms of Spark but this is a fairly common pattern in scala known as "pimp my library" pattern, you can easily find many generic example of using this pattern. If you want I can quickly cook up a short conplete example with rdd(although there is nothing really more

Re: Can I add a new method to RDD class?

So is there documentation of this I can refer to? > On Dec 5, 2016, at 1:07 AM, Tarun Kumar [via Apache Spark Developers List] > wrote: > > Hi Tenglong, > > In addition to trsell's reply, you can add any method to an rdd without > making changes to spark code. > > This can be achieved by us

Re: Can I add a new method to RDD class?

So im my sbt build script, I have the same line as instructed in the quickstart guide , libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.2" And since I was able to see all the other logs I added into the spark source code, s

Re: Can I add a new method to RDD class?

Hi Tenglong, In addition to trsell's reply, you can add any method to an rdd without making changes to spark code. This can be achieved by using implicit class in your own client code: implicit class extendRDD[T](rdd: RDD[T]){ def foo() } Then you basically nees to import this implicit class

Re: Can I add a new method to RDD class?

How does your application fetch the spark dependency? Perhaps list your project dependencies and check it's using your dev build. On Mon, 5 Dec 2016, 08:47 tenglong, wrote: > Hi, > > Apparently, I've already tried adding a new method to RDD, > > for example, > > class RDD { > def foo() // this

Can I add a new method to RDD class?

Hi, Apparently, I've already tried adding a new method to RDD, for example, class RDD { def foo() // this is the one I added def map() def collect() } I can build Spark successfully, but I can't compile my application code which calls rdd.foo(), and the error message says value foo is

Re: [VOTE] Apache Spark 2.1.0 (RC1)

with the current branch-2.1 after rc1 i am now also seeing this error in our unit tests: java.lang.UnsupportedOperationException: Cannot create encoder for Option of Product type, because Product type is represented as a row, and the entire row can not be null in Spark SQL like normal databases.

Re: [VOTE] Apache Spark 2.1.0 (RC1)

somewhere between rc1 and the current head of branch-2.1 i started seeing an NPE in our in-house unit tests for Dataset + Aggregator. i created SPARK-18711 for this. On Mon, Nov 28, 2016 at 8:25

Re: ability to provide custom serializers

Thanks Michael! > On Dec 2, 2016, at 7:29 PM, Michael Armbrust wrote: > > I would love to see something like this. The closest related ticket is > probably https://issues.apache.org/jira/browse/SPARK-7768 > (though maybe there are > enough p

Re: Future of the Python 2 support.

Echoing Nick. I don't see any strong reason to drop Python 2 support. We typically drop support for X when it is rarely used and support for X is long past EOL. Python 2 is still very popular, and depending on the statistics it might be more popular than Python 3. On Sun, Dec 4, 2016 at 9:29 AM N

Re: Future of the Python 2 support.

I don't think it makes sense to deprecate or drop support for Python 2.7 until at least 2020, when 2.7 itself will be EOLed. (As of Spark 2.0, Python 2.6 support is deprecated and will be removed by Spark 2.2. Python 2.7 is only version of Python 2 that's still fully supported.) Given the widespre

Future of the Python 2 support.

Hi, I am aware there was a previous discussion about dropping support for different platforms (http://apache-spark-developers-list.1001551.n3.nabble.com/Straw-poll-dropping-support-for-things-like-Scala-2-10-td19553.html) but somehow it has been dominated by Scala and JVM and never touched the sub