Thank you, Ryan. Didn’t there is a method for that!

> On Dec 5, 2016, at 4:10 PM, Shixiong(Ryan) Zhu <shixi...@databricks.com> 
> wrote:
> 
> RDD.sparkContext is public: 
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD@sparkContext:org.apache.spark.SparkContext
>  
> <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD@sparkContext:org.apache.spark.SparkContext>
> 
> On Mon, Dec 5, 2016 at 1:04 PM, Teng Long <longteng...@gmail.com 
> <mailto:longteng...@gmail.com>> wrote:
> Thank you for providing another answer, Holden.
> 
> So I did what Tarun and Michal suggested, and it didn’t work out as I want to 
> have a new transformation method in RDD class, and need to use that RDD’s 
> spark context which is private. So I guess the only thing I can do now is to 
> sbt publishLocal?
> 
>> On Dec 5, 2016, at 9:19 AM, Holden Karau <hol...@pigscanfly.ca 
>> <mailto:hol...@pigscanfly.ca>> wrote:
>> 
>> Doing that requires publishing a custom version of Spark, you can edit the 
>> version number do do a publishLocal - but maintaining that change is going 
>> to be difficult. The other approaches suggested are probably better, but 
>> also does your method need to be defined on the RDD class? Could you instead 
>> make a helper object or class to expose whatever functionality you need?
>> 
>> On Mon, Dec 5, 2016 at 6:06 PM long <longteng...@gmail.com 
>> <mailto:longteng...@gmail.com>> wrote:
>> Thank you very much! But why can’t I just add new methods in to the source 
>> code of RDD?
>> 
>> 
>>> On Dec 5, 2016, at 3:15 AM, Michal Šenkýř [via Apache Spark Developers 
>>> List] <[hidden email] <http://user/SendEmail.jtp?type=node&node=20107&i=0>> 
>>> wrote:
>>> 
>> 
>>> A simple Scala example of implicit classes:
>>> 
>>> implicit class EnhancedString(str: String) {
>>>   def prefix(prefix: String) = prefix + str
>>> }
>>> 
>>> println("World".prefix("Hello "))
>>> As Tarun said, you have to import it if it's not in the same class where 
>>> you use it.
>>> 
>>> Hope this makes it clearer,
>>> 
>>> Michal Senkyr
>>> 
>>> 
>>> On 5.12.2016 07:43, Tarun Kumar wrote:
>> 
>>>> Not sure if that's documented in terms of Spark but this is a fairly 
>>>> common pattern in scala known as "pimp my library" pattern, you can easily 
>>>> find many generic example of using this pattern.
>>>> 
>>>> If you want I can quickly cook up a short conplete example with 
>>>> rdd(although there is nothing really more to my example in earlier mail) ?
>>>> 
>>>> Thanks 
>>>> Tarun Kumar
>>>> 
>> 
>>>> On Mon, 5 Dec 2016 at 7:15 AM, long <<a 
>>>> href="x-msg://22/user/SendEmail.jtp?type=node&amp;node=20106&amp;i=0 <>" 
>>>> target="_top" rel="nofollow" link="external" class="">[hidden email]> 
>>>> wrote:
>> 
>>>> So is there documentation of this I can refer to? 
>>>> 
>>>>> On Dec 5, 2016, at 1:07 AM, Tarun Kumar [via Apache Spark Developers 
>>>>> List] <[hidden email] 
>>>>> <http://user/SendEmail.jtp?type=node&node=20104&i=0>> wrote:
>>>>> 
>>>> 
>>>>> Hi Tenglong,
>>>>> 
>>>>> In addition to trsell's reply, you can add any method to an rdd without 
>>>>> making changes to spark code.
>>>>> 
>>>>> This can be achieved by using implicit class in your own client code:
>>>>> 
>>>>> implicit class extendRDD[T](rdd: RDD[T]){
>>>>> 
>>>>>      def foo()
>>>>> 
>>>>> }
>>>>> 
>>>>> Then you basically nees to import this implicit class in scope where you 
>>>>> want to use the new foo method.
>>>>> 
>>>>> Thanks
>>>>> Tarun Kumar 
>>>>> 
>> 
>>>>> On Mon, 5 Dec 2016 at 6:59 AM, <<a href="<a 
>>>>> href="x-msg://19/user/SendEmail.jtp?type=node&amp;amp;node=20102&amp;amp;i=0
>>>>>  <>" 
>>>>> class="">x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=0 
>>>>> <>" target="_top" rel="nofollow" link="external" class="">[hidden email]> 
>>>>> wrote:
>>>> 
>>>>> How does your application fetch the spark dependency? Perhaps list your 
>>>>> project dependencies and check it's using your dev build.
>>>>> 
>>>>> 
>> 
>>>>> On Mon, 5 Dec 2016, 08:47 tenglong, <<a href="<a 
>>>>> href="x-msg://19/user/SendEmail.jtp?type=node&amp;amp;node=20102&amp;amp;i=1
>>>>>  <>" 
>>>>> class="">x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=1 
>>>>> <>" target="_top" rel="nofollow" link="external" class="">[hidden email]> 
>>>>> wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> Apparently, I've already tried adding a new method to RDD,
>>>>> 
>>>>> for example,
>>>>> 
>>>>> class RDD {
>>>>>   def foo() // this is the one I added
>>>>> 
>>>>>   def map()
>>>>> 
>>>>>   def collect()
>>>>> }
>>>>> 
>>>>> I can build Spark successfully, but I can't compile my application code
>>>>> which calls rdd.foo(), and the error message says
>>>>> 
>>>>> value foo is not a member of org.apache.spark.rdd.RDD[String]
>>>>> 
>>>>> So I am wondering if there is any mechanism prevents me from doing this or
>>>>> something I'm doing wrong?
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> View this message in context: 
>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100.html
>>>>>  
>>>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100.html>
>>>>> Sent from the Apache Spark Developers List mailing list archive at 
>>>>> Nabble.com <http://nabble.com/>.
>>>>> 
>>>>> ---------------------------------------------------------------------
>> 
>>>>> To unsubscribe e-mail: <a href="<a 
>>>>> href="x-msg://19/user/SendEmail.jtp?type=node&amp;amp;node=20102&amp;amp;i=2
>>>>>  <>" 
>>>>> class="">x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=2 
>>>>> <>" target="_top" rel="nofollow" link="external" class="">[hidden email]
>>>>> 
>> 
>>>>> 
>>>>> 
>>>>> If you reply to this email, your message will be added to the discussion 
>>>>> below:
>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20102.html
>>>>>  
>>>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20102.html>
>>>>> To unsubscribe from Can I add a new method to RDD class?, click here <>.
>>>>> NAML 
>>>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>> 
>>>> View this message in context: Re: Can I add a new method to RDD class? 
>>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20104.html>
>>>> Sent from the Apache Spark Developers List mailing list archive 
>>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/> at Nabble.com 
>>>> <http://nabble.com/>.
>> 
>>> If you reply to this email, your message will be added to the discussion 
>>> below:
>> 
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20106.html
>>>  
>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20106.html>
>>> To unsubscribe from Can I add a new method to RDD class?, click here <>.
>>> NAML 
>>> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>> View this message in context: Re: Can I add a new method to RDD class? 
>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20107.html>
>> Sent from the Apache Spark Developers List mailing list archive 
>> <http://apache-spark-developers-list.1001551.n3.nabble.com/> at Nabble.com 
>> <http://nabble.com/>.
> 
> 

Reply via email to