This is all covered in http://spark.apache.org/docs/latest/programming-guide.html#rdd-operations
By definition, RDD transformations take an RDD to another RDD; actions produce some other type as a value on the driver program. On Fri, Sep 12, 2014 at 11:15 PM, Deep Pradhan <pradhandeep1...@gmail.com> wrote: > Is it always true that whenever we apply operations on an RDD, we get > another RDD? > Or does it depend on the return type of the operation? > > On Sat, Sep 13, 2014 at 9:45 AM, Soumya Simanta <soumya.sima...@gmail.com> > wrote: > >> >> An RDD is a fault-tolerant distributed structure. It is the primary >> abstraction in Spark. >> >> I would strongly suggest that you have a look at the following to get a >> basic idea. >> >> http://www.cs.berkeley.edu/~pwendell/strataconf/api/core/spark/RDD.html >> http://spark.apache.org/docs/latest/quick-start.html#basics >> >> https://www.usenix.org/conference/nsdi12/technical-sessions/presentation/zaharia >> >> On Sat, Sep 13, 2014 at 12:06 AM, Deep Pradhan <pradhandeep1...@gmail.com >> > wrote: >> >>> Take for example this: >>> I have declared one queue *val queue = Queue.empty[Int]*, which is a >>> pure scala line in the program. I actually want the queue to be an RDD but >>> there are no direct methods to create RDD which is a queue right? What say >>> do you have on this? >>> Does there exist something like: *Create and RDD which is a queue *? >>> >>> On Sat, Sep 13, 2014 at 8:43 AM, Hari Shreedharan < >>> hshreedha...@cloudera.com> wrote: >>> >>>> No, Scala primitives remain primitives. Unless you create an RDD using >>>> one of the many methods - you would not be able to access any of the RDD >>>> methods. There is no automatic porting. Spark is an application as far as >>>> scala is concerned - there is no compilation (except of course, the scala, >>>> JIT compilation etc). >>>> >>>> On Fri, Sep 12, 2014 at 8:04 PM, Deep Pradhan < >>>> pradhandeep1...@gmail.com> wrote: >>>> >>>>> I know that unpersist is a method on RDD. >>>>> But my confusion is that, when we port our Scala programs to Spark, >>>>> doesn't everything change to RDDs? >>>>> >>>>> On Fri, Sep 12, 2014 at 10:16 PM, Nicholas Chammas < >>>>> nicholas.cham...@gmail.com> wrote: >>>>> >>>>>> unpersist is a method on RDDs. RDDs are abstractions introduced by >>>>>> Spark. >>>>>> >>>>>> An Int is just a Scala Int. You can't call unpersist on Int in Scala, >>>>>> and that doesn't change in Spark. >>>>>> >>>>>> On Fri, Sep 12, 2014 at 12:33 PM, Deep Pradhan < >>>>>> pradhandeep1...@gmail.com> wrote: >>>>>> >>>>>>> There is one thing that I am confused about. >>>>>>> Spark has codes that have been implemented in Scala. Now, can we run >>>>>>> any Scala code on the Spark framework? What will be the difference in >>>>>>> the >>>>>>> execution of the scala code in normal systems and on Spark? >>>>>>> The reason for my question is the following: >>>>>>> I had a variable >>>>>>> *val temp = <some operations>* >>>>>>> This temp was being created inside the loop, so as to manually throw >>>>>>> it out of the cache, every time the loop ends I was calling >>>>>>> *temp.unpersist()*, this was returning an error saying that *value >>>>>>> unpersist is not a method of Int*, which means that temp is an Int. >>>>>>> Can some one explain to me why I was not able to call *unpersist* >>>>>>> on *temp*? >>>>>>> >>>>>>> Thank You >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
