Let's say I have to apply a complex sequence of operations to a certain RDD.
In order to make code more modular/readable, I would typically have
something like this:
object myObject {
def main(args: Array[String]) {
val rdd1 = function1(myRdd)
val rdd2 = function2(rdd1)
val rdd3 = function3(rdd2)
}
def function1(rdd: RDD) : RDD = { doSomething }
def function2(rdd: RDD) : RDD = { doSomethingElse }
def function3(rdd: RDD) : RDD = { doSomethingElseYet }
}
So I am explicitly declaring vals for the intermediate steps. Does this end
up using more storage than if I just chained all of the operations and
declared only one val instead?
If yes, is there a better way to chain together the operations?
Ideally I would like to do something like:
val rdd = function1.function2.function3
Is there a way I can write the signature of my functions to accomplish
this? Is this also an efficiency issue or just a stylistic one?
Simone Franzini, PhD
http://www.linkedin.com/in/simonefranzini
