Thanks Akhil. For posterity, I ended up with:
https://gist.github.com/dokipen/aa07f351a970fe54fcff
I couldn't get rddToFilename() to work, but it's impl was pretty simple.
I'm a poet but I don't know it.
On Tue, Jun 9, 2015 at 3:10 AM Akhil Das <[email protected]> wrote:
> like this?
>
> myDStream.foreachRDD(rdd => rdd.saveAsTextFile("/sigmoid/", codec ))
>
>
> Thanks
> Best Regards
>
> On Mon, Jun 8, 2015 at 8:06 PM, Bob Corsaro <[email protected]> wrote:
>
>> It looks like saveAsTextFiles doesn't support the compression parameter
>> of RDD.saveAsTextFile. Is there a way to add the functionality in my client
>> code without patching Spark? I tried making my own saveFunc function and
>> calling DStream.foreachRDD but ran into trouble with invoking rddToFileName
>> and making the RDD type parameter work properly. It's probably just do to
>> my lack of Scala knowledge. Can anyone give me a hand?
>>
>> def saveAsTextFiles(prefix: String, suffix: String = ""): Unit =
>> ssc.withScope {
>> val saveFunc = (rdd: RDD[T], time: Time) => {
>> val file = rddToFileName(prefix, suffix, time)
>> rdd.saveAsTextFile(file)
>> }
>> this.foreachRDD(saveFunc)
>> }
>>
>>
>
