Depending on what you mean by "save," you might be able to use the Twitter Storehaus package to do this. There was a nice talk about this at a Spark meetup -- "Stores, Monoids and Dependency Injection - Abstractions for Spark Streaming Jobs." Video here: https://www.youtube.com/watch?v=C7gWtxelYNM&feature=youtu.be.
Jim Donahue Adobe -----Original Message----- From: Ron Gonzalez [mailto:[email protected]] Sent: Wednesday, August 06, 2014 7:18 AM To: Vida Ha Cc: [email protected] Subject: Re: Save an RDD to a SQL Database Hi Vida, It's possible to save an RDD as a hadoop file using hadoop output formats. It might be worthwhile to investigate using DBOutputFormat and see if this will work for you. I haven't personally written to a db, but I'd imagine this would be one way to do it. Thanks, Ron Sent from my iPhone > On Aug 5, 2014, at 8:29 PM, Vida Ha <[email protected]> wrote: > > > Hi, > > I would like to save an RDD to a SQL database. It seems like this would be a > common enough use case. Are there any built in libraries to do it? > > Otherwise, I'm just planning on mapping my RDD, and having that call a method > to write to the database. Given that a lot of records are going to be > written, the code would need to be smart and do a batch insert after enough > records have collected. Does that sound like a reasonable approach? > > > -Vida > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
