Re: Spark Streaming Output to DB

Thank you Akhil and Mayur. It will be really helpful. Thanks, On 27 Aug 2014 13:19, "Akhil Das" wrote: > Like Mayur said, its better to use mapPartition instead of map. > > Here's a piece of code which typically reads a text file and inserts each > raw into the database. I haven't tested it, It

Re: Spark Streaming Output to DB

Like Mayur said, its better to use mapPartition instead of map. Here's a piece of code which typically reads a text file and inserts each raw into the database. I haven't tested it, It might throw up some Serialization errors, In that case, you gotta serialize them! JavaRDD txtRDD

Re: Spark Streaming Output to DB

I would suggest you to use JDBC connector in mappartition instead of maps as JDBC connections are costly & can really impact your performance. Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi On Tue, Aug 26, 2014 at 6:45 PM,

Re: Spark Streaming Output to DB

Yes, you can open a jdbc connection at the beginning of the map method then close this connection at the end of map() and in between you can use this connection. Thanks Best Regards On Tue, Aug 26, 2014 at 6:12 PM, Ravi Sharma wrote: > Hello People, >> >> I'm using java spark streaming. I'm ju

Re: Spark Streaming Output to DB

> > Hello People, > > I'm using java spark streaming. I'm just wondering, Can I make simple jdbc > connection in JavaDStream map() method? > > Or > > Do I need to create jdbc connection for each JavaPairDStream, after map > task? > > Kindly give your thoughts. > > > Cheers, > Ravi Sharma >