Thanks for your reply! As you mentioned , the insert clause is not executed as
the results of args.map are never used anywhere, and after i modified the code
, it works.
qinwei
From: Tobias PfeifferDate: 2014-11-07 18:04To: qinweiCC: userSubject: Re:
about write mongodb in mapPartitions
Thanks for your reply！ According to your hint, the code should be like this:
// i want to save data in rdd to mongodb and hdfs
rdd.saveAsNewAPIHadoopFile() rdd.saveAsTextFile()
but will the application read hdfs twice?
qinwei
From: Akhil DasDate: 2014-11-07 18
Hi,
On Fri, Nov 7, 2014 at 6:23 PM, qinwei wrote:
>
> args.map(arg => {
> coll.insert(new BasicDBObject("pkg", arg))
> arg
> })
>
> mongoClient.close()
> args
>
As the results of args.map are never used anywhere, I t
Why not saveAsNewAPIHadoopFile?
//Define your mongoDB confs
val config = new Configuration()
config.set("mongo.output.uri", "mongodb://
127.0.0.1:27017/sigmoid.output")
//Write everything to mongo
rdd.saveAsNewAPIHadoopFile("file:///some/random", classOf[Any],
classOf[Any], classOf[com.m
