Thanks alot , Because i just want to log timestamp and unique message id and not full RDD .
On Tue, Jun 23, 2015 at 12:41 PM, Akhil Das <[email protected]> wrote: > Why don't you do a normal .saveAsTextFiles? > > Thanks > Best Regards > > On Mon, Jun 22, 2015 at 11:55 PM, anshu shukla <[email protected]> > wrote: > >> Thanx for reply !! >> >> YES , Either it should write on any machine of cluster or Can you >> please help me ... that how to do this . Previously i was using writing >> using collect () , so some of my tuples are missing while writing. >> >> //previous logic that was just creating the file on master - >> >> newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() { >> >> @Override >> public Void call(JavaRDD<String> v1, Time v2) throws Exception { >> for(String s:v1.collect()) { >> // System.out.println("v1 here is " + v1 + "-------" + s); >> spoutlog.batchLogwriter(System.currentTimeMillis(), >> "spout-MSGID," + msgeditor.getMessageId(s)); >> // System.out.println(msgeditor.getMessageId(s)); >> } >> return null; >> } >> }); >> >> >> >> >> On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher < >> [email protected]> wrote: >> >>> Is spoutLog just a non-spark file writer? If you run that in the map >>> call on a cluster its going to be writing in the filesystem of the executor >>> its being run on. I'm not sure if that's what you intended. >>> >>> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <[email protected]> >>> wrote: >>> >>>> Running perfectly in local system but not writing to file in cluster mode >>>> .ANY suggestions please .. >>>> >>>> >>>> //msgid is long counter >>>> >>>> JavaDStream<String> newinputStream=inputStream.map(new Function<String, >>>> String>() { >>>> @Override >>>> public String call(String v1) throws Exception { >>>> String s1=msgId+"@"+v1; >>>> System.out.println(s1); >>>> msgId++; >>>> try { >>>> *//filewriter logic >>>> spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + >>>> msgeditor.getMessageId(s1));* >>>> } catch (Exception e) { >>>> >>>> System.out.println("exeception is here"); >>>> e.printStackTrace(); >>>> throw e; >>>> } >>>> System.out.println("msgid,"+msgId); >>>> return msgeditor.addMessageId(v1,msgId); >>>> } >>>> }); >>>> >>>> >>>> -- >>>> Thanks & Regards, >>>> Anshu Shukla >>>> >>>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <[email protected]> >>>> wrote: >>>> >>>>> Can not we write some data to a txt file in parallel with multiple >>>>> executors running in parallel ?? >>>>> >>>>> >>>>> -- >>>>> Thanks & Regards, >>>>> Anshu Shukla >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks & Regards, >>>> Anshu Shukla >>>> >>> >>> >> >> >> -- >> Thanks & Regards, >> Anshu Shukla >> > > -- Thanks & Regards, Anshu Shukla
