Thanx for reply !!
YES , Either it should write on any machine of cluster or Can you please
help me ... that how to do this . Previously i was using writing using
collect () , so some of my tuples are missing while writing.
//previous logic that was just creating the file on master -
newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() {
@Override
public Void call(JavaRDD<String> v1, Time v2) throws Exception {
for(String s:v1.collect()) {
// System.out.println("v1 here is " + v1 + "-------" + s);
spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," +
msgeditor.getMessageId(s));
// System.out.println(msgeditor.getMessageId(s));
}
return null;
}
});
On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher <[email protected]
> wrote:
> Is spoutLog just a non-spark file writer? If you run that in the map call
> on a cluster its going to be writing in the filesystem of the executor its
> being run on. I'm not sure if that's what you intended.
>
> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <[email protected]>
> wrote:
>
>> Running perfectly in local system but not writing to file in cluster mode
>> .ANY suggestions please ..
>>
>>
>> //msgid is long counter
>>
>> JavaDStream<String> newinputStream=inputStream.map(new Function<String,
>> String>() {
>> @Override
>> public String call(String v1) throws Exception {
>> String s1=msgId+"@"+v1;
>> System.out.println(s1);
>> msgId++;
>> try {
>> *//filewriter logic
>> spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," +
>> msgeditor.getMessageId(s1));*
>> } catch (Exception e) {
>>
>> System.out.println("exeception is here");
>> e.printStackTrace();
>> throw e;
>> }
>> System.out.println("msgid,"+msgId);
>> return msgeditor.addMessageId(v1,msgId);
>> }
>> });
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <[email protected]>
>> wrote:
>>
>>> Can not we write some data to a txt file in parallel with multiple
>>> executors running in parallel ??
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anshu Shukla
>>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>
>
--
Thanks & Regards,
Anshu Shukla
