Then apply a transformation over the dstream to pull those required
information. :)
Thanks
Best Regards
On Tue, Jun 23, 2015 at 3:22 PM, anshu shukla <anshushuk...@gmail.com>
wrote:
> Thanks alot ,
>
> Because i just want to log timestamp and unique message id and not full
> RDD .
>
> On Tue, Jun 23, 2015 at 12:41 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Why don't you do a normal .saveAsTextFiles?
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Jun 22, 2015 at 11:55 PM, anshu shukla <anshushuk...@gmail.com>
>> wrote:
>>
>>> Thanx for reply !!
>>>
>>> YES , Either it should write on any machine of cluster or Can you
>>> please help me ... that how to do this . Previously i was using writing
>>> using collect () , so some of my tuples are missing while writing.
>>>
>>> //previous logic that was just creating the file on master -
>>>
>>> newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>()
>>> {
>>>
>>> @Override
>>> public Void call(JavaRDD<String> v1, Time v2) throws Exception {
>>> for(String s:v1.collect()) {
>>> // System.out.println("v1 here is " + v1 + "-------" + s);
>>> spoutlog.batchLogwriter(System.currentTimeMillis(),
>>> "spout-MSGID," + msgeditor.getMessageId(s));
>>> // System.out.println(msgeditor.getMessageId(s));
>>> }
>>> return null;
>>> }
>>> });
>>>
>>>
>>>
>>>
>>> On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher <
>>> rmarsc...@localytics.com> wrote:
>>>
>>>> Is spoutLog just a non-spark file writer? If you run that in the map
>>>> call on a cluster its going to be writing in the filesystem of the executor
>>>> its being run on. I'm not sure if that's what you intended.
>>>>
>>>> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <anshushuk...@gmail.com>
>>>> wrote:
>>>>
>>>>> Running perfectly in local system but not writing to file in cluster mode
>>>>> .ANY suggestions please ..
>>>>>
>>>>>
>>>>> //msgid is long counter
>>>>>
>>>>> JavaDStream<String> newinputStream=inputStream.map(new Function<String,
>>>>> String>() {
>>>>> @Override
>>>>> public String call(String v1) throws Exception {
>>>>> String s1=msgId+"@"+v1;
>>>>> System.out.println(s1);
>>>>> msgId++;
>>>>> try {
>>>>> *//filewriter logic
>>>>> spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," +
>>>>> msgeditor.getMessageId(s1));*
>>>>> } catch (Exception e) {
>>>>>
>>>>> System.out.println("exeception is here");
>>>>> e.printStackTrace();
>>>>> throw e;
>>>>> }
>>>>> System.out.println("msgid,"+msgId);
>>>>> return msgeditor.addMessageId(v1,msgId);
>>>>> }
>>>>> });
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anshu Shukla
>>>>>
>>>>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <anshushuk...@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Can not we write some data to a txt file in parallel with multiple
>>>>>> executors running in parallel ??
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Anshu Shukla
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anshu Shukla
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anshu Shukla
>>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
