I want to log timestamp of every element of the RDD so i have assigned the MSGid to every elemnt inside RDD,and increamented it.(static variable).
My code is giving distinct Msgid in local mode but in cluster mode this value is duplicated every 30-40 count. Please help !! //public static long msgId=0; newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() { @Override public Void call(JavaRDD<String> v1, Time v2) throws Exception { for(String s:v1.collect()) { spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s)); // System.out.println(msgeditor.getMessageId(s)); } return null; } }); -- Thanks & Regards, Anshu Shukla