Re: how to send JavaDStream RDD using foreachRDD using Java

2016-02-09 Thread unk1102
Hi Sachin, how did you write to Kafka from Spark I cant find the following method sendString and sendDataAsString in KafkaUtils can you please guide? KafkaUtil.sendString(p,topic,result.get(0)); KafkaUtils.sendDataAsString(MTP,topicName, result.get(0)); -- View this message in context: http

Re: how to send JavaDStream RDD using foreachRDD using Java

2015-02-03 Thread sachin Singh
Hi all, issue has bee resolved, when I used rdd.foreachRDD(new Function, Void>() { @Override public Void call(JavaRDD rdd) throws Exception { if(rdd!=null) { List result = rdd.col

Re: how to send JavaDStream RDD using foreachRDD using Java

2015-02-02 Thread Tathagata Das
Hello Sachin, While Akhil's solution is correct, this is not sufficient for your usecase. RDD.foreach (that Akhil is using) will run on the workers, but you are creating the Producer object on the driver. This will not work, a producer create on the driver cannot be used from the worker/executor.

Re: how to send JavaDStream RDD using foreachRDD using Java

2015-02-02 Thread Akhil Das
Here you go: JavaDStream textStream = ssc.textFileStream("/home/akhld/sigmoid/"); textStream.foreachRDD(new Function,Void>() { @Override public Void call(JavaRDD rdd) throws Exception { // TODO Auto-generated method stub rdd.foreach(new VoidFunction(){ @Override public void call