Resource allocation in SPARK streaming

I am not much clear about resource allocation (CPU/CORE/Thread level allocation) as per the parallelism by setting number of cores in spark standalone mode . Any guidelines for that . -- Thanks & Regards, Anshu Shukla

Parallelism of Custom receiver in spark

1 - How to increase the level of *parallelism in spark streaming custom RECEIVER* . 2 - Will ssc.receiverstream(/**anything //) will *delete the data stored in spark memory using store(s) * logic . -- Thanks & Regards, Anshu Shukla

ReceiverStream SPARK not able to cope up with 20,000 events /sec .

(","); } String s1=MsgIdAddandRemove.addMessageId(tuple.toString(),msgId); store(s1); } -- Thanks & Regards, Anshu Shukla

Re: Loss of data due to congestion

Thaks, I am talking about streaming. On 25 Jun 2015 5:37 am, "ayan guha" wrote: > Can you elaborate little more? Are you talking about receiver or streaming? > On 24 Jun 2015 23:18, "anshu shukla" wrote: > >> How spark guarantees that no RDD will fail /lost

Loss of data due to congestion

How spark guarantees that no RDD will fail /lost during its life cycle . Is there something like ask in storm or its does it by default . -- Thanks & Regards, Anshu Shukla

Calculating tuple count /input rate with time

, Void>() { @Override public Void call(JavaRDD stringJavaRDD) throws Exception { System.out.println(System.currentTimeMillis()+",spoutstringJavaRDD," + stringJavaRDD.count() ); return null; } }); -- Thanks & Regards, Anshu Shukla

Fwd: Verifying number of workers in Spark Streaming

not able figure out that my job is using all workers or not . -- Thanks & Regards, Anshu Shukla SERC-IISC

Latency between the RDD in Streaming

Is there any fixed way to find among RDD in stream processing systems , in the Distributed set-up . -- Thanks & Regards, Anshu Shukla

Implementing and Using a Custom Actor-based Receiver

Is there any good sample code in java to implement *Implementing and Using a Custom Actor-based Receiver .* -- Thanks & Regards, Anshu Shukla

Problem: Custom Receiver for getting events from a Dynamic Queue

Auto-generated method stub //System.out.println("Called IN SPOUT### "); try { this.eventQueue.put(event); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } -- Thanks & Regards, Anshu Shukla

Using queueStream

JavaDStream inputStream = ssc.queueStream(rddQueue); Can this rddQueue be of dynamic type in nature .If yes then how to make it run untill rddQueue is not finished . Any other way to get rddQueue from a dynamically updatable Normal Queue . -- Thanks & Regards, SERC-IISC Anshu Shukla

Re: Map one RDD into two RDD

PM > *To:* u...@spark.apache.org > *Subject:* Map one RDD into two RDD > > > > Hi all, > > I have a large RDD that I map a function to it. Based on the nature of > each record in the input RDD, I will generate two types of data. I would > like to save each type into its own RDD. But I can't seem to find an > efficient way to do it. Any suggestions? > > > > Many thanks. > > > > > > Bill > > > > -- > > Many thanks. > > Bill > > > > > > -- > > Many thanks. > > Bill > > > > > > -- > > Many thanks. > > Bill > > > -- Thanks & Regards, Anshu Shukla

Predict.scala using model for clustering In reference

rence-apps/blob/master/twitter_classifier/predict.md -- Thanks & Regards, Anshu Shukla

Re: Creating topology in spark streaming

0/improvements-to-kafka-integration-of-spark-streaming.html > > Hope that helps. > > Greetings, > > Juan > > 2015-05-06 10:32 GMT+02:00 anshu shukla : > >> But main problem is how to increase the level of parallelism for any >> particular bolt logic . >

Re: Creating topology in spark streaming

on a dstream will create another dstream. You may > want to take a look at foreachrdd? Also, kindly share your code so people > can help better > On 6 May 2015 17:54, "anshu shukla" wrote: > >> Please help guys, Even After going through all the examples given i >>

Creating topology in spark streaming

levele of parallelism since the logic of topology is not clear . -- Thanks & Regards, Anshu Shukla Indian Institute of Sciences

Re: Event generator for SPARK-Streaming from csv

hat helps, > > Greetings, > > Juan > > 2015-05-01 9:30 GMT+02:00 anshu shukla : > >> >> >> >> >> I have the real DEBS-TAxi data in csv file , in order to operate over it >> how to simulate a "Spout" kind of thing as event generato

spark log analyzer sample

Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4 I am not using any hadoop facility (not even hdfs) then why it is giving this error . -- Thanks & Regards, Anshu Shukla

Fwd: Event generator for SPARK-Streaming from csv

I have the real DEBS-TAxi data in csv file , in order to operate over it how to simulate a "Spout" kind of thing as event generator using the timestamps in CSV file. -- Thanks & Regards, Anshu Shukla

Event generator for SPARK-Streaming from csv

I have the real DEBS-TAxi data in csv file , in order to operate over it how to simulate a "Spout" kind of thing as event generator using the timestamps in CSV file. -- SERC-IISC Thanks & Regards, Anshu Shukla

Re: Integrating D3 with Spark

Hey Ousterhout , I found its amazing .Before this i used to use my own D3.js files that subscribes to the redis pub-shub database where output tuples are being published to the DB . So it was already including latency to push data to redis ,although it was very less. Once again tha

Support for cycles in spark streaming topology ?????

-- Thanks & Regards, Anshu Shukla