Re: Problem in Spark Streaming

2014-06-11 Thread vinay Bajaj
633: [ParNew: 290219K->17334K(309056K), 0.0170930 secs] > 295989K->23105K(995776K), 0.0173130 secs] [Times: user=0.12 sys=0.01, > real=0.02 secs] > > Can anyone help me to understand these messgaes related to GC ? > > > > -- > View this message in context: > http:

Re: Problem in Spark Streaming

2014-06-11 Thread nilmish
cs] 295989K->23105K(995776K), 0.0173130 secs] [Times: user=0.12 sys=0.01, real=0.02 secs] Can anyone help me to understand these messgaes related to GC ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7384.html Sent from the

Re: Problem in Spark Streaming

2014-06-10 Thread Ashish Rangole
xecution time? Finished TID 490 in 14 ms on (progress: 1/6) <--what is TID? and what is the progress? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7329.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Problem in Spark Streaming

2014-06-10 Thread Yingjun Wu
spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7329.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Problem in Spark Streaming

2014-06-10 Thread Boduo Li
nderutilized. I can't explain it either. I'm not sure if the cause is the same as yours. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7327.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Problem in Spark Streaming

2014-06-10 Thread nilmish
some instant and then it drops down again to the normal level. I want to get away with these spikes in between. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7325.html Sent from the Apache Spark User List mailing list

Re: Problem in Spark Streaming

2014-06-10 Thread Boduo Li
Hi Nilmish, What's the data rate/node when you see the high latency? (It seems the latency keeps increasing.) Do you still see it if you lower the data rate or the frequency of the windowed query? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Probl

Re: Problem in Spark Streaming

2014-06-10 Thread nilmish
You can measure the latency from the logs. Search for words like Total delay in the logs. This denotes the total end to end delay for a particular query. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7312.html Sent from

Re: Problem in Spark Streaming

2014-06-10 Thread Yingjun Wu
Hi Nilmish, I confront the same problem. I am wondering how do you measure the latency? Regards, Yingjun -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310p7311.html Sent from the Apache Spark User List mailing list archive

Problem in Spark Streaming

2014-06-10 Thread nilmish
560.n3.nabble.com/Problem-in-Spark-Streaming-tp7310.html Sent from the Apache Spark User List mailing list archive at Nabble.com.