We are working with Spark Standalone Cluster with 8 Cores and 32GB Ram, with 3 nodes cluster with same configuration.
Some times streaming batch completed in less than 1sec. some times it takes more than 10 secs at that time below log will appears in console. 2016-03-29 11:35:25,044 INFO TaskSchedulerImpl:59 - Removed TaskSet 18.0, whose tasks have all completed, from pool 2016-03-29 11:35:25,044 INFO DAGScheduler:59 - Job 18 finished: foreachRDD at EventProcessor.java:87, took 1.128755 s 2016-03-29 11:35:31,471 INFO JobScheduler:59 - Added jobs for time 1459231530000 ms 2016-03-29 11:35:35,004 INFO JobScheduler:59 - Added jobs for time 1459231535000 ms 2016-03-29 11:35:40,004 INFO JobScheduler:59 - Added jobs for time 1459231540000 ms 2016-03-29 11:35:45,136 INFO JobScheduler:59 - Added jobs for time 1459231545000 ms 2016-03-29 11:35:50,011 INFO JobScheduler:59 - Added jobs for time 1459231550000 ms 2016-03-29 11:35:55,004 INFO JobScheduler:59 - Added jobs for time 1459231555000 ms 2016-03-29 11:36:00,014 INFO JobScheduler:59 - Added jobs for time 1459231560000 ms 2016-03-29 11:36:05,003 INFO JobScheduler:59 - Added jobs for time 1459231565000 ms 2016-03-29 11:36:10,087 INFO JobScheduler:59 - Added jobs for time 1459231570000 ms 2016-03-29 11:36:15,004 INFO JobScheduler:59 - Added jobs for time 1459231575000 ms 2016-03-29 11:36:20,004 INFO JobScheduler:59 - Added jobs for time 1459231580000 ms 2016-03-29 11:36:25,139 INFO JobScheduler:59 - Added jobs for time 1459231585000 ms Can you please help, how to solve this problem. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Continuously-INFO-JobScheduler-59-Added-jobs-for-time-ms-in-my-Spark-Standalone-Cluster-tp26621.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
