Spark Streaming getting slower

2016-06-09 Thread John Simon
bugLevel sun.os.patch.level unknown user.country US user.dir /home/hadoop user.home /home/hadoop user.language en user.name hadoop user.timezone UTC ``` -- John Simon - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org Fo

Re: Spark Streaming getting slower

2016-06-09 Thread John Simon
Sorry, forgot to mention that I don't use broadcast variables. That's why I'm puzzled here. -- John Simon On Thu, Jun 9, 2016 at 11:09 AM, John Simon wrote: > Hi, > > I'm running Spark Streaming with Kafka Direct Stream, batch interval > is 10 seconds. > Afte

Re: Long Running Spark Streaming getting slower

2016-06-10 Thread John Simon
Hi Mich, batch interval is 10 seconds, and I don't use sliding window. Typical message count per batch is ~100k. -- John Simon On Fri, Jun 10, 2016 at 10:31 AM, Mich Talebzadeh wrote: > Hi John, > > I did not notice anything unusual in your env variables. > > However