1.4.0 resolves the problem. The total classes loaded for an updateStateByKey over Int and String types does not increase.
The total classes loaded for an updateStateByKey over case classes does increase over time, but the processing remains stable. Both memory consumption and CPU load remain bounded. The processing time graph for the case class test is much noisier, with repeated spikes to 2.5x over the base line. This is quite clear in the histogram, with two distinct clusters. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-updateStateByKey-fails-with-class-leak-when-using-case-classes-resend-tp22793p23364.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
