Hello, I am executing the SparkPageRank example. It uses the "cache()" API for persistence of RDDs. And if I am not wrong, it in turn uses MEMORY_ONLY storage level. However, in oprofile report it shows a lot of activity in writeObject0 function.
There is not even a single "Spilling in-memory..." message in the output/log. This is because I am using a huge heap size of 120GB. Can someone please tell me why do I see so much serialization happening, even though MEMORY_ONLY storage level is used? The spark version that I am using is 1.0.1 Thanks, Lokesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Lot-of-object-serialization-even-with-MEMORY-ONLY-tp10722.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
