I try to use org.apache.spark.util.collection.BitSet instead of RoaringBitMap, and it can save about 20% memories but runs much slower.
For the 200K tasks job, RoaringBitMap uses 3 Long[1024] and 1 Short[3392] =3*64*1024+16*3392=250880(bit) BitSet uses 1 Long[3125] = 3125*64=200000(bit) Memory saved = (250880-200000) / 250880 ≈20% -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/MapStatus-too-large-for-drvier-tp14704p14723.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
