Thanks. This is an important direction to explore and my apologies for the
late reply.
One thing that is really hard about this is that with different layers of
abstractions, we often use other libraries that might allocate large amount
of memory (e.g. snappy library, Parquet itself), which makes
https://issues.apache.org/jira/browse/SPARK-21157
Hi - often times, Spark applications are killed for overrunning available
memory by YARN, Mesos, or the OS. In SPARK-21157, I propose a design for
grabbing and reporting "total memory" usage for Spark executors - that is,
memory usage as visible fr
