we were running at 8 executors per node, and BARELY even stressing the machines (32 cores, ~230G RAM).
in the interest of actually using system resources, and giving ourselves some headroom, i upped the executors to 16 per node. i'll be keeping an eye on ganglia for the rest of the week to make sure everything's cool. i hope you all enjoy your freshly allocated capacity! :) shane