Hi, so I didn't manage to get the Broadcast variable with a new value distributed to my executors in YARN mode. In local mode it worked fine, but when running on YARN either nothing happened (when unpersist() was called on the driver) or I got a TimeoutException (when called on the executor). I finally dropped the use of broadcast variables and added a HTTP polling mechanism from the executors to the driver. I find that a bit suboptimal, in particular since there is this whole Akka infrastructure already running and I should be able to just send messages around. However, Spark does not seem to encourage this. (In general I find that "private" is a bit overused in the Spark codebase...)
Thanks Tobias
