All, Is a Function object in Spark reused on a given executor, or is sent and deserialized with each new task?
On my project, we have functions that incur a very large setup cost, but then could be called many times. Currently, I am using object deserialization to run this intensive setup, I am wondering if this function is reused (within the context of the executor), or I am I deserializing this object over and over again for each task sent to a given worker. Are there other ways to share objects between tasks on the same executor? Many thanks, Michael --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
