Yeah the main way to do this would be to have your own static cache of connections. These could be using an object in Scala or just a static variable in Java (for instance a set of connections that you can borrow from).
- Patrick On Thu, Dec 4, 2014 at 5:26 PM, Tobias Pfeiffer <[email protected]> wrote: > Hi, > > On Fri, Dec 5, 2014 at 3:56 AM, Akshat Aranya <[email protected]> wrote: >> >> Is it possible to have some state across multiple calls to mapPartitions >> on each partition, for instance, if I want to keep a database connection >> open? > > > If you're using Scala, you can use a singleton object, this will exist once > per JVM (i.e., once per executor), like > > object DatabaseConnector { > lazy val conn = ... > } > > Please be aware that shutting down the connection is much harder than > opening it, because you basically have no idea when processing is done for > an executor, AFAIK. > > Tobias > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
