Re: How to use data from Database and reload every hour

2015-11-05 Thread Sabarish Sasidharan
Theoretically the executor is a long lived container. So you could use some simple caching library or a simple Singleton to cache the data in your executors, once they load it from mysql. But note that with lots of executors you might choke your mysql. Regards Sab On 05-Nov-2015 7:03 pm, "Kay-Uwe

Re: How to use data from Database and reload every hour

2015-11-05 Thread Adrian Tanase
You should look at .transform – it’s a powerful transformation (sic) that allows you to dynamically load resources and it gets executed in every micro batch. Re-broadcasting something should be possible from inside transform as that code is executed on the driver but it’s still a controversial