I did a quick search but haven't found JIRA in this regard. If configuration is separate from checkpoint data, more use cases can be accommodated.
> On Mar 15, 2016, at 2:21 AM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > > Currently configuration is a part of checkpoint data, and when recovering > from failure, Spark Streaming will fetch the configuration from checkpoint > data, so even if you change the configuration file, recovered Spark Streaming > application will not use it. So from my understanding currently there's no > way to handle your situation. > > Thanks > Saisai > >> On Tue, Mar 15, 2016 at 5:12 PM, Ewan Leith <ewan.le...@realitymine.com> >> wrote: >> Has anyone seen a way of updating the Spark streaming job configuration >> while retaining the existing data in the write ahead log? >> >> >> >> e.g. if you’ve launched a job without enough executors and a backlog has >> built up in the WAL, can you increase the number of executors without losing >> the WAL data? >> >> >> >> Thanks, >> >> Ewan >> >
