This sounds very reasonable.
On May 21, 2015 9:34 PM, "Stephan Ewen" <se...@apache.org> wrote:
> I discussed a bit via Skype with Gyula and Paris.
>
>
> We thought about the following way to do it:
>
> - We add a dedicated streaming mode for now. The streaming mode supersedes
> the batch mode, so it can run both type of programs.
>
> - The streaming mode sets the memory manager to "lazy allocation".
> -> So long as it runs pure streaming jobs, the full heap will be
> available to window buffers and UDFs.
> -> Batch programs can still run, so mixed workloads are not prevented.
> Batch programs are a bit less robust there, because the memory manager does
> not pre-allocate memory. UDFs can eat into Flink's memory portion.
>
> - The streaming mode starts the necessary configured components/services
> for state backups
>
>
>
> Over the next versions, we want to bring these things together:
> - use the managed memory for window buffers
> - on-demand starting of the state backend
>
> Then, we deprecate the streaming mode, let both modes start the cluster in
> the same way.
>
>
>
>
>
> On Thu, May 21, 2015 at 4:01 PM, Aljoscha Krettek <aljos...@apache.org>
> wrote:
>
> > Would it not be possible to start the snapshot service once the user
> > starts the first streaming job? About 2) with checkpointing coming up,
> > would it not make sense to shift to managed memory rather sooner than
> > later. Then this point would become moot.
> >
> > On Thu, May 21, 2015 at 3:47 PM, Matthias J. Sax
> > <mj...@informatik.hu-berlin.de> wrote:
> > > What would be the consequences on "mixed" programs? (If there is any
> > > plan to support those?)
> > >
> > > Would it be necessary to have a third mode? Or would those programs
> > > simple run in streaming mode?
> > >
> > > -Matthias
> > >
> > > On 05/21/2015 03:12 PM, Stephan Ewen wrote:
> > >> Hi all!
> > >>
> > >> We discussed a while back about introducing a dedicated streaming mode
> > for
> > >> Flink. I would like to take a go at this and implement the changes,
> but
> > >> discuss them before.
> > >>
> > >>
> > >> Here is a brief summary why we wanted to introduce the dedicated
> > streaming
> > >> mode:
> > >> Even though both batch and streaming are executed by the same
> execution
> > >> engine,
> > >> a streaming setup of Flink varies a bit from a batch setup:
> > >>
> > >> 1) The streaming cluster starts an additional service to store the
> > >> distributed state snapshots.
> > >>
> > >> 2) Streaming mode uses memory a bit different, so we should configure
> > the
> > >> memory manager differently. This difference may eventually go away.
> > >>
> > >>
> > >>
> > >> Concretely, to implement this, I was thinking about introducing the
> > >> following externally visible changes
> > >>
> > >> - Additional scripts "start-streaming-cluster.sh" and
> > >> "start-streaming-local.sh"
> > >>
> > >> - An execution mode parameter for the TaskManager ("batch /
> streaming")
> > >>
> > >> - An execution mode parameter for the JobManager TaskManager ("batch
> /
> > >> streaming")
> > >>
> > >> - All local executors and mini clusters need a flag that specifies
> > whether
> > >> they will start
> > >> a streaming cluster, or a pure batch cluster.
> > >>
> > >>
> > >> Anything else that comes to your minds?
> > >>
> > >>
> > >> Greetings,
> > >> Stephan
> > >>
> > >
> >
>
