Hi Henry!
I think the idea was to have a dedicated streaming mode as long as the
default cluster mode does not support batch and streaming equally well.
Once we have reached this level in the dedicated streaming mode, this will
be the default cluster mode. I share your doubts about whether it is a
Ah yes, technically the streaming mode could run batch jobs as well in Flink.
I am thinking that it could cause confusion with users since most
systems that does batch and stream (well, pretty much Spark ^_^) does
not differentiate the deployment topologies for the cluster to support
different mode
The streaming mode runs batch jobs as well :-)
There should be slightly reduced predictability in the memory management in
the streaming mode, but otherwise there should not be a problem.
So if you want to run mixed workloads, you start the streaming mode.
(Note: Currently, the batch mode runs
One immediate concern I have is the deployment topology. With
streaming has its own cluster deployment, this means that in
standalone mode, if ops would like to deploy Flink it has to know what
mode it needs to deploy Flink as, either batch or Streaming. So, if
the use case was to support both batc
+1 great changes coming up! I like the idea that, ultimately, Flink will
handle streaming and batch programs equally well independently of the
chosen cluster startup mode.
What is the time frame for these changes?
On Tue, May 26, 2015 at 7:34 AM, Henry Saputra
wrote:
> Thanks Aljoscha and Steph
Thanks Aljoscha and Stephan, this helps
- Henry
On Fri, May 22, 2015 at 4:37 AM, Stephan Ewen wrote:
> Aljoscha is right. There are plans to migrate the streaming state to the
> MemoryManager as well, but streaming state is not managed at this point.
>
> What is managed in streaming jobs is the
Aljoscha is right. There are plans to migrate the streaming state to the
MemoryManager as well, but streaming state is not managed at this point.
What is managed in streaming jobs is the data buffered and cached in the
network stack. But that is a different memory pool than the memory manager.
We
Hi,
streaming currently does not use any memory manager. All state is kept
in Java Objects on the Java Heap, for example an ArrayList<> for the
window buffer.
On Thu, May 21, 2015 at 11:56 PM, Henry Saputra wrote:
> Hi Stephan, Gyula, Paris,
>
> How does streaming currently different in term of m
Hi Stephan, Gyula, Paris,
How does streaming currently different in term of memory management?
Currently we only have one MemoryManager which is used by both modes I
believe.
- Henry
On Thu, May 21, 2015 at 12:34 PM, Stephan Ewen wrote:
> I discussed a bit via Skype with Gyula and Paris.
>
>
>
Huge +1 from my side :)
Sorry for the late response.
On Thu, May 21, 2015 at 9:54 PM, Aljoscha Krettek
wrote:
> This sounds very reasonable.
> On May 21, 2015 9:34 PM, "Stephan Ewen" wrote:
>
> > I discussed a bit via Skype with Gyula and Paris.
> >
> >
> > We thought about the following way t
This sounds very reasonable.
On May 21, 2015 9:34 PM, "Stephan Ewen" wrote:
> I discussed a bit via Skype with Gyula and Paris.
>
>
> We thought about the following way to do it:
>
> - We add a dedicated streaming mode for now. The streaming mode supersedes
> the batch mode, so it can run both t
I discussed a bit via Skype with Gyula and Paris.
We thought about the following way to do it:
- We add a dedicated streaming mode for now. The streaming mode supersedes
the batch mode, so it can run both type of programs.
- The streaming mode sets the memory manager to "lazy allocation".
Would it not be possible to start the snapshot service once the user
starts the first streaming job? About 2) with checkpointing coming up,
would it not make sense to shift to managed memory rather sooner than
later. Then this point would become moot.
On Thu, May 21, 2015 at 3:47 PM, Matthias J. S
What would be the consequences on "mixed" programs? (If there is any
plan to support those?)
Would it be necessary to have a third mode? Or would those programs
simple run in streaming mode?
-Matthias
On 05/21/2015 03:12 PM, Stephan Ewen wrote:
> Hi all!
>
> We discussed a while back about intr
Hi all!
We discussed a while back about introducing a dedicated streaming mode for
Flink. I would like to take a go at this and implement the changes, but
discuss them before.
Here is a brief summary why we wanted to introduce the dedicated streaming
mode:
Even though both batch and streaming ar
Today we had a discussion with Robert on this issue. I would like to
eventually have the streaming grouped and the windowing buffers/state maybe
along with the crucial state of the user in the managed memory. If we had
this separating the two modes could became less important as streaming
would als
+1
I agree it’s a proper way to go.
On 18 Feb 2015, at 10:41, Max Michels mailto:m...@apache.org>>
wrote:
+1
On Tue, Feb 17, 2015 at 2:40 PM, Aljoscha Krettek
mailto:aljos...@apache.org>> wrote:
+1
On Tue, Feb 17, 2015 at 1:34 PM, Till Rohrmann
mailto:trohrm...@apache.org>> wrote:
+1
On Tu
+1
On Tue, Feb 17, 2015 at 2:40 PM, Aljoscha Krettek wrote:
> +1
>
> On Tue, Feb 17, 2015 at 1:34 PM, Till Rohrmann wrote:
>> +1
>>
>> On Tue, Feb 17, 2015 at 1:34 PM, Kostas Tzoumas wrote:
>>
>>> +1
>>>
>>> On Tue, Feb 17, 2015 at 12:14 PM, Márton Balassi
>>> wrote:
>>>
>>> > When it comes to
+1
On Tue, Feb 17, 2015 at 1:34 PM, Till Rohrmann wrote:
> +1
>
> On Tue, Feb 17, 2015 at 1:34 PM, Kostas Tzoumas wrote:
>
>> +1
>>
>> On Tue, Feb 17, 2015 at 12:14 PM, Márton Balassi
>> wrote:
>>
>> > When it comes to the current use cases I'm for this separation.
>> > @Ufuk: As Gyula has alre
+1
On Tue, Feb 17, 2015 at 1:34 PM, Kostas Tzoumas wrote:
> +1
>
> On Tue, Feb 17, 2015 at 12:14 PM, Márton Balassi
> wrote:
>
> > When it comes to the current use cases I'm for this separation.
> > @Ufuk: As Gyula has already pointed out with the current design of
> > integration it should not
+1
On Tue, Feb 17, 2015 at 12:14 PM, Márton Balassi
wrote:
> When it comes to the current use cases I'm for this separation.
> @Ufuk: As Gyula has already pointed out with the current design of
> integration it should not be a problem. Even if we submitted programs to
> the wrong clusters it wou
When it comes to the current use cases I'm for this separation.
@Ufuk: As Gyula has already pointed out with the current design of
integration it should not be a problem. Even if we submitted programs to
the wrong clusters it would only cause performance issues.
Eventually it would be nice to have
So the current setup is to share results between the two apis by files. So
I dont see any reason why this couldnt work with the 2 cluster setup. It
makes deployment a little trickier but still feasible.
On Tue, Feb 17, 2015 at 11:55 AM, Ufuk Celebi wrote:
> I think this separation reflects the w
I think this separation reflects the way that Flink is used currently
anyways. I would be in favor of it as well.
- What about the ongoing efforts (I think by Gyula) to combine both the
batch and stream processing APIs? I assume that this would only effect the
performance and wouldn't pose a funda
+1
Let's do this soon to avoid performance issues for streaming.
On Tue, Feb 17, 2015 at 11:39 AM, Fabian Hueske wrote:
> sounds like a good idea to me.
> +1
>
> 2015-02-17 11:28 GMT+01:00 Stephan Ewen :
>
> > Hi everyone!
> >
> > What do you think about making the streaming execution mode of th
sounds like a good idea to me.
+1
2015-02-17 11:28 GMT+01:00 Stephan Ewen :
> Hi everyone!
>
> What do you think about making the streaming execution mode of the system
> explicit? That means that people start a Flink cluster explicitly in Batch
> mode or in Streaming mode.
>
> The rational behin
Hi everyone!
What do you think about making the streaming execution mode of the system
explicit? That means that people start a Flink cluster explicitly in Batch
mode or in Streaming mode.
The rational behind this idea is that I am not sure how batch and streaming
clusters are really shared in a
27 matches
Mail list logo
