I understand your argument about performance, though I am unsure it will be
that noticeable and warrants the added complexity,
Besides performance, I believe the other two arguments in the previous
mails are quite important to consider:
- Keeping complexity out of the core is a good goal. It is
Hi Stephan,
In terms of the performance concern, please see my understanding below.
## Breaking the pipeline v.s. adding a sink.
If two operators are initially chained, they will belong to the same stage
in the DAG and the same task, therefore the main processing path will just
have one task with
## About the improvements you mentioned in (1)
- I am not sure that this helps to improve performance by avoiding to
break the pipeline.
Attaching an additional sink, would in virtually any case add even more
overhead than the pipeline breaking.
What is your reasoning why it would be fas
Hi Stephan,
Sorry for the belated reply. You are right that the functionality proposed
in this FLIP can be implemented out of the Flink core as an ecosystem
project.
The main motivation of this FLIP is two folds:
1. Improve the performance of intermediate result sharing in the same
session.
Usin
Sorry for the late response. So many FLIPs these days.
I am a bit unsure about the motivation here, and that this need to be a
part of Flink. It sounds like this can be perfectly built around Flink as a
minimal library on top of it, without any change in the core APIs or
runtime.
The proposal to
Hi folks,
I would like to start the FLIP discussion thread about the pluggable
intermediate result storage.
This is phase 2 of FLIP-36: Support Interactive Programming in Flink Skip to
end of metadata. While the FLIP-36 provides a default implementation of the
intermediate result storage using
