HI All:
I am sorry about working with
StreamConfig(https://github.com/apache/flink/pull/4241) which may conflicts
with others' work before discussing.
Motivation:
A Task contains one or more operators with chainning, however configs
of operator and task are all put in StreamConfig. For example, when an opeator
is setup with the StreamConfig, it can see the interface about physicalEdges or
chained.task.configs, which are confused. Similarly a streamTask should not
see the interface about chain.index.
So we need to separate OperatorConfig from StreamConfig. A streamTask
inits with the streamConfig, and then extracts operatorConfigs from it, build
streamOperators with every operatorConfig.
OperatorConfig: for the streamOperator to setup with, it constains
informations that only belong to the streamOperator. It contains:
1) operator information: name, id
2) streamOperator
3) input serializer.
4) output edges and serializers.
5) chain.index
6) state.key.serializer
StreamConfig: for the streamTask to use:
1) in.physical.edges
2) out.physical.edges
3) chained OperatorConfigs
4) execution environment: checkpoint, state.backend and so on...
Proposed Change
I propose overall changes:
1) Builde jobGraph from streamGraph
2) StreamOperator is setup with a operatorConfig, so the setup interface
need to change
(1) Build jobGraph from streamGraph
When building, first we get every operatorConfig from the streamNode.
And then put operatorConfigs of streamNodes to a streamConfig when we chain
them to a jobVertex.
(2) StreamOperator setup with OperatorProperties
An OperatorConfig is provided instead of streamConfig when the
streamOperator sets up. Thanks to the advice of StephanEwan, OperatorConfig is
no need to have a Map of "configKey" to values, just is a serializable class
with the respective fields, And StreamConfig still relys on an underlying
Configuration, because the streamConfig flows by its underlying configuration.
There are people who have already thought about this, maybe someone has
been working on it. I need your advice.
Thanks a lot for replying and Best Regards.
JiPing
