Kostas Kloudas created FLINK-6215:
-------------------------------------
Summary: Make the StatefulSequenceSource scalable.
Key: FLINK-6215
URL: https://issues.apache.org/jira/browse/FLINK-6215
Project: Flink
Issue Type: Bug
Components: DataStream API
Affects Versions: 1.3.0
Reporter: Kostas Kloudas
Fix For: 1.3.0
Currently the {{StatefulSequenceSource}} instantiates all the elements to emit
first and keeps them in memory. This is not scalable as for large sequences of
elements this can lead to out of memory exceptions.
To solve this, we can pre-partition the sequence of elements based on the
{{maxParallelism}} parameter, and just keep state (to checkpoint) per such
partition.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
