user@flink.apache.org
Betreff: Re: Writing Intermediates to disk
I do not know if I understand completely, but I would create a new DataSet
based on filtering the condition and then persist this DataSet.
So :
DataSet ds2 = DataSet1.filter(Condition)
2ds.output(...)
On Mon, May 9, 2016 at 11:09 AM,
I do not know if I understand completely, but I would create a new DataSet
based on filtering the condition and then persist this DataSet.
So :
DataSet ds2 = DataSet1.filter(Condition)
2ds.output(...)
On Mon, May 9, 2016 at 11:09 AM, Ufuk Celebi wrote:
> Flink has support for spillable in
Flink has support for spillable intermediate results. Currently they
are only set if necessary to avoid pipeline deadlocks.
You can force this via
env.getConfig().setExecutionMode(ExecutionMode.BATCH);
This will write shuffles to disk, but you don't get the fine-grained
control you probably need
Hi Mailing List,
I want to write and read intermediates to/from disk.
The following foo- codesnippet may illustrate my intention:
public void mapPartition(Iterable tuples, Collector out) {
for (T tuple : tuples) {
if (Condition)
