Hi,
the following code should do what you want.
I included an implementation of an IdMapper.
At the end, I print the execution plan which is generated after the
optimization (so the pipeline is working until then).
Best, Fabian
val data: Seq[Seq[Int]] = (1 until 315).map(i => Seq(1, 2, 3))
val
Dear Fabian,
Thanks to your answer (I think you said same in StackOverflow) but as you
see in my code your solution does not work anymore:
Here is the code, it's split the datasets to list (each list contains
maximum 60 datasets)
After that, I reduce the dataset using union and map with an IdMap
Hi b0c1,
This is an limitation in Flink's optimizer.
Internally, all binary unions are merged into a single n-ary union. The
optimizer restricts the number of inputs for an operator to 64.
You can work around this limitation with an identity mapper which prevents
the union operators from merging:
Hi guys!
I have one input (from mongo) and I split the incoming data to multiple
datasets (each created dynamically from configuration) and before I write
back the result I want to merge it to one dataset (there is some common
transformation).
so the flow:
DataSet from Mongod =>
Create Mappers dy
