That sounds promising.
Yet, I have the problem that I need the candidates in different operators.
While feeding them forward is probably easy, e.g., via broadcasts, feeding the
candidates “backwards” to the next iteration seems to be more of a problem.
As I am only building a prototype, I might
>From the Flink documentation:
Conditions for a class to be treated as a POJO by Flink:
- The class must be public
- It must have a public constructor without arguments
- All fields either have to be public or there must be getters and
setters for all non-public fields.
In your example, the z
UDFs exist intentionally across iterations, it is a feature, to allow you
to keep state. To Figure out when an iteration starts and ends, you can use
a RichFunctions, which get calls to open() and close() for each iteration.
On Wed, Feb 11, 2015 at 10:40 AM, Kruse, Sebastian
wrote:
> Thanks for
Thanks for your answers.
I am trying to build an apriori-like algorithm to find key candidates in a
relational dataset. I was considering delta iterations, because the algorithm
should maintain two datasets: a set of column combinations to be checked (as
delta set) and a set of tuples which are
In case you want to contribute or follow the discussion, here's the JIRA:
https://issues.apache.org/jira/browse/FLINK-1511
Again, thanks for reporting!
2015-02-11 9:59 GMT+01:00 Fabian Hueske :
> Hi,
>
> you are right, there is a problem. I reproduced the problem and it seems
> that regular Scal
You can also use a bulk iteration and just keep the state yourself. Since
the functions love across iterations, it is easily doable to just gather
the state in a HashMap yourself. Use map(), or mapPartition(), a manual
partition() call - that should do the trick...
Am 10.02.2015 21:44 schrieb "Alex
Hi,
you are right, there is a problem. I reproduced the problem and it seems
that regular Scala classes are not properly analyzed and not identified as
Pojos.
As long as you only need these classes to be data holders without custom
logic, you could go with case classes.
I will open a JIRA to exte
