Re: Objects accessible from all Flink nodes

2017-01-13 Thread Fabian Hueske
Hi Matt, it is not possible to share an object across different task of the same operator or even different operators. This would be globally mutable state which is in general hard to get efficient in distributed systems. Something that might work is to use a CoFlatMapOperator with one input bein

Re: Objects accessible from all Flink nodes

2017-01-13 Thread Matt
Errata: How can an *object (such as the classifier, line 1)* be accessed by any Flink node [...] Just in case, the classifier itself can't be serialized I believe, it's part of a framework which I can't modify. In any case, even if it's serialized, I guess the cost of moving it to one node and the

Objects accessible from all Flink nodes

2017-01-12 Thread Matt
Hello, I have a stream of objects which I use to update the model of a classification algorithm and another stream with the objects I need to classify in real time. The problem is that the instances for training and evaluation are processed on potentially different Flink nodes, but the classifier