Hi, thank you for putting our discussion to the mailing list. This is indeed where such discussions belong. For the others, we started discussing here: https://github.com/apache/flink/pull/304
I think there is one additional approach, which is probably close to (1): We only register those serializers by default which we don't see in the pre-flight phase (I think right now thats only GenericData.Array from Avro). We would come across all the other classes (Jodatime, Protobuf, Avro, Thrift, ...) when traversing the class hierarchy, as proposed in FLINK-1417. With this approach, users get the best out-of-the box experience and the number of registered classes / serializers is kept at a minimum. We can still offer means to register additional serializers (I think thats already merged to master). My main concern with this particular issue is a good out of the box user experience. If there is an issue with type serialization, users will notice it very early. (In my experience people often have their existing datatypes they use with other systems, and they want to continue using them) Therefore, I want to put some effort into making it as good as possible. I would actually sacrifice performance over stability/usability here. Our system is flexible enough to replace it later with a more efficient serialization if that becomes an issue. But maybe my suggestion above is already sufficient. We could also think about introducing a configuration variable which allows users to disable the default serializers. Regarding the second question: Is there a downside registering all types for tagging? We reduce the overall I/O which is good for performance. Best, Robert On Mon, Jan 19, 2015 at 8:24 PM, Stephan Ewen <se...@apache.org> wrote: > Hi all! > > We have various pending pull requests that add support for certain types by > adding extra kryo serializers. > > I think we need to decide how we want to handle the support for extra > types, because more are certainly to come. > > As I understand it, we have three broad options: > > (1) > Add as many serializers to Kryo by default as possible. > Pro: > - Many types work out of the box > Contra: > - We may eventually overload the kryo registry with serializers > that are not needed for most cases and suffer in performance > - It is hard to guess which types work out of the box (intransparent) > > > (2) > We create a collection of serializers and a registration util. > -------- > val env = ExecutionEnvironemnt.getExecutionEnviroment() > > Serializers.registerProtoBufSerializers(env); > Serializers.registerJavaUtilSerializers(env); > --------- > Pro: > - Easy for users > - We can grow the set of supported types very large without overloading > Kryo > - It is transparent what gets registered > > Contra: > - Not quite as convenient as if things just run > > > (3) > We do nothing and let the user create and register whatever is needed. > > We could have a library and utility for serializers for certain libraries. > Users could use this to conveniently add serializers for the libraries they > use. > Pro: > - Simple for us ;-) > Contra: > - More repeated work for users > > > ======================== > > For approach (1) and (2), there is an orthogonal question of whether we > want to simply register default serializers (that enable that types work) > or also register types for tags, to speed up the serialization of those > types. > > > Greetings, > Stephan >
