Hello Till, > Why did you decide to generate the code on the TMs?
If we generated on the client side, then we would need to serialize instances of the generated classes when shipping the job to the TMs, but we would really like to avoid serializing instances of the generated classes. In the other code generation effort [1] (which is for the serializers and comparators), we tried to do the generation on the client side, because the serializers and comparators get instantiated on the client side. However, it turned out that serializing and shipping instances of a generated class to an other machine is a really big mess: When you want to deserialize an object on an other machine then you need to be able to load the class of that object on that machine. However, the classloaders on that machine don't know about the class that was generated in the client machine. (We ended up including the source code of the generated class into the serialization as a string, and essentially redoing the generation in readObject.) Best, Gábor [1] https://github.com/apache/flink/pull/2211 2017-02-08 12:01 GMT+01:00 Till Rohrmann <trohrm...@apache.org>: > Hi Pat, Serkan and Gabor, > > I really like your design document and the preliminary results look really > good. Impressive! I think your document can be converted into a FLIP. > > I was wondering whether it makes sense (or not) to generate the code on the > client side? Then we would not have to introduce the `TemplateManager`, > which avoids increasing the complexity of the TM, and we wouldn't need to > ship TypeInformation to the TMs (I know this already happens but initially > the idea was that TypeInformation is only used in the pre-flight phase). > Why did you decide to generate the code on the TMs? > > Cheers, > Till > > > > > > On Tue, Feb 7, 2017 at 10:34 PM, Pattarawat Chormai <pat.chor...@gmail.com> > wrote: > >> Hi all, >> >> We’re working on FLINK-5734 : Implement code generation for >> NormalizedKeySorter. We have just finished a design document that describes >> how we’re going to proceed this implementation. >> >> Because this implement will touch several important parts of Flink, so we >> would appreciate if you could give us some feedback before starting >> implementation. >> Here is the link to the document : https://docs.google.com/document/d/ >> 1anGQhBn9qI0yqe7twVvrDIiym4U4gxalJkZzM4Ar4QM/edit?usp=sharing < >> https://docs.google.com/document/d/1anGQhBn9qI0yqe7twVvrDIiym4U4g >> xalJkZzM4Ar4QM/edit?usp=sharing> >> >> Also, we would like to confirm that NormalizedKeySorter is always >> instantiated at TaskManager, but we could not find any solution to verify >> that yet. If you know how to do it, we would be very pleased to hear that. >> >> Thank you in advance for your feedback. >> >> Best regards, >> Serkan, Gábor and Pat
