I'm not sure how the code generation and compilation works in detail, but aren't you able to write a class file as the result of the compilation? Then this class file could be uploaded to the JM and from there to all TMs. The class basically becomes another job dependency like the user code jar and will then be part of the user code class loader. Then the deserialization should be possible, right? But I'm not sure whether you have the user code class loader available at the places where you deserialize the code.
Cheers, Till On Wed, Feb 8, 2017 at 10:54 PM, Greg Hogan <c...@greghogan.com> wrote: > Hi Pat, Serkan, and Gábor, > > This looks very nice. I'll treat this like a pre-FLIP and ask my question > here. > > Do I understand correctly that the generated code is only dependent on the > length of the sort key? So we could separate the writing and reading of > keys and records and from the generated code to compare and swap. Most jobs > will use similar key lengths so the generated code could be stored in an > LRU cache which would not impact startup performance. > > From the document, "we have found that the DividedByConstant approach is > almost as good as UsingBitwiseOperators, and does not require > indexEntriesPerSegment to be a power of 2". How do you propose computing > indexEntriesPerSegment? Is the improved performance not dependent on this > value being a power-of-2? > > Are you able to compare with FLINK-3722 [1]? A disadvantage of both > DividedByConstant and UsingBitwiseOperators when using a power-of-2 is that > up to half of the sort memory can be unused. FLINK-3722 paginates QuickSort > to use increment and decrement in place of divide and modulus. > > [1] https://github.com/apache/flink/pull/2628/files > > Greg > > On Tue, Feb 7, 2017 at 4:34 PM, Pattarawat Chormai <pat.chor...@gmail.com> > wrote: > > > Hi all, > > > > We’re working on FLINK-5734 : Implement code generation for > > NormalizedKeySorter. We have just finished a design document that > describes > > how we’re going to proceed this implementation. > > > > Because this implement will touch several important parts of Flink, so we > > would appreciate if you could give us some feedback before starting > > implementation. > > Here is the link to the document : https://docs.google.com/document/d/ > > 1anGQhBn9qI0yqe7twVvrDIiym4U4gxalJkZzM4Ar4QM/edit?usp=sharing < > > https://docs.google.com/document/d/1anGQhBn9qI0yqe7twVvrDIiym4U4g > > xalJkZzM4Ar4QM/edit?usp=sharing> > > > > Also, we would like to confirm that NormalizedKeySorter is always > > instantiated at TaskManager, but we could not find any solution to verify > > that yet. If you know how to do it, we would be very pleased to hear > that. > > > > Thank you in advance for your feedback. > > > > Best regards, > > Serkan, Gábor and Pat >
