Hi Pat, Serkan, and Gábor, This looks very nice. I'll treat this like a pre-FLIP and ask my question here.
Do I understand correctly that the generated code is only dependent on the length of the sort key? So we could separate the writing and reading of keys and records and from the generated code to compare and swap. Most jobs will use similar key lengths so the generated code could be stored in an LRU cache which would not impact startup performance. >From the document, "we have found that the DividedByConstant approach is almost as good as UsingBitwiseOperators, and does not require indexEntriesPerSegment to be a power of 2". How do you propose computing indexEntriesPerSegment? Is the improved performance not dependent on this value being a power-of-2? Are you able to compare with FLINK-3722 [1]? A disadvantage of both DividedByConstant and UsingBitwiseOperators when using a power-of-2 is that up to half of the sort memory can be unused. FLINK-3722 paginates QuickSort to use increment and decrement in place of divide and modulus. [1] https://github.com/apache/flink/pull/2628/files Greg On Tue, Feb 7, 2017 at 4:34 PM, Pattarawat Chormai <pat.chor...@gmail.com> wrote: > Hi all, > > We’re working on FLINK-5734 : Implement code generation for > NormalizedKeySorter. We have just finished a design document that describes > how we’re going to proceed this implementation. > > Because this implement will touch several important parts of Flink, so we > would appreciate if you could give us some feedback before starting > implementation. > Here is the link to the document : https://docs.google.com/document/d/ > 1anGQhBn9qI0yqe7twVvrDIiym4U4gxalJkZzM4Ar4QM/edit?usp=sharing < > https://docs.google.com/document/d/1anGQhBn9qI0yqe7twVvrDIiym4U4g > xalJkZzM4Ar4QM/edit?usp=sharing> > > Also, we would like to confirm that NormalizedKeySorter is always > instantiated at TaskManager, but we could not find any solution to verify > that yet. If you know how to do it, we would be very pleased to hear that. > > Thank you in advance for your feedback. > > Best regards, > Serkan, Gábor and Pat
