Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-03-08 Thread Greg Hogan
tting PR to Flink repo. > > Best, > Pat > > [1] https://github.com/heytitle/flink/pull/1 > [2] > https://docs.google.com/document/d/1anGQhBn9qI0yqe7twVvrDIiym4U4gxalJkZzM4Ar4QM > > > > -- > View this message in context: > http://apache-flink-mailin

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-03-07 Thread pat.chormai
actually submitting PR to Flink repo. Best, Pat [1] https://github.com/heytitle/flink/pull/1 [2] https://docs.google.com/document/d/1anGQhBn9qI0yqe7twVvrDIiym4U4gxalJkZzM4Ar4QM -- View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/FLINK-5734-Code-Generation

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-14 Thread Greg Hogan
e-flink-mailing-list-archive.1008284.n3. > nabble.com/FLINK-5734-Code-Generation-for-NormalizedKeySorter-tt15804. > html#a15936 > Maybe the problem is caused by the formatting. > > > FLINK-3722 > > approach seems to be the fastest one. > > OK, then I would suggest to

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-14 Thread Gábor Gévay
Hello, Pat, the table in your email is somehow not visible in my gmail, but it is visible here: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/FLINK-5734-Code-Generation-for-NormalizedKeySorter-tt15804.html#a15936 Maybe the problem is caused by the formatting. > FLINK-3

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-14 Thread pat.chormai
.nabble.com/FLINK-5734-Code-Generation-for-NormalizedKeySorter-tp15804p15936.html Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-09 Thread Greg Hogan
l/ > optimization/DividedByConstant.java#L353 > [2] > https://blogs.msdn.microsoft.com/devdev/2005/12/12/integer- > division-by-constants/ > > > > -- > View this message in context: http://apache-flink-mailing- > list-archive.1008284.n3.nabble.com/FLINK-5734-Code-Generation-for- > NormalizedKeySorter-tp15804p15860.html > Sent from the Apache Flink Mailing List archive. mailing list archive at > Nabble.com. >

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-09 Thread pat.chormai
text: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/FLINK-5734-Code-Generation-for-NormalizedKeySorter-tp15804p15860.html Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-09 Thread Till Rohrmann
I'm not sure how the code generation and compilation works in detail, but aren't you able to write a class file as the result of the compilation? Then this class file could be uploaded to the JM and from there to all TMs. The class basically becomes another job dependency like the user code jar and

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-08 Thread Greg Hogan
Hi Pat, Serkan, and Gábor, This looks very nice. I'll treat this like a pre-FLIP and ask my question here. Do I understand correctly that the generated code is only dependent on the length of the sort key? So we could separate the writing and reading of keys and records and from the generated cod

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-08 Thread Gábor Gévay
Hello Till, > Why did you decide to generate the code on the TMs? If we generated on the client side, then we would need to serialize instances of the generated classes when shipping the job to the TMs, but we would really like to avoid serializing instances of the generated classes. In the other

Re: FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-08 Thread Till Rohrmann
Hi Pat, Serkan and Gabor, I really like your design document and the preliminary results look really good. Impressive! I think your document can be converted into a FLIP. I was wondering whether it makes sense (or not) to generate the code on the client side? Then we would not have to introduce t

FLINK-5734 : Code Generation for NormalizedKeySorter

2017-02-07 Thread Pattarawat Chormai
Hi all, We’re working on FLINK-5734 : Implement code generation for NormalizedKeySorter. We have just finished a design document that describes how we’re going to proceed this implementation. Because this implement will touch several important parts of Flink, so we would appreciate if you coul