Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-05-29 Thread Gábor Horváth
Hi! I would like to give you some status updates on the Google Summer of Code project. I started to implement the proposed features [1]. Status of code generation in general: * I can compile the generated code using Janino compiler * I can load the compiled classes and use them * For some mysteri

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-23 Thread Gábor Horváth
Hi, The GSoC project proposal was accepted! Thank you for all your support. I will do my best to live up to the challenges and deliver everything that way planned for this summer. Best Regards, Gábor On 20 April 2016 at 16:18, Gábor Horváth wrote: > On the second thought I think you are right.

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-20 Thread Gábor Horváth
On the second thought I think you are right. I had the impression that there is cyclic dependency between TypeInformation and the serializers but that is not the case. So there is no rewrite needed for TypeInformation in order to be able to use Scala for serializers. According to the proposal unle

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-20 Thread Fabian Hueske
Why would you need to rewrite the TypeInformation in Scala? I think we need a way to replace Serializer implementations anyway unless the generated serializers are compatible to the current ones. 2016-04-20 15:53 GMT+02:00 Gábor Horváth : > Hi Fabian, > > I agree that it would be awesome to move

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-20 Thread Gábor Horváth
Hi Fabian, I agree that it would be awesome to move this to its own module/plugin. However in order to be able to write the code generation in Scala I would need to rewrite the type information to use Scala as well. I think I will not have time to do this during the summer, so I think I will stick

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-19 Thread Fabian Hueske
Hi Gabor, you are right, a codegen serializer module would depend on flink-core and in the current design flink-core would need to know about the type infos / serializers / comparators. Decoupling implementations of type info, serializers, and comparators from flink-core and resolving the cyclic

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-18 Thread Gábor Horváth
Unfortunately making code generation a separate module would introduce cyclic dependency. Code generation requires the TypeInfo which is available in flink-core and flink-core requires the generated serializers from the code generation module. Do you have a solution for this? I think if we can com

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-18 Thread Fabian Hueske
+1 for not mixing Java and Scala in flink-core. Maybe it makes sense to implement the code generated serializers / comparators as a separate module which can be plugged-in. This could be pure Scala. In general, I think it would be good to have some kind of "version management" for serializers in p

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-18 Thread Chiwan Park
Yes, I know Janino is a pure Java project. I meant if we add Scala code to flink-core, we should add Scala dependency to flink-core and it could be confusing. Regards, Chiwan Park > On Apr 18, 2016, at 2:49 PM, Márton Balassi wrote: > > Chiwan, just to clarify Janino is a Java project. [1] >

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-17 Thread Márton Balassi
Chiwan, just to clarify Janino is a Java project. [1] [1] https://github.com/aunkrig/janino On Mon, Apr 18, 2016 at 3:40 AM, Chiwan Park wrote: > I prefer to avoid Scala dependencies in flink-core. If flink-core includes > Scala dependencies, Scala version suffix (_2.10 or _2.11) should be adde

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-17 Thread Chiwan Park
I prefer to avoid Scala dependencies in flink-core. If flink-core includes Scala dependencies, Scala version suffix (_2.10 or _2.11) should be added. I think that users could be confused. Regards, Chiwan Park > On Apr 17, 2016, at 3:49 PM, Márton Balassi wrote: > > Hi Gábor, > > I think that

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-16 Thread Márton Balassi
Hi Gábor, I think that adding the Janino dep to flink-core should be fine, as it has quite slim dependencies [1,2] which are generally orthogonal to Flink's main dependency line (also it is already used elsewhere). As for mixing Scala code that is used from the Java parts of the same maven module

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-04-16 Thread Gábor Horváth
Hi! Table API already uses code generation and the Janino compiler [1]. Is it a dependency that is ok to add to flink-core? In case it is ok, I think I will use the same in order to be consistent with the other code generation efforts. I started to look at the Table API code generation [2] and it

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-03-19 Thread Gábor Horváth
Thank you! I finalized the project. On 18 March 2016 at 10:29, Márton Balassi wrote: > Thanks Gábor, now I also see it on the internal GSoC interface. I have > indicated that I wish to mentor your project, I think you can hit finalize > on your project there. > > On Mon, Mar 14, 2016 at 11:16 AM

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-03-19 Thread Márton Balassi
Thanks Gábor, now I also see it on the internal GSoC interface. I have indicated that I wish to mentor your project, I think you can hit finalize on your project there. On Mon, Mar 14, 2016 at 11:16 AM, Gábor Horváth wrote: > Hi, > > I have updated this draft to include preliminary benchmarks, m

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-03-14 Thread Gábor Horváth
Hi, I have updated this draft to include preliminary benchmarks, mentioned the interaction of annotations with savepoints, extended it with a timeline, and some notes about scala case classes. Regards, Gábor On 9 March 2016 at 16:12, Gábor Horváth wrote: > Hi! > > As far as I can see the forma

Re: GSoC Project Proposal Draft: Code Generation in Serializers

2016-03-09 Thread Gábor Horváth
Hi! As far as I can see the formatting was not correct in my previous mail. A better formatted version is available here: https://docs.google.com/document/d/1VC8lCeErx9kI5lCMPiUn625PO0rxR-iKlVqtt3hkVnk Sorry for that. Regards, Gábor On 9 March 2016 at 15:51, Gábor Horváth wrote: > Hi,I did not

GSoC Project Proposal Draft: Code Generation in Serializers

2016-03-09 Thread Gábor Horváth
Hi,I did not want to send this proposal out before the I have some initial benchmarks, but this issue was mentioned on the mailing list ( http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Tuple-performance-and-the-curious-JIT-compiler-td10666.html), and I wanted to make this informatio