Hi David,
* Yes, we’re taking the approach of having Calcite depend on flink-core and flink-table-code-splitter. So not really porting now, just utilizing this component in Flink. * I don’t really have an answer for the question about Janino post-processing optimizations. * We’re planning to use the same default max length for generated code as Flink but also make it user-configurable. From: David Radley <david_rad...@uk.ibm.com> Date: Tuesday, September 10, 2024 at 6:29 AM To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: [DISCUSS]: Flink code generation porting to Calcite Hi James, Some thoughts on this, * The title of this email chain is “Flink code generation porting to Calcite”. It sounds like we are now considering leaving the Flink module in Flink and bringing it in as a dependancy? Would this mean that for example Flink 2 would have a dependency on a version of Calcite that brought in Flink 1.20 dependency. When you say “I would prefer the approach of using the Flink module as a dependency so that fixes are propagated more easily.” To upgrade the code splitter would still require Calcite to be upgraded to specify the new dependency. This would be mitigated somewhat if the Calcite dependency on the Flink module was a Maven provided dependency. * Could the code generator (Janino) be updated to not need this post processing optimizations (with config for the TableConfigOptions.MAX_LENGTH_GENERATED_CODE change) ? * Are there any other reasons why the Flink rewriters need to something different from the standard code generator (TableConfigOptions.MAX_LENGTH_GENERATED_CODE if something that is Flink specific at the moment, are there other cases like this)? WDYT? Kind regards, David. From: James Duong <james.du...@improving.com.INVALID> Date: Friday, 2 August 2024 at 23:52 To: dev@flink.apache.org <dev@flink.apache.org> Subject: [EXTERNAL] Re: [DISCUSS]: Flink code generation porting to Calcite Thanks Sergey. I’m treating JavaCodeSplitter.split() as the “entry point” for the code-splitter. This method fails if I exclude flink-core because it uses the (Flink) Preconditions class at runtime. I’m prototyping this work in the following Calcite PR: https://github.com/apache/calcite/pull/3901 From: Sergey Nuyanzin <snuyan...@gmail.com> Date: Thursday, July 25, 2024 at 12:40 PM To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: [DISCUSS]: Flink code generation porting to Calcite thanks for the clarification given the fact how flink annotations are used across Flink repo I don't think it is a right way to not use it here as well... At the same time they are not used at runtime and I guess could be excluded I tried with "hello world" code split app and the way of declaring dependencies as below and it seems working. . Could it be a way to go for you? <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-code-splitter</artifactId> <version>${code-splitter.version}</version> <exclusions> <exclusion> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> </exclusion> </exclusions> </dependency> PS. IIRC currently there is no circular deps so far since there are only few modules depending on Calcite flink-connector-hive, flink-sql-parser, flink-sql-client, flink-table-calcite-bridge, flink-table-planner On Thu, Jul 25, 2024 at 2:09 PM James Duong <james.du...@improving.com.invalid> wrote: > > Hi Sergey, > > I would prefer the approach of using the Flink module as a dependency so that > fixes are propagated more easily. > > From: Sergey Nuyanzin <snuyan...@gmail.com> > Date: Thursday, July 25, 2024 at 12:44 AM > To: dev@flink.apache.org <dev@flink.apache.org> > Subject: Re: [DISCUSS]: Flink code generation porting to Calcite > Hi James > > can you please elaborate about your scenario since I feel getting lost > with this. > > from one side the thread is called > >Flink code generation porting to Calcite > from another side it was mentioned > >We really want to avoid having circular dependencies between Flink and > >Calcite. > > are you going to use it like a dependency or to make a port of it in Calcite? > Or may be something else... > > On Thu, Jul 25, 2024 at 12:34 AM James Duong > <james.du...@improving.com.invalid> wrote: > > > > This is really helpful. The JavaCodeSplitter looks fairly easy to use. > > Thanks Jark! > > > > Right now I’m trying to work out the best way to integrate this module from > > an architecture perspective. It looks like there aren’t very many Flink > > dependencies within the module. > > > > We really want to avoid having circular dependencies between Flink and > > Calcite. As you mentioned before, code-splitting is a common need and shows > > up in projects with code generation. How would you feel about further > > de-coupling the code from Flink? There are a few minor dependencies on > > Flink annotations and shaded Preconditions. > > > > From: Jark Wu <imj...@gmail.com> > > Date: Wednesday, July 17, 2024 at 7:08 PM > > To: dev@flink.apache.org <dev@flink.apache.org> > > Subject: Re: [DISCUSS]: Flink code generation porting to Calcite > > Hi James, > > > > After reading the comments in CALCITE-3094, I think what you are looking > > for is the Flink code-splitting tools. > > Code splitting is a common need for Java code generation and Flink has > > extracted the code splitting into a > > separate module "flink-table-code-splitter"[1] with little dependencies. > > And this is how Flink SQL use it [2]. > > > > Best, > > Jark > > > > [1]: > > https://github.com/apache/flink/blob/master/flink-table/flink-table-code-splitter/pom.xml > > [2]: > > https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/generated/GeneratedClass.java#L58 > > > > On Thu, 18 Jul 2024 at 07:12, James Duong > > <james.du...@improving.com.invalid> > > wrote: > > > > > Hi Flink developer community, > > > > > > I’m contributing to the Apache Calcite project and there’s interest in > > > making use of Flink’s code generation. See the comments on > > > https://issues.apache.org/jira/browse/CALCITE-3094 > > > > > > Would someone be able to point me to where Flink integrates Janino? I > > > originally thought it would be related to this class: > > > https://nightlies.apache.org/flink/flink-docs-release-1.3/api/java/org/apache/flink/table/codegen/CodeGenerator.html > > > > > > But I haven’t been able to find this on main. > > > > > > Thanks! > > > > > Warning: The sender of this message could not be validated and may not be > > the actual sender. > > > > -- > Best regards, > Sergey > Warning: The sender of this message could not be validated and may not be the > actual sender. -- Best regards, Sergey Warning: The sender of this message could not be validated and may not be the actual sender. Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU Warning: The sender of this message could not be validated and may not be the actual sender.
