OK, that's really interesting. I hope you will add "Gluten-Trino <https://github.com/oap-project/Gluten-Trino>" and "Gluten-Flink" as sub-projects(plugin module) under the main Gluten project and follow a single release cycle?
Is it technically/architecturally feasible to develop a "Gluten-Hive" plugin (as we know Hive uses Apache Tez DAG) as a long term plan ? Thanks, Rakesh On Tue, Dec 26, 2023 at 4:30 PM ShaoFeng Shi <shaofeng...@apache.org> wrote: > Hi Rakesh, > > Thanks for your comment. The Trino plugin is under PoC, not ready for > widely use at this moment, so it is staged in another git repository: > https://github.com/oap-project/Gluten-Trino > Flink support is still in early discussion stage, need more collaboration > in the future. > > Best regards, > > Shaofeng Shi 史少锋 > Apache Kylin PMC, > Apache Incubator PMC, > Email: shaofeng...@apache.org > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html > Join Kylin user mail group: user-subscr...@kylin.apache.org > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > Rakesh Radhakrishnan <rake...@apache.org> 于2023年12月26日周二 11:24写道： > > > Thanks Shaofeng and team for the efforts! > > > > Its really interesting to see that Flink and Trino projects can get the > > benefits of Gluten and boost their performance. Much appreciated, if you > > could share more on this integration, perf_numbers, > pullrequest_reference, > > jira_reference etc. Will Flink and Trino already have a plugin mechanism > in > > place to seamlessly integrate with Gluten ? > > > > Thanks, > > Rakesh > > > > On Sat, Dec 23, 2023 at 6:13 PM ShaoFeng Shi <shaofeng...@apache.org> > > wrote: > > > > > Hi Enrico, > > > > > > This is a good question. Actually I asked the same to the team when > they > > > find me for coaching. > > > > > > As the proposal mentioned, Gluten is a middle layer between the > JVM-based > > > SQL engine and native libraries. It connects the two sides, and build > > > common services like unified plan transformation, seamless native > > > integration, clear JNI interfaces, etc. Spark is the first engine it > > > supports now, but the framework and mechanism can also work for other > > > engines like Trino, Flink SQL in the future (the proposal also > mentioned > > > this). Underlying Gluten, it can integrate with different libararies, > > like > > > Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope > is > > > different, that's why we put it as a new project. > > > > > > Best regards, > > > > > > Shaofeng Shi 史少锋 > > > Apache Kylin PMC, > > > Apache Incubator PMC, > > > Email: shaofeng...@apache.org > > > > > > Apache Kylin FAQ: > https://kylin.apache.org/docs/gettingstarted/faq.html > > > Join Kylin user mail group: user-subscr...@kylin.apache.org > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > > > > > > > > > > > Enrico Olivelli <eolive...@gmail.com> 于2023年12月22日周五 22:43写道： > > > > > > > Shaofeng, > > > > > > > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi > > > > <shaofeng...@apache.org> ha scritto: > > > > > > > > > > Hi IPMC members, > > > > > > > > > > I would like to propose a new project to the ASF incubator - > Gluten. > > > > > > > > > > Gluten[1] is a middle layer responsible for offloading Apache Spark > > SQL > > > > > queries to native engines. This project aims to address the CPU > > > > > computational bottleneck to offload SparkSQL operators to native > > > engines > > > > in > > > > > data loading scenarios based on Apache Spark. > > > > > > > > Have you considered making this project a subproject of Apache Spark > ? > > > > > > > > Enrico > > > > > > > > > > > > > > Here is the proposal - > > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal > > > > > > > > > > I would be the Champion of the project. I will mentor and help the > > > > project > > > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [ > > > > y...@apache.org] > > > > > and Wenli Zhang [ovi...@apache.org] . > > > > > > > > > > We are open to hearing the feedback from the incubator. > > > > > > > > > > Best, > > > > > Shaofeng Shi. > > > > > > > > > > [1] https://oap-project.github.io/gluten/ > > > > > > > > > > Best regards, > > > > > > > > > > Shaofeng Shi 史少锋 > > > > > Apache Kylin PMC, > > > > > Apache Incubator PMC, > > > > > Email: shaofeng...@apache.org > > > > > > > > > > Apache Kylin FAQ: > > > https://kylin.apache.org/docs/gettingstarted/faq.html > > > > > Join Kylin user mail group: user-subscr...@kylin.apache.org > > > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > > > > > > > >
