Hi everyone,
thanks for the positive feedback we received so far. I converted the
design document into a actual FLIP in the wiki. You can find it here:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
I'm in the process of creating the first JIRA issues to kick off the
preparation efforts. Most of the issues will be first or second level
children of FLINK-11439:
https://issues.apache.org/jira/browse/FLINK-11439
Feel free to further join the discussions and provide feedback. I'm sure
there will be a lot of stuff to discuss :-)
Thanks,
Timo
Am 25.01.19 um 16:43 schrieb fudian.fd:
Hi Timo,
Thanks a lot for bringing up the discussion and the detailed implementation
plan. The implementation plan makes much sense to me. +1 to the FLIP.
Regards,
Dian
在 2019年1月25日，下午11:00，Jark Wu <imj...@gmail.com> 写道：
Hi Timo,
Thanks for the detailed design. +1 for the FLIP.
It's very nice to have Blink planner and runtime as a plugin in the early
stages. This will keep flink-table stable as much as possible.
Best,
Jark
On Fri, 25 Jan 2019 at 22:26, Hequn Cheng <chenghe...@gmail.com> wrote:
Hi Timo,
+1 for the FLIP!
Great work and thanks a lot for the detailed document! The task dependency
graph is very helpful for teasing out relationships between tasks.
Looking forward to the JIRAs and hoping to contribute to it!
Best,
Hequn
On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski <pi...@da-platform.com>
wrote:
Hi,
+1 from my side for this plan. The proposed decoupling TypeSystem and
most
interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
merging MVP Blink's runtime gives us best shot of handling Blink merging
as
fluently and as painless as possible. I’m also looking forward to the
follow up discussions that will need to take place in order to achieve
the
final goal :)
Piotrek
On 25 Jan 2019, at 07:53, jincheng sun <sunjincheng...@gmail.com>
wrote:
Hi Timo,
Thanks a lot for bring up the FLIP-32 discussion and the very detailed
implementation plan document !
Restructure `flink-table` is an important part of flink merge blink,
looking forward to the JIRAs which will be opened !
Cheers，
Jincheng
Timo Walther <twal...@apache.org> 于2019年1月24日周四 下午9:06写道：
Hi everyone,
as Stephan already announced on the mailing list [1], the Flink
community will receive a big code contribution from Alibaba. The
flink-table module is one of the biggest parts that will receive many
new features and major architectural improvements. Instead of waiting
until the next major version of Flink or introducing big API-breaking
changes, we would like to gradually build up the Blink-based planner
and
runtime while keeping the Table & SQL API mostly stable. Users will be
able to play around with the current merge status of the new planner
or
fall back to the old planner until the new one is stable.
We have prepared a design document that discusses a restructuring of
the
flink-table module and suggests a rough implementation plan:
https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
I will briefly summarize the steps we would like to do:
- Split the flink-table module similar to the proposal of FLIP-28 [3]
which is outdated. This is a preparation to separate API from core
(targeted for Flink 1.8).
- Perform minor API changes to separate API from actual implementation
(targeted for Flink 1.8).
- Merge a MVP Blink SQL planner given that necessary Flink
core/runtime
changes have been completed.
The merging will happen in stages (e.g. basic planner framework,
then
operator by operator). The exact merging plan still needs to be
determined.
- Rework the type system in order to unblock work on unified table
environments, UDFs, sources/sinks, and catalog.
- Enable full end-to-end batch and stream execution features.
Our mid-term goal:
Run full TPC-DS on a unified batch/streaming runtime. Initially, we
will
only support ingesting data coming from the DataStream API. Once we
reworked the sources/sink interfaces, we will target full end-to-end
TPC-DS query execution with table connectors.
A rough task dependency graph is illustrated in the design document. A
more detailed task dependency structure will be added to JIRA after we
agreed on this FLIP.
Looking forward to any feedback.
Thanks,
Timo
[1]
https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
[2]
https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
