Hi Shuyi,
I'm wondering if you folks still have the bandwidth working on this.
We have some dedicated resource and like to move this forward. We can
collaborate.
Thanks,
Xuefu
------------------------------------------------------------------
发件人：wenlong.lwl<wenlong88....@gmail.com>
日 期：2018年11月05日 11:15:35
收件人：<dev@flink.apache.org>
主 题：Re: [DISCUSS] Flink SQL DDL Design
Hi, Shuyi, thanks for the proposal.
I have two concerns about the table ddl:
1. how about remove the source/sink mark from the ddl, because it is not
necessary, the framework determine the table referred is a source or a sink
according to the context of the query using the table. it will be more
convenient for use defining a table which can be both a source and sink,
and more convenient for catalog to persistent and manage the meta infos.
2. how about just keeping one pure string map as parameters for table, like
create tabe Kafka10SourceTable (
intField INTEGER,
stringField VARCHAR(128),
longField BIGINT,
rowTimeField TIMESTAMP
) with (
connector.type = ’kafka’,
connector.property-version = ’1’,
connector.version = ’0.10’,
connector.properties.topic = ‘test-kafka-topic’,
connector.properties.startup-mode = ‘latest-offset’,
connector.properties.specific-offset = ‘offset’,
format.type = 'json'
format.prperties.version=’1’,
format.derive-schema = 'true'
);
Because:
1. in TableFactory, what user use is a string map properties, defining
parameters by string-map can be the closest way to mapping how user use the
parameters.
2. The table descriptor can be extended by user, like what is done in Kafka
and Json, it means that the parameter keys in connector or format can be
different in different implementation, we can not restrict the key in a
specified set, so we need a map in connector scope and a map in
connector.properties scope. why not just give user a single map, let them
put parameters in a format they like, which is also the simplest way to
implement DDL parser.
3. whether we can define a format clause or not, depends on the
implementation of the connector, using different clause in DDL may make a
misunderstanding that we can combine the connectors with arbitrary formats,
which may not work actually.
On Sun, 4 Nov 2018 at 18:25, Dominik Wosiński <wos...@gmail.com> wrote:
+1, Thanks for the proposal.
I guess this is a long-awaited change. This can vastly increase the
functionalities of the SQL Client as it will be possible to use complex
extensions like for example those provided by Apache Bahir[1].
Best Regards,
Dom.
[1]
https://github.com/apache/bahir-flink
sob., 3 lis 2018 o 17:17 Rong Rong <walter...@gmail.com> napisał(a):
+1. Thanks for putting the proposal together Shuyi.
DDL has been brought up in a couple of times previously [1,2].
Utilizing
DDL will definitely be a great extension to the current Flink SQL to
systematically support some of the previously brought up features such
as
[3]. And it will also be beneficial to see the document closely aligned
with the previous discussion for unified SQL connector API [4].
I also left a few comments on the doc. Looking forward to the alignment
with the other couple of efforts and contributing to them!
Best,
Rong
[1]
http://mail-archives.apache.org/mod_mbox/flink-dev/201805.mbox/%3CCAMZk55ZTJA7MkCK1Qu4gLPu1P9neqCfHZtTcgLfrFjfO4Xv5YQ%40mail.gmail.com%3E
[2]
http://mail-archives.apache.org/mod_mbox/flink-dev/201810.mbox/%3CDC070534-0782-4AFD-8A85-8A82B384B8F7%40gmail.com%3E
[3] https://issues.apache.org/jira/browse/FLINK-8003
[4]
http://mail-archives.apache.org/mod_mbox/flink-dev/201810.mbox/%3c6676cb66-6f31-23e1-eff5-2e9c19f88...@apache.org%3E
On Fri, Nov 2, 2018 at 10:22 AM Bowen Li <bowenl...@gmail.com> wrote:
Thanks Shuyi!
I left some comments there. I think the design of SQL DDL and
Flink-Hive
integration/External catalog enhancements will work closely with each
other. Hope we are well aligned on the directions of the two designs,
and I
look forward to working with you guys on both!
Bowen
On Thu, Nov 1, 2018 at 10:57 PM Shuyi Chen <suez1...@gmail.com>
wrote:
Hi everyone,
SQL DDL support has been a long-time ask from the community.
Current
Flink
SQL support only DML (e.g. SELECT and INSERT statements). In its
current
form, Flink SQL users still need to define/create table sources and
sinks
programmatically in Java/Scala. Also, in SQL Client, without DDL
support,
the current implementation does not allow dynamical creation of
table,
type
or functions with SQL, this adds friction for its adoption.
I drafted a design doc [1] with a few other community members that
proposes
the design and implementation for adding DDL support in Flink. The
initial
design considers DDL for table, view, type, library and function.
It
will
be great to get feedback on the design from the community, and
align
with
latest effort in unified SQL connector API [2] and Flink Hive
integration
[3].
Any feedback is highly appreciated.
Thanks
Shuyi Chen
[1]
https://docs.google.com/document/d/1TTP-GCC8wSsibJaSUyFZ_5NBAHYEB1FVmPpP7RgDGBA/edit?usp=sharing
[2]
https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit?usp=sharing
[3]
https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit?usp=sharing
--
"So you have to trust that the dots will somehow connect in your
future."
