Hi jingsong,
Watermark is not a standard syntax, that's why we had a FLIP and long
discussion to add it to
Flink's SQL syntax. I think if we want to add INSERT OVERWRITE and
PARTITION syntax to
Flink's own syntax, we also need a FLIP or a VOTE, and this may can't
happen soon (we should
hear more people's opinions on this).
Regarding to the sql-dialect configuration, I was not saying to involve the
whole FLIP-89. I mean we can just
start a VOTE to expose it as `table.planner.sql-dialect` and include it in
1.10. The change can be very small, by
adding a ConfigOption and changing the implementation of
TableConfig#getSqlDialect/setSqlDialect. I believe
it is smaller and safer than changing the parser.
Btw, I cc'ed Yu Li and Gary into the discussion, because release managers
should be aware of this.
Best,
Jark
On Thu, 12 Dec 2019 at 11:47, Danny Chan <yuzhao....@gmail.com> wrote:
> Thanks Jingsong for bring up this discussion ~
>
> After reviewing FLIP-63, it seems that we have made a conclusion for the
> syntax
>
> - INSERT OVERWRITE ...
> - INSERT INTO … PARTITION
>
> Which means that they should not have the Hive dialect limitation, so I’m
> inclined that the behaviors for SQL-CLI is unexpected, or a “bug” that need
> to fix.
>
> We did not make a conclusion for the syntax:
>
> - CREATE TABLE … PARTITIONED BY ...
>
> Which means that the behavior of it is under-discussion, so it is okey to
> be without the HIVE dialect limitation, we do not actually have any table
> sources/sinks that support such a DDL so for current code base, users
> should not be influenced by the behaviors change.
>
> So I’m
>
> +1 to remove the hive dialect limitations for INSERT OVERWRITE and INSERT
> PARTITION
> +0 to add yaml dialect conf to SQL-CLI because FLIP-89 is not finished
> yet, we better do this until FLIP-89 is resolved.
>
> Best,
> Danny Chan
> 在 2019年12月11日 +0800 PM5:29，Jingsong Li <jingsongl...@gmail.com>，写道：
> > Hi Dev,
> >
> > After cutting out the branch of 1.10, I tried the following functions of
> > SQL-CLI and found that it does not support:
> > - insert overwrite
> > - PARTITION (partcol1=val1, partcol2=val2 ...)
> > The SQL pattern is:
> > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1,
> > partcol2=val2 ...) select_statement1 FROM from_statement;
> > It is a surprise to me.
> > The reason is that we only allow these two grammars in hive dialect. And
> > SQL-CLI does not have an interface to switch dialects.
> >
> > Because it directly hinders the SQL-CLI's insert syntax in hive
> integration
> > and seriously hinders the practicability of SQL-CLI.
> > And we have introduced these two grammars in FLIP-63 [1] to Flink.
> > Here are my question:
> > 1.Should we remove hive dialect limitation for these two grammars?
> > 2.Should we fix this in 1.10?
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
> >
> > Best,
> > Jingsong Lee
>
