Hi Jark, > The dialect restriction is introduced on purpose, because OVERWRITE and PARTITION syntax are not SQL standard.
My understanding is that watermark [1] is also a non-standard grammar. We can extend SQL standard syntax. > Even in the discussion of FLIP-63, the community have different opinion on whether the partition fields should be declared in schema part or not [1]. The status of FLIP-63 is that limit the proposed syntax in Hive dialect, and we may propose new built-in syntax for creating partition tables in the future. No, I'm not talking about creating partition tables. I'm talking about inserting tables. FLIP-63 doesn't mean it's only for hive dialect. Flink itself can need the support of partition table. I'm sure it will further strengthen the work in 1.11. I remember that we only had disputes on creating partition tables, so I moved it to the chapter to be discussed, and limited it to hive dialect, only creating partition tables, right? > FLIP-89 proposed to make TableConfig configurable including the dialect configuration, i.g. `table.planner.sql-dialect`. So the only thing we need to do is introducing such a configuration. I'm not sure we should introduce FLIP-89, and dialect has no other function except for overwrite and partition inserting at present, I think they should be Flink syntax. And remove hive dialect limitation is a very small change/fix, Just remove a few lines of check. What do you think? (If you think so, after some discussion, we can start a vote later.) [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+Time+Attribute+in+SQL+DDL Best, Jingsong Lee On Thu, Dec 12, 2019 at 11:11 AM Rui Li <lirui.fu...@gmail.com> wrote: > +1 to fix it in 1.10. If this feature doesn't work via SQL CLI, I guess it > doesn't work for most Hive users. > +0 to remove the dialect check. I don't see much benefit this check can > bring to users, except that it prevents users from accidentally using some > Hive features, which doesn't seem to be very harmful anyway. But I don't > have a strong opinion for it and it probably needs more thorough > discussion. > > > On Thu, Dec 12, 2019 at 7:49 AM Jark Wu <imj...@gmail.com> wrote: > > > Thanks Jingsong, > > > > OVERWRITE and PARTITION are very fundamental features for Hive users. > > I'm sorry to hear that it doesn't work in SQL CLI. > > > > > Remove hive dialect limitation for these two grammars? > > The dialect restriction is introduced on purpose, because OVERWRITE and > > PARTITION syntax > > are not SQL standard. Even in the discussion of FLIP-63, the community > > have different opinion on > > whether the partition fields should be declared in schema part or not > [1]. > > The status of FLIP-63 is > > that limit the proposed syntax in Hive dialect, and we may propose new > > built-in syntax for creating > > partition tables in the future. So I'm -1 to removing dialect > limitation. > > > > > Should we fix this in 1.10? > > From my point of view, the problem is that users can't switch dialects in > > SQL CLI, because it is not > > exposed as a configuration. FLIP-89 [2] proposed to make TableConfig > > configurable including the > > dialect configuration, i.g. `table.planner.sql-dialect`. So the only > > thing we need to do is introducing > > such a configuration, but this is definitely is not a *bug*. However, > > considering it is a small change, > > and from user perspective, I'm +1 to introducing such a configuration. > > > > Best, > > Jark > > > > [1]: > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-63-Rework-table-partition-support-tp32770p33510.html > > [2]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-89%3A+Improve+usability+of+TableConfig > > > > On Thu, 12 Dec 2019 at 03:24, Bowen Li <bowenl...@gmail.com> wrote: > > > >> Hi Jingsong, > >> > >> Thanks a lot for reporting this issue. > >> > >> IIRC, we added [INSERT OVERWRITE] and [PARTITION] clauses to support > Hive > >> integration before FLIP-63 was proposed to introduce generic partition > >> support to Flink. Thus when we added these syntax, we were intentionally > >> conservative and limited their scope to Hive dialect. @Rui may help to > >> confirm that. I'm a bit surprised about it as well. As core APIs of > >> FLIP-63 > >> were done, I don't see why we would limit these syntax to Hive dialect > >> alone. It's just unfortunately that we may have forgot to revisit this > >> topic and we apparently missed some test cases on SQL CLI side. Sorry > for > >> that. > >> > >> From a product perspective, SQL CLI is super critical for Flink-Hive > >> integration and Flink SQL iteself. INSERT OVERWRITE and PARTITION are > two > >> of the most commonly used syntax in Hive, the product wouldn't be > useable > >> without such proper support. > >> > >> Thus, I think this is a *bug*, we can stop limiting them in Hive > dialect, > >> and should fix it in 1.10. We should also add more test coverage for SQL > >> CLI to avoid such surprise. > >> > >> Cheers, > >> Bowen > >> > >> On Wed, Dec 11, 2019 at 1:29 AM Jingsong Li <jingsongl...@gmail.com> > >> wrote: > >> > >> > Hi Dev, > >> > > >> > After cutting out the branch of 1.10, I tried the following functions > of > >> > SQL-CLI and found that it does not support: > >> > - insert overwrite > >> > - PARTITION (partcol1=val1, partcol2=val2 ...) > >> > The SQL pattern is: > >> > INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION > (partcol1=val1, > >> > partcol2=val2 ...) select_statement1 FROM from_statement; > >> > It is a surprise to me. > >> > The reason is that we only allow these two grammars in hive dialect. > And > >> > SQL-CLI does not have an interface to switch dialects. > >> > > >> > Because it directly hinders the SQL-CLI's insert syntax in hive > >> integration > >> > and seriously hinders the practicability of SQL-CLI. > >> > And we have introduced these two grammars in FLIP-63 [1] to Flink. > >> > Here are my question: > >> > 1.Should we remove hive dialect limitation for these two grammars? > >> > 2.Should we fix this in 1.10? > >> > > >> > [1] > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > >> > > >> > Best, > >> > Jingsong Lee > >> > > >> > > > > -- > Best regards! > Rui Li > -- Best, Jingsong Lee
