Hi Jark, Thanks for the inputs.
>Do we need to provide DESC EXTENDED <table_name> statement like Spark[1] to >show statistic for table/partition/columns? We do have supported `DESC EXTENDED` syntax, but currently only table schema will be display, I think we just need a JIRA to support it. > is it possible to ignore execution mode and force using batch mode for the > statement? As I replied above, The semantics of `ANALYZE TABLE` does not distinguish batch and streaming, It works for both batch and streaming, but the result of unbounded sources is meaningless. Currently, I throw exception for streaming mode, and we can support streaming mode with bounded source in the future. Best, Godfrey Jark Wu <imj...@gmail.com> 于2022年6月14日周二 17:56写道： > > Hi Godfrey, thanks for starting this discussion, this is a great feature > for batch users. > > The FLIP looks good to me in general. > > I only have 2 comments: > > 1) How do users know whether the given table or partition contains required > statistics? > Do we need to provide DESC EXTENDED <table_name> statement like Spark[1] to > show statistic for table/partition/columns? > > 2) If ANALYZE TABLE can only run in batch mode, is it possible to ignore > execution mode > and force using batch mode for the statement? From my perspective, ANALYZE > TABLE > is an auxiliary statement similar to SHOW TABLES but heavier, which doesn't > care about > environment execution mode. > > Best, > Jark > > [1]: > https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-aux-analyze-table.html > > On Tue, 14 Jun 2022 at 13:52, Jing Ge <j...@ververica.com> wrote: > > > Hi 华宗 > > > > 退订请发送任意消息至dev-unsubscr...@flink.apache.org > > In order to unsubscribe, please send an email to > > dev-unsubscr...@flink.apache.org > > > > Thanks > > > > Best regards, > > Jing > > > > > > On Tue, Jun 14, 2022 at 2:05 AM 华宗 <zhanghuaz...@126.com> wrote: > > > > > 退订 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2022-06-13 22:44:24, "cao zou" <zoucao...@gmail.com> wrote: > > > >Hi godfrey, thanks for your detail explanation. > > > >After explaining and glancing over the FLIP-231, I think it is > > > >really need, +1 for this and looking forward to it. > > > > > > > >best > > > >zoucao > > > > > > > >godfrey he <godfre...@gmail.com> 于2022年6月13日周一 14:43写道： > > > > > > > >> Hi Ingo, > > > >> > > > >> The semantics does not distinguish batch and streaming, > > > >> It works for both batch and streaming, but the result of > > > >> unbounded sources is meaningless. > > > >> Currently, I throw exception for streaming mode, > > > >> and we can support streaming mode with bounded source > > > >> in the future. > > > >> > > > >> Best, > > > >> Godfrey > > > >> > > > >> Ingo Bürk <airbla...@apache.org> 于2022年6月13日周一 14:17写道： > > > >> > > > > >> > Hi Godfrey, > > > >> > > > > >> > thank you for the explanation. A SELECT is definitely more generic > > and > > > >> > will work for all connectors automatically. As such I think it's a > > > good > > > >> > baseline solution regardless. > > > >> > > > > >> > We can also think about allowing connector-specific optimizations in > > > the > > > >> > future, but I do like your idea of letting the optimizer rules > > > perform a > > > >> > lot of the work here already by leveraging existing optimizations. > > > >> > Similarly things like non-null counts of non-nullable columns would > > > (or > > > >> > at least could) be handled by the optimizer rules already. > > > >> > > > > >> > So as far as that point goes, +1 to the generic approach. > > > >> > > > > >> > One more point, though: In general we should avoid supporting > > features > > > >> > only in specific modes as it breaks the unification promise. Given > > > that > > > >> > ANALYZE is a manual and completely optional operation I'm OK with > > > doing > > > >> > that here in principle. However, I wonder what will happen in the > > > >> > streaming / unbounded case. Do you plan to throw an error? Or do we > > > >> > complete the command as successful but without doing anything? > > > >> > > > > >> > > > > >> > Best > > > >> > Ingo > > > >> > > > > >> > On 13.06.22 05:50, godfrey he wrote: > > > >> > > Hi Ingo, > > > >> > > > > > >> > > Thanks for the inputs. > > > >> > > > > > >> > > I think converting `ANALYZE TABLE` to `SELECT` statement is > > > >> > > more generic approach. Because query plan optimization is more > > > generic, > > > >> > > we can provide more optimization rules to optimize not only > > > `SELECT` > > > >> statement > > > >> > > converted from `ANALYZE TABLE` but also the `SELECT` statement > > > written > > > >> by users. > > > >> > > > > > >> > >> JDBC connector can get a row count estimate without performing a > > > >> > >> SELECT COUNT(1) > > > >> > > To optimize such cases, we can implement a rule to push aggregate > > > into > > > >> > > table source. > > > >> > > Currently, there is a similar rule: SupportsAggregatePushDown, > > which > > > >> > > supports only pushing > > > >> > > local aggregate into source now. > > > >> > > > > > >> > > > > > >> > > Best, > > > >> > > Godfrey > > > >> > > > > > >> > > Ingo Bürk <airbla...@apache.org> 于2022年6月10日周五 17:15写道： > > > >> > >> > > > >> > >> Hi Godfrey, > > > >> > >> > > > >> > >> compared to the solution proposed in the FLIP (using a SELECT > > > >> > >> statement), I wonder if you have considered adding APIs to > > > catalogs / > > > >> > >> connectors to perform this task as an alternative? > > > >> > >> I could imagine that for many connectors, statistics could be > > > >> > >> implemented in a less expensive way by leveraging the underlying > > > >> system > > > >> > >> (e.g. a JDBC connector can get a row count estimate without > > > >> performing a > > > >> > >> SELECT COUNT(1)). > > > >> > >> > > > >> > >> > > > >> > >> Best > > > >> > >> Ingo > > > >> > >> > > > >> > >> > > > >> > >> On 10.06.22 09:53, godfrey he wrote: > > > >> > >>> Hi all, > > > >> > >>> > > > >> > >>> I would like to open a discussion on FLIP-240: Introduce > > "ANALYZE > > > >> > >>> TABLE" Syntax. > > > >> > >>> > > > >> > >>> As FLIP-231 mentioned, statistics are one of the most important > > > >> inputs > > > >> > >>> to the optimizer. Accurate and complete statistics allows the > > > >> > >>> optimizer to be more powerful. "ANALYZE TABLE" syntax is a very > > > >> common > > > >> > >>> but effective approach to gather statistics, which is already > > > >> > >>> introduced by many compute engines and databases. > > > >> > >>> > > > >> > >>> The main purpose of discussion is to introduce "ANALYZE TABLE" > > > >> syntax > > > >> > >>> for Flink sql. > > > >> > >>> > > > >> > >>> You can find more details in FLIP-240 document[1]. Looking > > > forward to > > > >> > >>> your feedback. > > > >> > >>> > > > >> > >>> [1] > > > >> > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217386481 > > > >> > >>> [2] POC: https://github.com/godfreyhe/flink/tree/FLIP-240 > > > >> > >>> > > > >> > >>> > > > >> > >>> Best, > > > >> > >>> Godfrey > > > >> > > > > >
