Hi, Sergey. thanks. first step we can support filtering for show operations
in this FLIP try to align other engines.
If we want to support describe other objects, of course we need to design
how to get these metadatas or informations and printAsStyle. (So it maybe
need another FLIP for more details).
> Does it make sense to add support for connectors e.g. show
> {sink|source|all} connectors?
I think we can support it, currently flink do support some operations only
for flink itself such as showJobs. But it's not clear to me what exactly
you want to display? Is it the name of the plugin?
Just Like:
Kafka
Hudi
Files
Best Regards,
Ran Tao
Sergey Nuyanzin <snuyan...@gmail.com> 于2023年2月24日周五 19:11写道：
> Thanks for driving the FLIP
>
> I have a couple of questions
> Am I right that INFORMATION_SCHEMA mentioned by Timo[1] is out of scope of
> this FLIP?
> I noticed there are some support of it in Spark[2]/Hive[3]/Snowflake[4] and
> others
>
> Does it make sense to add support for connectors e.g. show
> {sink|source|all} connectors?
>
> [1] https://lists.apache.org/thread/2g108qlfwbhb56wnoc5qj0yq29dvq1vv
> [2] https://issues.apache.org/jira/browse/SPARK-16452
> [3] https://issues.apache.org/jira/browse/HIVE-1010
> [4] https://docs.snowflake.com/en/sql-reference/info-schema
>
>
> On Fri, Feb 24, 2023 at 4:19 AM Jark Wu <imj...@gmail.com> wrote:
>
> > Hi Jing,
> >
> > > we'd better reduce the dependency chain and follow the Law of
> > Demeter(LoD, clean code).
> > > Adding a new method in TableEnvironment is therefore better than
> calling
> > an API chain
> >
> > I think I don't fully agree that LoD is a good practice. Actually, I
> would
> > prefer to keep the API clean and concise.
> > This is also the Java Collection Framework design principle [1]: "High
> > power-to-weight ratio". Otherwise,
> > it will explode the API interfaces with different combinations of
> methods.
> > Currently, TableEnvironment
> > already provides 60+ methods.
> >
> > IMO, with the increasing methods of accessing and manipulating metadata,
> > they can be extracted to
> > a separate interface, where we can add richer methods. This work can be
> > aligned with the
> > CatalogManager interface (FLIP-295) [2].
> >
> > Best,
> > Jark
> >
> > [1]:
> >
> >
> https://stackoverflow.com/questions/7568819/why-no-tail-or-head-method-in-list-to-get-last-or-first-element
> > [2]: https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> >
> >
> > On Fri, 24 Feb 2023 at 10:38, Aitozi <gjying1...@gmail.com> wrote:
> >
> > > Hi,
> > > Thanks for the nice proposal, Ran.
> > > Regarding this api usage, I have some discussion with @twalthr
> before
> > > as here <
> > > https://github.com/apache/flink/pull/15137#issuecomment-1356124138>
> > > Personally, I think leaking the Catalog to the user side is not
> suitable,
> > > since there are some read/write operations in the Catalog, the
> > > TableEnvironment#getCatalog will expose all of them to the user side.
> So
> > I
> > > learned to add a new api in TableEnvironment to reduce reliance on the
> > > current TableEnvironment#getCatalog.
> > >
> > > Thanks,
> > > Aitozi
> > >
> > >
> > > Ran Tao <chucheng...@gmail.com> 于2023年2月23日周四 23:44写道：
> > >
> > > > Hi, JingSong, Jing.
> > > >
> > > > thank for sharing your opinions.
> > > >
> > > > What you say makes sense, both approaches have pros and cons.
> > > >
> > > > If it is a modification of `TableEnvrionment`, such as
> > > > listDatabases(catalog). It is actually consistent with the other
> > > overloaded
> > > > methods before,
> > > > and defining this method means that TableEnvrionment does provide
> this
> > > > capability (rather than relying on the functionality of another
> class).
> > > > The disadvantage is that api changes may be required, and may
> continue
> > to
> > > > be modified in the future.
> > > > But from the TableEnvrionment itself, it really doesn't pay attention
> > to
> > > > how the underlying layer is implemented.
> > > > (Although it is actually taken from the catalogManager at present,
> this
> > > is
> > > > another question)
> > > >
> > > > Judging from the current dependencies, flink-table-api-java strongly
> > > relies
> > > > on flink-table-common to use various common classes and interfaces,
> > > > especially the catalog interface.
> > > > So we can extract various metadata information in the catalog through
> > > > `tEnv.getCatalog`.
> > > > The advantage is that it will not cause api modification, but this
> > method
> > > > of use breaks LoD.
> > > > In fact, the current flink-table-api-java design is completely bound
> to
> > > the
> > > > catalog interface.
> > > >
> > > > If the mandatory modification of PublicApi is constrained (may be
> > > modified
> > > > again and later), I tend to use `tEnv.getCatalog` directly, otherwise
> > > > It would actually be more standard to add overloaded methods to
> > > > `TableEnvrionment`.
> > > >
> > > > Another question, can the later capabilities of TableEnvrionment be
> > > > implemented through SupportXXX?
> > > > In order to solve the problem that the method needs to be added in
> the
> > > > future. This kind of usage occurs frequently in flink.
> > > >
> > > > Looking forward to your other considerations,
> > > > and also try to wait to see if there are other relevant API designers
> > or
> > > > committers to provide comments.
> > > >
> > > >
> > > > Best Regards,
> > > > Ran Tao
> > > >
> > > > Jing Ge <j...@ververica.com.invalid> 于2023年2月23日周四 18:58写道：
> > > >
> > > > > Hi Jingson,
> > > > >
> > > > > Thanks for sharing your thoughts. Please see my reply below.
> > > > >
> > > > > On Thu, Feb 23, 2023 at 10:16 AM Jingsong Li <
> jingsongl...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jing Ge,
> > > > > >
> > > > > > First, flink-table-common contains all common classes of Flink
> > Table,
> > > > > > I think it is hard to bypass its dependence.
> > > > > >
> > > > >
> > > > > If any time when we use flink-table-api-java, we have to cross
> > through
> > > > > flink-table-api-java and use flink-table-common, we should
> reconsider
> > > the
> > > > > design of these two modules and how interfaces/classes are
> classified
> > > > into
> > > > > those modules.
> > > > >
> > > > >
> > > > > >
> > > > > > Secondly, almost all methods in Catalog looks useful to me, so if
> > we
> > > > > > are following LoD, we should add all methods again to
> > > > > > TableEnvironment. I think it is redundant.
> > > > > >
> > > > >
> > > > > That is the enlarged issue I mentioned previously. A simple
> solution
> > is
> > > > to
> > > > > move Catalog to the top level API. The fact is that a catalog
> package
> > > > > already exists in flink-table-api-java but the Catalog interface is
> > in
> > > > > flink-table-common. I don't know the historical context of this
> > design.
> > > > > Maybe you could share some insight with us? Thanks in advance.
> Beyond
> > > > that,
> > > > > there should be other AOP options but need more time to figure it
> > out.
> > > > >
> > > > >
> > > > > >
> > > > > > And, this API chain does not look deep.
> > > > > > -
> "tEnv.getCatalog(tEnv.getCurrentCatalog()).get().listDatabases()"
> > > > > > looks a little complicated. The complex part is ahead.
> > > > > > - If we have a method to get Catalog directly, can be simplify to
> > > > > > "tEnv.catalog().listDatabase()", this is simple.
> > > > > >
> > > > >
> > > > > Commonly, it will need more effort to always follow LoD, but for
> the
> > > top
> > > > > level facade API like TableEnvironment, both the API developer, API
> > > > > consumer and the project itself from a long-term perspective will
> > > benefit
> > > > > from sticking to LoD. Since we already have the getCatalog(String
> > > > catalog)
> > > > > method in TableEnvironment, it also makes sense to follow your
> > > > suggestion,
> > > > > if we only want to limit/avoid public API changes. But please be
> > aware
> > > > that
> > > > > we will have all known long-term drawbacks because of LoD
> > > > > violation, especially the cross modules violation. I just checked
> all
> > > > > usages of getCatalog(String catalog) in the master branch.
> Currently
> > > > there
> > > > > are very limited calls. It is better to pay attention to it before
> it
> > > > goes
> > > > > worse. Just my 2 cents. :)
> > > > >
> > > > >
> > > > > >
> > > > > > Best,
> > > > > > Jingsong
> > > > > >
> > > > > > On Thu, Feb 23, 2023 at 4:47 PM Jing Ge
> <j...@ververica.com.invalid
> > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Jingson,
> > > > > > >
> > > > > > > Thanks for the knowledge sharing. IMHO, it looks more like a
> > design
> > > > > > > guideline question than just avoiding public API change. Please
> > > > correct
> > > > > > me
> > > > > > > if I'm wrong.
> > > > > > >
> > > > > > > Catalog is in flink-table-common module and TableEnvironment is
> > in
> > > > > > > flink-table-api-java. Depending on how and where those features
> > > > > proposed
> > > > > > in
> > > > > > > this FLIP will be used, we'd better reduce the dependency chain
> > and
> > > > > > follow
> > > > > > > the Law of Demeter(LoD, clean code) [1]. Adding a new method in
> > > > > > > TableEnvironment is therefore better than calling an API chain.
> > It
> > > is
> > > > > > also
> > > > > > > more user friendly for the caller, because there is no need to
> > > > > understand
> > > > > > > the internal structure of the called API. The downside of doing
> > > this
> > > > is
> > > > > > > that we might have another issue with the current
> > TableEnvironment
> > > > > > design -
> > > > > > > the TableEnvironment interface got enlarged with more wrapper
> > > > methods.
> > > > > > This
> > > > > > > is a different issue that could be solved with improved
> > abstraction
> > > > > > design
> > > > > > > in the future. After considering pros and cons, if we want to
> add
> > > > those
> > > > > > > features now, I would prefer following LoD than API chain
> calls.
> > > > WDYT?
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://hackernoon.com/object-oriented-tricks-2-law-of-demeter-4ecc9becad85
> > > > > > >
> > > > > > > On Thu, Feb 23, 2023 at 6:26 AM Ran Tao <chucheng...@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Jingsong. thanks. i got it.
> > > > > > > > In this way, there is no need to introduce new API changes.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Ran Tao
> > > > > > > >
> > > > > > > >
> > > > > > > > Jingsong Li <jingsongl...@gmail.com> 于2023年2月23日周四 12:26写道：
> > > > > > > >
> > > > > > > > > Hi Ran,
> > > > > > > > >
> > > > > > > > > I mean we can just use
> > > > > > > > >
> > > > >
> TableEnvironment.getCatalog(getCurrentCatalog).get().listDatabases().
> > > > > > > > >
> > > > > > > > > We don't need to provide new apis just for utils.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Jingsong
> > > > > > > > >
> > > > > > > > > On Thu, Feb 23, 2023 at 12:11 PM Ran Tao <
> > > chucheng...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Jingsong, thanks.
> > > > > > > > > >
> > > > > > > > > > The implementation of these statements in
> > > TableEnvironmentImpl
> > > > is
> > > > > > > > called
> > > > > > > > > > through the catalog api.
> > > > > > > > > >
> > > > > > > > > > but it does support some new override methods on the
> > catalog
> > > > api
> > > > > > side,
> > > > > > > > > and
> > > > > > > > > > I will update it later. Thank you.
> > > > > > > > > >
> > > > > > > > > > e.g.
> > > > > > > > > > TableEnvironmentImpl
> > > > > > > > > > @Override
> > > > > > > > > > public String[] listDatabases() {
> > > > > > > > > > return catalogManager
> > > > > > > > > >
> > > .getCatalog(catalogManager.getCurrentCatalog())
> > > > > > > > > > .get()
> > > > > > > > > > .listDatabases()
> > > > > > > > > > .toArray(new String[0]);
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > Best Regards,
> > > > > > > > > > Ran Tao
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jingsong Li <jingsongl...@gmail.com> 于2023年2月23日周四
> > 11:47写道：
> > > > > > > > > >
> > > > > > > > > > > Thanks for the proposal.
> > > > > > > > > > >
> > > > > > > > > > > +1 for the proposal.
> > > > > > > > > > >
> > > > > > > > > > > I am confused about "Proposed TableEnvironment SQL API
> > > > > Changes",
> > > > > > can
> > > > > > > > > > > we just use catalog api for this requirement?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Jingsong
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Feb 23, 2023 at 10:48 AM Jacky Lau <
> > > > > liuyong...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Ran:
> > > > > > > > > > > > Thanks for driving the FLIP. the google doc looks
> > really
> > > > > good.
> > > > > > it
> > > > > > > > is
> > > > > > > > > > > > important to improve user interactive experience. +1
> to
> > > > > support
> > > > > > > > this
> > > > > > > > > > > > feature.
> > > > > > > > > > > >
> > > > > > > > > > > > Jing Ge <j...@ververica.com.invalid> 于2023年2月23日周四
> > > > 00:51写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Ran,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for driving the FLIP. It looks overall
> good.
> > > > Would
> > > > > > you
> > > > > > > > > like to
> > > > > > > > > > > add
> > > > > > > > > > > > > a description of useLike and notLike? I guess
> useLike
> > > > true
> > > > > > is for
> > > > > > > > > > > "LIKE"
> > > > > > > > > > > > > and notLike true is for "NOT LIKE" but I am not
> sure
> > > if I
> > > > > > > > > understood it
> > > > > > > > > > > > > correctly. Furthermore, does it make sense to
> support
> > > > > "ILIKE"
> > > > > > > > too?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Feb 22, 2023 at 1:17 PM Ran Tao <
> > > > > > chucheng...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Currently flink sql auxiliary statements has
> > > supported
> > > > > some
> > > > > > > > good
> > > > > > > > > > > features
> > > > > > > > > > > > > > such as catalog/databases/table support.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > But these features are not very complete compared
> > > with
> > > > > > other
> > > > > > > > > popular
> > > > > > > > > > > > > > engines such as spark, presto, hive and
> commercial
> > > > > engines
> > > > > > such
> > > > > > > > > as
> > > > > > > > > > > > > > snowflake.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For example, many engines support show operation
> > with
> > > > > > filtering
> > > > > > > > > > > except
> > > > > > > > > > > > > > flink, and support describe other object(flink
> only
> > > > > support
> > > > > > > > > describe
> > > > > > > > > > > > > > table).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I wonder can we add these useful features for
> > flink?
> > > > > > > > > > > > > > You can find details in this doc.[1] or FLIP.[2]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Also, please let me know if there is a mistake.
> > > Looking
> > > > > > forward
> > > > > > > > > to
> > > > > > > > > > > your
> > > > > > > > > > > > > > reply.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/
> > > > > > > > > > > > > > [2]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-297%3A+Improve+Auxiliary+Sql+Statements
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > > > Ran Tao
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>
