Currently I think we can load from the jar and check the services file to
get the connector type. but is it necessary we may continue to discuss.
Hi, Sergey, WDYT？
Another idea is FactoryUtil#discoverFactories and
check if it implements DynamicTableSourceFactory or DynamicTableSinkFactory
with versions it could be trickier...
Moreover it seems the version could be a part of the name sometimes[1].
I think name and type could be enough or please correct me if I'm wrong
or can we open a single ticket under this FLIP?
I have a relatively old jira issue[2] for showing connectors with a poc pr.
Could I propose to move this jira issue as a subtask under the FLIP one and
revive it?
[1]
https://github.com/apache/flink/blob/161014149e803bfd1d3653badb230b2ed36ce3cb/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/Factory.java#L65-L69
[2] https://issues.apache.org/jira/browse/FLINK-25788
On Tue, Feb 28, 2023 at 11:56 AM Ran Tao <chucheng...@gmail.com> wrote:
Hi, Jark. thanks.
About ILIKE
I have updated the FLIP for ILIKE support (Including existing showTables
&
showColumns how to change).
About show connectors @Sergey,
Currently I think we can load from the jar and check the services file to
get the connector type. but is it necessary we may continue to discuss.
Hi, Sergey, WDYT？or can we open a single ticket under this FLIP?
Best Regards,
Ran Tao
Jark Wu <imj...@gmail.com> 于2023年2月28日周二 17:45写道：
Besides, if we introduce the ILIKE, we should also add this feature for
the previous SHOW with LIKE statements. They should be included in this
FLIP.
Best,
Jark
2023年2月28日 17:40，Jark Wu <imj...@gmail.com> 写道：
Hi Ran,
Could you add descriptions about what’s the behavior and differences
between the LIKE and ILIKE?
Besides, I don’t see the SHOW CONNECTOR syntax and description and
how
it works in the FLIP. Is it intended to be included in this FLIP?
Best,
Jark
2023年2月28日 10:58，Ran Tao <chucheng...@gmail.com> 写道：
Hi, guys. thanks for advices.
allow me to make a small summary:
1.Support ILIKE
2.Using catalog api to support show operations
3.Need a dedicated FLIP try to support INFORMATION_SCHEMA
4.Support SHOW CONNECTORS
If there are no other questions, i will try to start a VOTE for this
FLIP.
WDYT?
Best Regards,
Ran Tao
Sergey Nuyanzin <snuyan...@gmail.com> 于2023年2月27日周一 21:12写道：
Hi Jark,
thanks for your comment.
Considering they
are orthogonal and information schema requires more complex design
and
discussion, it deserves a separate FLIP
I'm ok with a separate FLIP for INFORMATION_SCHEMA.
Sergey, are you willing to contribute this FLIP?
Seems I need to have more research done for that.
I would try to help/contribute here
On Mon, Feb 27, 2023 at 3:46 AM Ran Tao <chucheng...@gmail.com>
wrote:
HI, Jing. thanks.
@about ILIKE, from my collections of some popular engines founds
that
just
snowflake has this syntax in show with filtering.
do we need to support it? if yes, then current some existed show
operations
need to be addressed either.
@about ShowOperation with like. it's a good idea. yes, two
parameters
for
constructor can work. thanks for your advice.
Best Regards,
Ran Tao
Jing Ge <j...@ververica.com.invalid> 于2023年2月27日周一 06:29写道：
Hi,
@Aitozi
This is exactly why LoD has been introduced: to avoid exposing
internal
structure(2nd and lower level API).
@Jark
IMHO, there is no conflict between LoD and "High power-to-weight
ratio"
with the given example, List.subList() returns List interface
itself,
no
internal or further interface has been exposed. After offering
tEvn.getCatalog(), "all" methods in Catalog Interface have been
provided
by
TableEnvironment(via getCatalog()). From user's perspective and
maintenance
perspective there is no/less difference between providing them
directly
via
TableEnvironment or via getCatalog(). They are all exposed. Using
getCatalog() will reduce the number of boring wrapper methods,
but
on
the
other hand not every method in Catalog needs to be exposed, so
the
number
of wrapper methods would be limited/less, if we didn't expose
Catalog.
Nevertheless, since we already offered getCatalog(), it makes
sense
to
continue using it. The downside is the learning effort for users
-
they
have to know that listDatabases() is hidden in Catalog, go to
another
haystack and then find the needle in there.
+1 for Information schema with a different FLIP. From a design
perspective,
information schema should be the first choice for most cases and
easy
to
use. Catalog, on the other hand, will be more powerful and offer
more
advanced features.
Best regards,
Jing
On Sat, Feb 25, 2023 at 3:57 PM Jark Wu <imj...@gmail.com>
wrote:
Hi Sergey,
I think INFORMATION_SCHEMA is a very interesting idea, and I
hope
we
can
support it. However, it doesn't conflict with the idea of
auxiliary
statements. I can see different benefits of them. The
information
schema
provides powerful and flexible capabilities but needs to learn
the
complex
entity relationship[1]. The auxiliary SQL statements are super
handy
and
can resolve most problems, but they offer limited features.
I can see almost all the mature systems support both of them. I
think
it
also makes sense to support both of them in Flink. Considering
they
are orthogonal and information schema requires more complex
design
and
discussion, it deserves a separate FLIP. Sergey, are you willing
to
contribute this FLIP?
Best,
Jark
[1]:
https://docs.databricks.com/sql/language-manual/sql-ref-information-schema.html
On Fri, 24 Feb 2023 at 22:43, Ran Tao <chucheng...@gmail.com>
wrote:
Thanks John.
It seems that most people prefer the information_schema
implementation.
information_schema does have more benefits (however, the show
operation
is
also an option and supplement).
Otherwise, the sql syntax and keywords may be changed
frequently.
Of course, it will be more complicated than the extension of
the
show
operation.
It is necessary to design various tables in information_schema,
which
may
take a period of effort.
I will try to design the information_schema and integrate it
with
flink.
This may be a relatively big feature for me. I hope you guys
can
give
comments and opinions later.
Thank you all.
Best Regards,
Ran Tao
John Roesler <vvcep...@apache.org> 于2023年2月24日周五 21:53写道：
Hello Ran,
Thanks for the FLIP!
Do you mind if we revisit the topic of doing this by adding an
Information
schema? The SHOW approach requires modifying the
parser/language
for
every
gap we identify. On the flip side, an Information schema is
much
easier
to
discover and remember how to use, and the ability to run
queries
on
it
is
quite valuable for admins. It’s also better for Flink
maintainers,
because
the information tables’ schemas can be evolved over time just
like
regular
tables, whereas every change to a SHOW statement would be a
breaking
change.
I understand that it may seem like a big effort, but we’re
proposing
quite
a big extension to the space of SHOW statement, so it seems
appropriate
to
take the opportunity and migrate to a better framework rather
than
incrementally building on (and tying us even more firmly to)
the
existing
approach.
Thanks for your consideration,
John
On Fri, Feb 24, 2023, at 05:58, Sergey Nuyanzin wrote:
thanks for explanation
But it's not clear to me what exactly
you want to display? Is it the name of the plugin?
I was thinking about name, type (source/sink) and may be
version
(not
sure
if it's possible right now)
On Fri, Feb 24, 2023 at 12:46 PM Ran Tao <
chucheng...@gmail.com>
wrote:
Hi, Sergey. thanks. first step we can support filtering for
show
operations
in this FLIP try to align other engines.
If we want to support describe other objects, of course we
need
to
design
how to get these metadatas or informations and printAsStyle.
(So
it
maybe
need another FLIP for more details).
Does it make sense to add support for connectors e.g. show
{sink|source|all} connectors?
I think we can support it, currently flink do support some
operations
only
for flink itself such as showJobs. But it's not clear to me
what
exactly
you want to display? Is it the name of the plugin?
Just Like:
Kafka
Hudi
Files
Best Regards,
Ran Tao
Sergey Nuyanzin <snuyan...@gmail.com> 于2023年2月24日周五
19:11写道：
Thanks for driving the FLIP
I have a couple of questions
Am I right that INFORMATION_SCHEMA mentioned by Timo[1] is
out
of
scope
of
this FLIP?
I noticed there are some support of it in
Spark[2]/Hive[3]/Snowflake[4]
and
others
Does it make sense to add support for connectors e.g. show
{sink|source|all} connectors?
[1]
https://lists.apache.org/thread/2g108qlfwbhb56wnoc5qj0yq29dvq1vv
[2] https://issues.apache.org/jira/browse/SPARK-16452
[3] https://issues.apache.org/jira/browse/HIVE-1010
[4]
https://docs.snowflake.com/en/sql-reference/info-schema
On Fri, Feb 24, 2023 at 4:19 AM Jark Wu <imj...@gmail.com>
wrote:
Hi Jing,
we'd better reduce the dependency chain and follow the
Law
of
Demeter(LoD, clean code).
Adding a new method in TableEnvironment is therefore
better
than
calling
an API chain
I think I don't fully agree that LoD is a good practice.
Actually, I
would
prefer to keep the API clean and concise.
This is also the Java Collection Framework design
principle
[1]:
"High
power-to-weight ratio". Otherwise,
it will explode the API interfaces with different
combinations
of
methods.
Currently, TableEnvironment
already provides 60+ methods.
IMO, with the increasing methods of accessing and
manipulating
metadata,
they can be extracted to
a separate interface, where we can add richer methods.
This
work
can be
aligned with the
CatalogManager interface (FLIP-295) [2].
Best,
Jark
[1]:
https://stackoverflow.com/questions/7568819/why-no-tail-or-head-method-in-list-to-get-last-or-first-element
[2]:
https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
On Fri, 24 Feb 2023 at 10:38, Aitozi <
gjying1...@gmail.com>
wrote:
Hi,
Thanks for the nice proposal, Ran.
Regarding this api usage, I have some discussion
with
@twalthr
before
as here <
https://github.com/apache/flink/pull/15137#issuecomment-1356124138
Personally, I think leaking the Catalog to the user side
is
not
suitable,
since there are some read/write operations in the
Catalog,
the
TableEnvironment#getCatalog will expose all of them to
the
user
side.
So
I
learned to add a new api in TableEnvironment to reduce
reliance
on
the
current TableEnvironment#getCatalog.
Thanks,
Aitozi
Ran Tao <chucheng...@gmail.com> 于2023年2月23日周四 23:44写道：
Hi, JingSong, Jing.
thank for sharing your opinions.
What you say makes sense, both approaches have pros
and
cons.
If it is a modification of `TableEnvrionment`, such as
listDatabases(catalog). It is actually consistent with
the
other
overloaded
methods before,
and defining this method means that TableEnvrionment
does
provide
this
capability (rather than relying on the functionality
of
another
class).
The disadvantage is that api changes may be required,
and
may
continue
to
be modified in the future.
But from the TableEnvrionment itself, it really
doesn't
pay
attention
to
how the underlying layer is implemented.
(Although it is actually taken from the catalogManager
at
present,
this
is
another question)
Judging from the current dependencies,
flink-table-api-java
strongly
relies
on flink-table-common to use various common classes
and
interfaces,
especially the catalog interface.
So we can extract various metadata information in the
catalog
through
`tEnv.getCatalog`.
The advantage is that it will not cause api
modification,
but
this
method
of use breaks LoD.
In fact, the current flink-table-api-java design is
completely
bound
to
the
catalog interface.
If the mandatory modification of PublicApi is
constrained
(may
be
modified
again and later), I tend to use `tEnv.getCatalog`
directly,
otherwise
It would actually be more standard to add overloaded
methods
to
`TableEnvrionment`.
Another question, can the later capabilities of
TableEnvrionment be
implemented through SupportXXX?
In order to solve the problem that the method needs to
be
added
in
the
future. This kind of usage occurs frequently in flink.
Looking forward to your other considerations,
and also try to wait to see if there are other
relevant
API
designers
or
committers to provide comments.
Best Regards,
Ran Tao
Jing Ge <j...@ververica.com.invalid> 于2023年2月23日周四
18:58写道：
Hi Jingson,
Thanks for sharing your thoughts. Please see my
reply
below.
On Thu, Feb 23, 2023 at 10:16 AM Jingsong Li <
jingsongl...@gmail.com
wrote:
Hi Jing Ge,
First, flink-table-common contains all common
classes
of
Flink
Table,
I think it is hard to bypass its dependence.
If any time when we use flink-table-api-java, we
have
to
cross
through
flink-table-api-java and use flink-table-common, we
should
reconsider
the
design of these two modules and how
interfaces/classes
are
classified
into
those modules.
Secondly, almost all methods in Catalog looks
useful
to
me,
so
if
we
are following LoD, we should add all methods again
to
TableEnvironment. I think it is redundant.
That is the enlarged issue I mentioned previously. A
simple
solution
is
to
move Catalog to the top level API. The fact is that
a
catalog
package
already exists in flink-table-api-java but the
Catalog
interface
is
in
flink-table-common. I don't know the historical
context
of
this
design.
Maybe you could share some insight with us? Thanks
in
advance.
Beyond
that,
there should be other AOP options but need more time
to
figure it
out.
And, this API chain does not look deep.
-
"tEnv.getCatalog(tEnv.getCurrentCatalog()).get().listDatabases()"
looks a little complicated. The complex part is
ahead.
- If we have a method to get Catalog directly, can
be
simplify
to
"tEnv.catalog().listDatabase()", this is simple.
Commonly, it will need more effort to always follow
LoD,
but
for
the
top
level facade API like TableEnvironment, both the API
developer,
API
consumer and the project itself from a long-term
perspective
will
benefit
from sticking to LoD. Since we already have the
getCatalog(String
catalog)
method in TableEnvironment, it also makes sense to
follow
your
suggestion,
if we only want to limit/avoid public API changes.
But
please
be
aware
that
we will have all known long-term drawbacks because
of
LoD
violation, especially the cross modules violation. I
just
checked
all
usages of getCatalog(String catalog) in the master
branch.
Currently
there
are very limited calls. It is better to pay
attention
to
it
before
it
goes
worse. Just my 2 cents. :)
Best,
Jingsong
On Thu, Feb 23, 2023 at 4:47 PM Jing Ge
<j...@ververica.com.invalid
wrote:
Hi Jingson,
Thanks for the knowledge sharing. IMHO, it looks
more
like a
design
guideline question than just avoiding public API
change.
Please
correct
me
if I'm wrong.
Catalog is in flink-table-common module and
TableEnvironment
is
in
flink-table-api-java. Depending on how and where
those
features
proposed
in
this FLIP will be used, we'd better reduce the
dependency
chain
and
follow
the Law of Demeter(LoD, clean code) [1]. Adding
a
new
method
in
TableEnvironment is therefore better than
calling
an
API
chain.
It
is
also
more user friendly for the caller, because there
is
no
need
to
understand
the internal structure of the called API. The
downside
of
doing
this
is
that we might have another issue with the
current
TableEnvironment
design -
the TableEnvironment interface got enlarged with
more
wrapper
methods.
This
is a different issue that could be solved with
improved
abstraction
design
in the future. After considering pros and cons,
if
we
want to
add
those
features now, I would prefer following LoD than
API
chain
calls.
WDYT?
Best regards,
Jing
[1]
https://hackernoon.com/object-oriented-tricks-2-law-of-demeter-4ecc9becad85
On Thu, Feb 23, 2023 at 6:26 AM Ran Tao <
chucheng...@gmail.com
wrote:
Hi Jingsong. thanks. i got it.
In this way, there is no need to introduce new
API
changes.
Best Regards,
Ran Tao
Jingsong Li <jingsongl...@gmail.com>
于2023年2月23日周四
12:26写道：
Hi Ran,
I mean we can just use
TableEnvironment.getCatalog(getCurrentCatalog).get().listDatabases().
We don't need to provide new apis just for
utils.
Best,
Jingsong
On Thu, Feb 23, 2023 at 12:11 PM Ran Tao <
chucheng...@gmail.com>
wrote:
Hi Jingsong, thanks.
The implementation of these statements in
TableEnvironmentImpl
is
called
through the catalog api.
but it does support some new override
methods
on
the
catalog
api
side,
and
I will update it later. Thank you.
e.g.
TableEnvironmentImpl
@Override
public String[] listDatabases() {
return catalogManager
.getCatalog(catalogManager.getCurrentCatalog())
.get()
.listDatabases()
.toArray(new String[0]);
}
Best Regards,
Ran Tao
Jingsong Li <jingsongl...@gmail.com>
于2023年2月23日周四
11:47写道：
Thanks for the proposal.
+1 for the proposal.
I am confused about "Proposed
TableEnvironment
SQL
API
Changes",
can
we just use catalog api for this
requirement?
Best,
Jingsong
On Thu, Feb 23, 2023 at 10:48 AM Jacky
Lau <
liuyong...@gmail.com
wrote:
Hi Ran:
Thanks for driving the FLIP. the
google
doc
looks
really
good.
it
is
important to improve user interactive
experience.
+1
to
support
this
feature.
Jing Ge <j...@ververica.com.invalid>
于2023年2月23日周四
00:51写道：
Hi Ran,
Thanks for driving the FLIP. It
looks
overall
good.
Would
you
like to
add
a description of useLike and
notLike?
I
guess
useLike
true
is for
"LIKE"
and notLike true is for "NOT LIKE"
but I
am
not
sure
if I
understood it
correctly. Furthermore, does it make
sense
to
support
"ILIKE"
too?
Best regards,
Jing
On Wed, Feb 22, 2023 at 1:17 PM Ran
Tao
<
chucheng...@gmail.com>
wrote:
Currently flink sql auxiliary
statements
has
supported
some
good
features
such as catalog/databases/table
support.
But these features are not very
complete
compared
with
other
popular
engines such as spark, presto,
hive
and
commercial
engines
such
as
snowflake.
For example, many engines support
show
operation
with
filtering
except
flink, and support describe other
object(flink
only
support
describe
table).
I wonder can we add these useful
features
for
flink?
You can find details in this
doc.[1]
or
FLIP.[2]
Also, please let me know if there
is a
mistake.
Looking
forward
to
your
reply.
[1]
https://docs.google.com/document/d/1hAiOfPx14VTBTOlpyxG7FA2mB1k5M31VnKYad2XpJ1I/
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-297%3A+Improve+Auxiliary+Sql+Statements
Best Regards,
Ran Tao
--
Best regards,
Sergey
--
Best regards,
Sergey
--
Best regards,
Sergey
--
Best regards,
Sergey
