[
https://issues.apache.org/jira/browse/FLINK-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670794#comment-16670794
]
Xuefu Zhang commented on FLINK-10618:
-------------------------------------
Hi [~dangdangdang] and [~twalthr], Thanks for the interest and input. This JIRA
might be just a subtask of FLINK-10556, which is about Hive integration, but it
might be big enough to stand on its own. The design doc in FLINK-10556 covers
the issue here as well. Please refer to the design doc and let me know your
feedback for this JIRA (and other issues).
The current design is to let TableEnvironment contain a catalog manager, which
further contains a list of external catalogs, one of which will be used to
store flink meta objects, such as tables, temp tables, functions, temp
functions, and views. We plan to provide two implementation of this catalog: an
in-memory one and one using Hive. Catalogs are referenced by their names,
though one of them may be designated as default for which catalog name can be
skipped.
I'm not sure if the TableEnvironment needs to implement a catalog interface,
mostly because it's rather an aggregation of many catalogs. I'm open to this
idea though.
The current design didn't take in consideration the connectors (or other
objects that need persistency). I'd like to understand this a bit further in
order to give my opinion.
I think it's a good idea to convert or consolidate table APIs to java. We can
include it as a dependent task for the work here.
Sorry that the JIRAs are not align well with the relevant document. I need to
do some cleanup to make them clearer.
Please let me know your further thoughts.
> Introduce catalog for Flink tables
> ----------------------------------
>
> Key: FLINK-10618
> URL: https://issues.apache.org/jira/browse/FLINK-10618
> Project: Flink
> Issue Type: New Feature
> Components: SQL Client
> Affects Versions: 1.6.1
> Reporter: Xuefu Zhang
> Assignee: Xuefu Zhang
> Priority: Major
>
> Besides meta objects such as tables that may come from an
> {{ExternalCatalog}}, Flink also deals with tables/views/functions that are
> created on the fly (in memory), or specified in a configuration file. Those
> objects don't belong to any {{ExternalCatalog}}, yet Flink either stores them
> in memory, which are non-persistent, or recreates them from a file, which is
> a big pain for the user. Those objects are only known to Flink but Flink has
> a poor management for them.
> Since they are typical objects in a database catalog, it's natural to have a
> catalog that manages those objects. The interface will be similar to
> {{ExternalCatalog}}, which contains meta objects that are not managed by
> Flink. There are several possible implementations of the Flink internal
> catalog interface: memory, file, external registry (such as confluent schema
> registry or Hive metastore), and relational database, etc.
> The initial functionality as well as the catalog hierarchy could be very
> simple. The basic functionality of the catalog will be mostly create, alter,
> and drop tables, views, functions, etc. Obviously, this can evolve over the
> time.
> We plan to provide implementations with memory, file, and Hive metastore, and
> will be plugged in at SQL-Client layer.
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
