[
https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601690#comment-16601690
]
Rong Rong edited comment on FLINK-9172 at 9/2/18 10:41 PM:
-----------------------------------------------------------
Hi folks. Sorry for not taking this task for so long. I think [~twalthr] had
put int a very nice unification on external catalog by merging the change in
[https://github.com/apache/flink/pull/6343] for FLINK-9852.
As of now I think there are several things missing:
1. how the external catalogs can be configured in SQL-client. I am thinking
adding the following in sql-client yaml config:
{code}
#==============================================================================
# Data Catalogs
#==============================================================================
catalogs: [] # empty list
# A typical data catalog definition looks like:
# - name: ...
# catalog-type: [? extends ExternalCatalog.class] # such as HCatalog,
AvroCatalog
# datastore-type: [batch, streaming]
# connection: [? extends ConnectorDescriptor]
{code}
since ExternalCatalogTable only requires (1) a concrete implementation class
and (2) ConnectorDescriptor for instantiate the connection
2. how to use the ExternalCatalog
We definitely by default register all {{ExternalCatalog}} implementations into
{{TableEnvironment}}. I tried with a POC and it can be easily put into use by
simply invoking the external catalog tables with
{code}
SELECT cols FROM catalog_name.subschema_name.table_name
{code}
I haven't put too much thought in to the {{TableSink}} part, maybe this is
already supported by
{code}
INSERT INTO catalog_name.table_sink_name SELECT * FROM
catalog_name.table_source_name
{code}
[~eronwright] what do you think about the config YAML change. anything else we
can add? I was thinking about credential / security but they can probably be
put in as part of the ConnectorDescriptor
was (Author: walterddr):
Hi folks. Sorry for not taking this task for so long. I think [~twalthr] had
put int a very nice unification on external catalog by merging the change in
[https://github.com/apache/flink/pull/6343] for FLLINK-9852.
As of now I think there are several things missing:
1. how the external catalogs can be configured in SQL-client. I am thinking
adding the following in sql-client yaml config:
{code}
#==============================================================================
# Data Catalogs
#==============================================================================
catalogs: [] # empty list
# A typical data catalog definition looks like:
# - name: ...
# catalog-type: [? extends ExternalCatalog.class] # such as HCatalog,
AvroCatalog
# datastore-type: [batch, streaming]
# connection: [? extends ConnectorDescriptor]
{code}
since ExternalCatalogTable only requires (1) a concrete implementation class
and (2) ConnectorDescriptor for instantiate the connection
2. how to use the ExternalCatalog
We definitely by default register all {{ExternalCatalog}} implementations into
{{TableEnvironment}}. I tried with a POC and it can be easily put into use by
simply invoking the external catalog tables with
{code}
SELECT cols FROM catalog_name.subschema_name.table_name
{code}
I haven't put too much thought in to the {{TableSink}} part, maybe this is
already supported by
{code}
INSERT INTO catalog_name.table_sink_name SELECT * FROM
catalog_name.table_source_name
{code}
[~eronwright] what do you think about the config YAML change. anything else we
can add? I was thinking about credential / security but they can probably be
put in as part of the ConnectorDescriptor
> Support external catalog factory that comes default with SQL-Client
> -------------------------------------------------------------------
>
> Key: FLINK-9172
> URL: https://issues.apache.org/jira/browse/FLINK-9172
> Project: Flink
> Issue Type: Sub-task
> Reporter: Rong Rong
> Assignee: Rong Rong
> Priority: Major
>
> It will be great to have SQL-Client to support some external catalogs
> out-of-the-box for SQL users to configure and utilize easily. I am currently
> think of having an external catalog factory that spins up both streaming and
> batch external catalog table sources and sinks. This could greatly unify and
> provide easy access for SQL users.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
