[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867
]
Jane Chan edited comment on FLINK-21045 at 2/3/21, 10:41 AM:
-------------------------------------------------------------
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is also backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
----
h2. API Change
h3. ModuleManager
We introduce two methods to support USE MODULES and SHOW FULL MODULES. Beyond
that, the behavior of methods listModules, listFunctions, and
getFunctionDefinition will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List<Pair<String, Boolean>> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(List<String> names) {
// list all loaded modules with status
}
}
{code}
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate ModuleFactory to extends
org.apache.flink.table.factories.Factory. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
|LOAD MODULE module_name WITH ('prop' = 'myProp', ...)|
Explanation
* WITH is necessary
* Module name x is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
-------------------
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
-------------------
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
|UNLOAD MODULE module_name|
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
| ml | false |
-------------------
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
-------------------
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
|USE MODULES x [,y, z, ...]|
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name
[unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
{code}
h3. SHOW [FULL] MODULES
Display the loaded module names in declared order.
|SHOW [FULL] MODULES|
Explanation
* If `FULL` is specified, all loaded modules and their status will be listed.
* If `FULL` is omitted, only used modules will be listed.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> SHOW MODULES;
-------------------
| modules | used |
-------------------
| core | true |
-------------------
{code}
h2. YAML Change
“type” is deprecated and removed from the configuration.
{code:java}
modules:
- name: core
- name: hive
hive-version: 2.2.1
{code}
Best,
Jane
was (Author: qingyue):
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
----
h2. API Change
h3. ModuleManager
We introduce two methods to support USE MODULES and SHOW FULL MODULES. Beyond
that, the behavior of methods listModules, listFunctions, and
getFunctionDefinition will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List<Pair<String, Boolean>> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(List<String> names) {
// list all loaded modules with status
}
}
{code}
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate ModuleFactory to extends
org.apache.flink.table.factories.Factory. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
|LOAD MODULE module_name WITH ('prop' = 'myProp', ...)|
Explanation
* WITH is necessary
* Module name x is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
-------------------
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
-------------------
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
|UNLOAD MODULE module_name|
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | true |
-------------------
Flink SQL> UNLOAD MODULE ml;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
-------------------
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
|USE MODULES x [,y, z, ...]|
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name
[unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
{code}
h3. SHOW [FULL] MODULES
Display the loaded module names in declared order.
|SHOW [FULL] MODULES|
Explanation
* If `FULL` is specified, all loaded modules and their status will be listed.
* If `FULL` is omitted, only used modules will be listed.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> SHOW MODULES;
-------------------
| modules | used |
-------------------
| core | true |
-------------------
{code}
h2. YAML Change
“type” is deprecated and removed from the configuration.
{code:java}
modules:
- name: core
- name: hive
hive-version: 2.2.1
{code}
Best,
Jane
> Support 'load module' and 'unload module' SQL syntax
> ----------------------------------------------------
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
> Affects Versions: 1.13.0
> Reporter: Nicholas Jiang
> Assignee: Jane Chan
> Priority: Major
> Fix For: 1.13.0
>
>
> At present, Flink SQL doesn't support the 'load module' and 'unload module'
> SQL syntax. It's necessary for uses in the situation that users load and
> unload user-defined module through table api or sql client.
> SQL syntax has been proposed in FLIP-68:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
