[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867
]
Jane Chan edited comment on FLINK-21045 at 2/3/21, 12:10 PM:
-------------------------------------------------------------
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is also backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
----
h2. API Change
h3. ModuleManager
We introduce two methods to support {{USE MODULES}} and {{SHOW FULL MODULES}}.
Beyond that, the behavior of methods {{listModules}}, {{listFunctions}}, and
{{getFunctionDefinition}} will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List<ModuleEntry> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(String... names) {
// list all loaded modules with status
}
/** A POJO to represent a module's name and status. */
static class ModuleEntry {
private final String name;
private final boolean used;
ModuleEntry(String name, boolean used) {
this.name = name;
this.used = used;
// getters
}
}
{code}
h3. TableEnvironment
We introduce two methods {{listFullModules()}} and {{useModules(String…
names)}} to {{TableEnvironment}} interface. The impl will simply call
{{moduleManager#listFullModules()}} and {{moduleManager#useModules(String…
names)}}.
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate {{ModuleFactory}} to extends
{{org.apache.flink.table.factories.Factory}}. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
{code:java}
LOAD MODULE module_name [WITH ('prop' = 'myProp', ...)]{code}
Explanation
* Module name {{module_name}} is a simple identifier instead of a string
literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
-------------------
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
-------------------
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
{code:java}
UNLOAD MODULE module_name{code}
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
| ml | false |
-------------------
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
-------------------
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
{code:java}
USE MODULES x [,y, z, ...]{code}
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name
[unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
{code}
h3. SHOW [FULL] MODULES
Display the loaded module names in declared order.
{code:java}
SHOW [FULL] MODULES{code}
Explanation
* If `FULL` is specified, all loaded modules and their status will be listed.
* If `FULL` is omitted, only used modules will be listed.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> SHOW MODULES;
--------------
| modules |
--------------
| core |
--------------
{code}
h2. YAML Change
“type” is deprecated and removed from the configuration.
{code:yaml}
modules:
- name: core
- name: hive
hive-version: 2.2.1
{code}
Best,
Jane
was (Author: qingyue):
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is
anything missed or not corrected, please let me know. For more information,
please reach to the [discussion
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html].
This summary is also backed up on the [google
doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
----
h2. API Change
h3. ModuleManager
We introduce two methods to support {{USE MODULES}} and {{SHOW FULL MODULES}}.
Beyond that, the behavior of methods {{listModules}}, {{listFunctions}}, and
{{getFunctionDefinition}} will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List<ModuleEntry> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded
but not on the
* name list will become unused but still loaded with relative order
unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(String... names) {
// list all loaded modules with status
}
/** A POJO to represent a module's name and status. */
static class ModuleEntry {
private final String name;
private final boolean status;
ModuleEntry(String name, boolean status) {
this.name = name;
this.status = status;
// getters
}
}
{code}
h3. TableEnvironment
We introduce two methods listFullModules() and useModules(String… names) to
TableEnvironment interface. The impl will simply call
moduleManager#listFullModules() and moduleManager#useModules(String… names).
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides,
there will be a long-term plan to migrate ModuleFactory to extends
org.apache.flink.table.factories.Factory. This plan is currently beyond the
scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
{code:java}
LOAD MODULE module_name WITH ('prop' = 'myProp', ...){code}
Explanation
* WITH is necessary
* Module name x is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery
will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
-------------------
Flink SQL> CREATE MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
-------------------
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the
same relative positions.
{code:java}
UNLOAD MODULE module_name{code}
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does
not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
| ml | false |
-------------------
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
-------------------
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
{code:java}
USE MODULES x [,y, z, ...]{code}
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...]
in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared
module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name
[unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
{code}
h3. SHOW [FULL] MODULES
Display the loaded module names in declared order.
{code:java}
SHOW [FULL] MODULES{code}
Explanation
* If `FULL` is specified, all loaded modules and their status will be listed.
* If `FULL` is omitted, only used modules will be listed.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> SHOW MODULES;
--------------
| modules |
--------------
| core |
--------------
{code}
h2. YAML Change
“type” is deprecated and removed from the configuration.
{code:yaml}
modules:
- name: core
- name: hive
hive-version: 2.2.1
{code}
Best,
Jane
> Support 'load module' and 'unload module' SQL syntax
> ----------------------------------------------------
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
> Affects Versions: 1.13.0
> Reporter: Nicholas Jiang
> Assignee: Jane Chan
> Priority: Major
> Fix For: 1.13.0
>
>
> At present, Flink SQL doesn't support the 'load module' and 'unload module'
> SQL syntax. It's necessary for uses in the situation that users load and
> unload user-defined module through table api or sql client.
> SQL syntax has been proposed in FLIP-68:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
