This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 5c456ace045 [doc](udf) add some doc about java-udf handle static load
(#583)
5c456ace045 is described below
commit 5c456ace045efb95f068ecc98100002def7c5e62
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Wed May 15 18:20:20 2024 +0800
[doc](udf) add some doc about java-udf handle static load (#583)
Co-authored-by: morningman <morning...@163.com>
---
docs/query/udf/java-user-defined-function.md | 338 ++++++++++++---------
.../query/udf/java-user-defined-function.md | 218 +++++++------
.../query/udf/java-user-defined-function.md | 217 ++++++++-----
.../query/udf/java-user-defined-function.md | 215 ++++++++-----
.../query/udf/java-user-defined-function.md | 323 ++++++++++++--------
.../query/udf/java-user-defined-function.md | 320 +++++++++++--------
6 files changed, 995 insertions(+), 636 deletions(-)
diff --git a/docs/query/udf/java-user-defined-function.md
b/docs/query/udf/java-user-defined-function.md
index 09fcafaaf95..78618dec4e1 100644
--- a/docs/query/udf/java-user-defined-function.md
+++ b/docs/query/udf/java-user-defined-function.md
@@ -24,83 +24,25 @@ specific language governing permissions and limitations
under the License.
-->
-# Java UDF
-
-<version since="1.2.0">
-
-Java UDF
-
-</version>
-
-Java UDF provides users with a Java interface written in UDF to facilitate the
execution of user-defined functions in Java language. Compared with native UDF
implementation, Java UDF has the following advantages and limitations:
-1. The advantages
-* Compatibility: Using Java UDF can be compatible with different Doris
versions, so when upgrading Doris version, Java UDF does not need additional
migration. At the same time, Java UDF also follows the same programming
specifications as hive / spark and other engines, so that users can directly
move Hive / Spark UDF jar to Doris.
-* Security: The failure or crash of Java UDF execution will only cause the JVM
to report an error, not the Doris process to crash.
-* Flexibility: In Java UDF, users can package the third-party dependencies
together in the user jar.
-
-2. Restrictions on use
-* Performance: Compared with native UDF, Java UDF will bring additional JNI
overhead, but through batch execution, we have minimized the JNI overhead as
much as possible.
-* Vectorized engine: Java UDF is only supported on vectorized engine now.
-
-### Type correspondence
-
-|Type|UDF Argument Type|
-|----|---------|
-|Bool|Boolean|
-|TinyInt|Byte|
-|SmallInt|Short|
-|Int|Integer|
-|BigInt|Long|
-|LargeInt|BigInteger|
-|Float|Float|
-|Double|Double|
-|Date|LocalDate|
-|Datetime|LocalDateTime|
-|String|String|
-|Decimal|BigDecimal|
-|```struct<Type...>```|```ArrayList<Object>```|
-|```array<Type>```|```ArrayList<Type>```|
-|```map<Type1,Type2>```|```HashMap<Type1,Type2>```|
-
-* Array/Map types can nested other types, Eg: In Doris:
```array<array<int>>``` corresponds to JAVA UDF Argument Type:
```ArrayList<ArrayList<Integer>>```, and so on.
-
-:::caution Warning
-When creating a function, don't use `varchar` instead of `string`, otherwise
the function may fail to execute.
+:::tip
+Java UDF is supported starting from Doris version 1.2.
:::
-## Write UDF functions
+## Introduction to Java UDF
-This section mainly introduces how to develop a Java UDF. Samples for the Java
version are provided under `samples/doris-demo/java-udf-demo/` for your
reference, Check it out
[here](https://github.com/apache/incubator-doris/tree/master/samples/doris-demo/java-udf-demo)
+Java UDF provides users with a Java interface for writing UDFs, making it
convenient for users to execute custom functions using the Java language.
-To use Java UDF, the main entry of UDF must be the `evaluate` function. This
is consistent with other engines such as Hive. In the example of `AddOne`, we
have completed the operation of adding an integer as the UDF.
+Doris supports writing UDFs, UDAFs, and UDTFs using JAVA. Unless otherwise
specified, UDF is used as a general term for all user-defined functions in the
following text.
-It is worth mentioning that this example is not only the Java UDF supported by
Doris, but also the UDF supported by Hive, that's to say, for users, Hive UDF
can be directly migrated to Doris.
+## Creating UDF
-## Create UDF
+The implemented jar package can be placed locally or stored on a remote server
for download via HTTP, but each FE and BE node must be able to access the jar
package.
-```sql
-CREATE FUNCTION
-name ([,...])
-[RETURNS] rettype
-PROPERTIES (["key"="value"][,...])
-```
-Instructions:
+Otherwise, an error message `Couldn't open file ......` will be returned.
-1. `symbol` in properties represents the class name containing UDF classes.
This parameter must be set.
-2. The jar package containing UDF represented by `file` in properties must be
set.
-3. The UDF call type represented by `type` in properties is native by default.
When using java UDF, it is transferred to `Java_UDF`.
-4. In PROPERTIES `always_nullable` indicates whether there may be a NULL value
in the UDF return result. It is an optional parameter. The default value is
true.
-5. `name`: A function belongs to a DB and name is of the
form`dbName`.`funcName`. When `dbName` is not explicitly specified, the db of
the current session is used`dbName`.
+For more syntax help, refer to [CREATE
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Create/CREATE-FUNCTION.md).
-Sample：
-
-```JAVA
-public class AddOne extends UDF {
- public Integer evaluate(Integer value) {
- return value == null ? null : value + 1;
- }
-}
-```
+### UDF
```sql
CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
@@ -110,28 +52,113 @@ CREATE FUNCTION java_udf_add_one(int) RETURNS int
PROPERTIES (
"type"="JAVA_UDF"
);
```
-* "file"=" http://IP:port/udf -code. Jar ", you can also use http to download
jar packages in a multi machine environment.
-* The "always_nullable" is optional attribute, if there is special treatment
for the NULL value in the calculation, it is determined that the result will
not return NULL, and it can be set to false, so that the performance may be
better in the whole calculation process.
+### UDAF
-* If you use the local path method, the jar package that the database driver
depends on, the FE and BE nodes must be placed here
+```sql
+CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
-* The implemented jar package can be stored at local or in a remote server and
downloaded via http, And each FE/BE node must be able to obtain the jar package;
-Otherwise, the error status message "Couldn't open file..." will be returned
+### UDTF
-## Create UDAF
-<br/>
-When using Java code to write UDAF, there are some functions that must be
implemented (mark required) and an inner class State, which will be explained
with a specific example below.
-The following SimpleDemo will implement a simple function similar to sum, the
input parameter is INT, and the output parameter is INT
+:::tip
+UDTF is supported starting from Doris version 3.0.
+:::
-```JAVA
+```sql
+CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string>
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+
+## Using UDF
+
+To utilize UDFs, users must possess the `SELECT` privilege for the
corresponding database.
+
+The usage of UDFs is identical to standard functions, with the primary
distinction being that built-in functions have a global scope, while UDFs are
scoped within the DB.
+
+When the session is linked within the database, directly using the UDF name
will search for the corresponding UDF within the current DB. Otherwise, users
must explicitly specify the UDF's database name, for example, `dbName.funcName`.
+
+## Dropping UDF
+
+If a UDF is no longer needed, it can be dropped using the following command,
as detailed in [DROP
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Drop/DROP-FUNCTION.md).
+
+## Type Correspondence
+
+| Type | UDF Argument Type |
+|-----------------------|------------------------------|
+| Bool | Boolean |
+| TinyInt | Byte |
+| SmallInt | Short |
+| Int | Integer |
+| BigInt | Long |
+| LargeInt | BigInteger |
+| Float | Float |
+| Double | Double |
+| Date | LocalDate |
+| Datetime | LocalDateTime |
+| String | String |
+| Decimal | BigDecimal |
+| `array<Type>` | `ArrayList<Type>` |
+| `map<Type1,Type2>` | `HashMap<Type1,Type2>` |
+| `struct<Type...>` | `ArrayList<Object>` (from version 3.0.0) |
+
+:::tip
+`array/map/struct` types can be nested with other types. For instance, Doris:
`array<array<int>>` corresponds to JAVA UDF Argument Type:
`ArrayList<ArrayList<Integer>>`. Other types follow the same pattern.
+:::
+
+:::caution Warning
+When creating functions, avoid using `varchar` in place of `string`, as this
may cause the function to fail.
+:::
+
+## Writing UDF
+
+This section mainly introduces how to develop a Java UDF. Examples are
provided in `samples/doris-demo/java-udf-demo/` for reference. Click
[here](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
to view them.
+
+When writing a UDF in Java, the main entry point must be the `evaluate`
function. This is consistent with other engines like Hive. In this example, we
write an `AddOne` UDF to perform an increment operation on integer inputs.
+
+It is worth mentioning that this example not only supports Java UDFs in Doris
but is also a UDF supported by Hive. This means that Hive UDFs can be directly
migrated to Doris.
+
+Additionally, if the UDF being defined needs to load large resource files or
if you want to define global static variables, you can refer to the static
variable loading method described at the bottom of the document.
+
+### UDF
+
+```java
+public class AddOne extends UDF {
+ public Integer evaluate(Integer value) {
+ return value == null ? null : value + 1;
+ }
+}
+```
+
+### UDAF
+
+When writing a UDAF using Java code, there are some required functions (marked
as required) and an inner class State that must be implemented. Below is a
specific example to illustrate.
+
+#### Example 1
+
+The following SimpleDemo will implement a simple function similar to sum, with
the input parameter being INT and the output parameter being INT.
+
+```java
package org.apache.doris.udf.demo;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
+import java.util.logging.Logger;
public class SimpleDemo {
+
+ Logger log = Logger.getLogger("SimpleDemo");
+
//Need an inner class to store data
/*required*/
public static class State {
@@ -206,16 +233,9 @@ public class SimpleDemo {
```
-```sql
-CREATE AGGREGATE FUNCTION simple_sum(INT) RETURNS INT PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.SimpleDemo",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+#### Example 2
-```JAVA
+```java
package org.apache.doris.udf.demo;
@@ -225,30 +245,30 @@ import java.math.BigDecimal;
import java.util.Arrays;
import java.util.logging.Logger;
-/*UDAF for calculating the median*/
+/*UDAF 计算中位数*/
public class MedianUDAF {
Logger log = Logger.getLogger("MedianUDAF");
- // State storage
+ //状态存储
public static class State {
- // Precision of the result
+ //返回结果的精度
int scale = 0;
- // Whether this is the first time to execute add() for the data under
a certain aggregation condition of a certain tablet
+ //是否是某一个 tablet 下的某个聚合条件下的数据第一次执行 add 方法
boolean isFirst = true;
- //Data storage
+ //数据存储
public StringBuilder stringBuilder;
}
- //State initialization
+ //状态初始化
public State create() {
State state = new State();
- //Pre-initialize based on the amount of data to be aggregated for each
aggregation condition under each tablet, for improved performance
+ //根据每个 tablet 下的聚合条件需要聚合的数据量大小，预先初始化，增加性能
state.stringBuilder = new StringBuilder(1000);
return state;
}
- // Handle the data for each unit under each aggregation condition for each
tablet
+ //处理执行单位处理各自 tablet 下的各自聚合条件下的每个数据
public void add(State state, Double val, int scale) {
try {
if (val != null && state.isFirst) {
@@ -258,24 +278,24 @@ public class MedianUDAF {
state.stringBuilder.append(val).append(",");
}
} catch (Exception e) {
- // If it is not guaranteed that there will be no exceptions, it is
recommended to maximize the exception capture for each method, as the
processing of java-thrown exceptions is currently not supported
- log.info("Exception encountered while retrieving data: " +
e.getMessage());
+ //如果不能保证一定不会异常，建议每个方法都最大化捕获异常，因为目前不支持处理 java 抛出的异常
+ log.info("获取数据异常：" + e.getMessage());
}
}
- // Output the data after processing for aggregation
+ //处理数据完需要输出等待聚合
public void serialize(State state, DataOutputStream out) {
try {
- // Only DataOutputStream is currently provided, if object
serialization is required, consider methods such as concatenating strings,
converting to json, serializing to byte arrays, etc.
- // If you want to serialize the State object, you may need to
implement the serialization interface for the inner class State yourself
- // In the end, it will be transmitted through DataOutputStream
+ //目前暂时只提供 DataOutputStream，如果需要序列化对象可以考虑拼接字符串，转换 json，序列化成字节数组等方式
+ //如果要序列化 State 对象，可能需要自己将 State 内部类实现序列化接口
+ //最终都是要通过 DataOutputStream 传输
out.writeUTF(state.stringBuilder.toString());
} catch (Exception e) {
- log.info("Exception encountered while serializing data:" +
e.getMessage());
+ log.info("序列化异常：" + e.getMessage());
}
}
- // Retrieve the data output by each data processing unit
+ //获取处理数据执行单位输出的数据
public void deserialize(State state, DataInputStream in) {
try {
String string = in.readUTF();
@@ -283,24 +303,24 @@ public class MedianUDAF {
StringBuilder stringBuilder = new
StringBuilder(string.substring(2));
state.stringBuilder = stringBuilder;
} catch (Exception e) {
- log.info("Exception encountered while deserializing data: " +
e.getMessage());
+ log.info("反序列化异常：" + e.getMessage());
}
}
- // Merge the processing results of data under a certain key according to
the aggregation condition, where state1 is the initialized instance for the
first merge of each key
+ //聚合执行单位按照聚合条件合并某一个键下数据的处理结果 ,每个键第一次合并时，state1 参数是初始化的实例
public void merge(State state1, State state2) {
try {
state1.scale = state2.scale;
state1.stringBuilder.append(state2.stringBuilder.toString());
} catch (Exception e) {
- log.info("Exception encountered while merging results: " +
e.getMessage());
+ log.info("合并结果异常：" + e.getMessage());
}
}
- // Aggregate the data for each key after merging and output the final
result
+ //对每个键合并后的数据进行并输出最终结果
public Double getValue(State state) {
try {
- String[] strings = state.stringBuilder.toString( ).split(",");
+ String[] strings = state.stringBuilder.toString().split(",");
double[] doubles = new double[strings.length + 1];
doubles =
Arrays.stream(strings).mapToDouble(Double::parseDouble).toArray();
@@ -316,12 +336,12 @@ public class MedianUDAF {
BigDecimal decimal = new BigDecimal(value);
return decimal.setScale(state.scale,
BigDecimal.ROUND_HALF_UP).doubleValue();
} catch (Exception e) {
- log.info("Exception encountered while calculating result：" +
e.getMessage());
+ log.info("计算异常：" + e.getMessage());
}
return 0.0;
}
- //This method is executed after each processing unit is completed
+ //每个执行单位执行完都会执行
public void destroy(State state) {
}
@@ -329,22 +349,11 @@ public class MedianUDAF {
```
-```sql
-CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
-
-
-## Create UDTF
+### UDTF
-UDTF, just like UDF functions, requires users to independently implement an
`evaluate` method. However, the return value of a UDTF function must be of
Array type.
-
-In addition, table functions in Doris behave differently depending on the
presence of the _outer suffix. You can refer to
[explode-numbers-outer](../sql-manual/sql-functions/table-functions/explode-numbers-outer).
+Similar to UDFs, UDTFs require users to implement an `evaluate` method.
However, the return value of a UDTF must be of the Array type.
+Additionally, table functions in Doris may exhibit different behaviors due to
the `_outer` suffix. For more details, refer to [OUTER
combinator](../sql-manual/sql-functions/table-functions/explode-numbers-outer.md).
```JAVA
public class UDTFStringTest {
@@ -358,31 +367,72 @@ public class UDTFStringTest {
}
```
-```sql
-CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string>
PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+## Best Practices
+
+## Loading static variables
+
+Currently, in Doris, executing a UDF function, e.g., `select udf(col) from
table`, will load the udf.jar package for each concurrent instance, and unload
the udf.jar package when the instance ends. If the udf.jar file needs to load a
file of several hundred MBs, the memory usage will increase sharply due to
concurrency, potentially leading to OOM (Out of Memory).
+
+The solution is to split the resource loading code, generate a separate jar
package, and have other packages directly reference this resource jar package.
+
+Assume the files have been split into DictLibrary and FunctionUdf.
-## Use UDF
+1. Compile the DictLibrary file separately to generate an independent jar
package, resulting in a resource file DictLibrary.jar:
-Users must have the `SELECT` permission of the corresponding database to use
UDF/UDAF.
+ ```shell
+ javac ./DictLibrary.java
+ jar -cf ./DictLibrary.jar ./DictLibrary.class
+ ```
-The use of UDF is consistent with ordinary function methods. The only
difference is that the scope of built-in functions is global, and the scope of
UDF is internal to DB. When the link session is inside the data, directly using
the UDF name will find the corresponding UDF inside the current DB. Otherwise,
the user needs to display the specified UDF database name, such as
`dbName`.`funcName`.
+2. Then compile the FunctionUdf file, directly referencing the resource
package from the previous step, resulting in the FunctionUdf.jar package:
+
+ ```shell
+ javac -cp ./DictLibrary.jar ./FunctionUdf.java
+ jar -cvf ./FunctionUdf.jar ./FunctionUdf.class
+ ```
+
+3. After the above two steps, you will get two jar packages. To allow the
resource jar package to be referenced by all concurrent instances, place it in
the BE deployment path `be/lib/java_extensions/java-udf`. After restarting BE,
it will be loaded with the JVM startup.
+
+4. Finally, use the `create function` statement to create a UDF function, with
the file path pointing to the FunctionUdf.jar package. This way, the resource
package will be loaded and released with the BE startup and shutdown. The
FunctionUdf.jar will be loaded and released with the SQL execution cycle.
+
+ ```java
+ public class DictLibrary {
+ private static HashMap<String, String> res = new HashMap<>();
+
+ static {
+ // suppose we built this dictionary from a certain local file.
+ res.put("key1", "value1");
+ res.put("key2", "value2");
+ res.put("key3", "value3");
+ res.put("0", "value4");
+ res.put("1", "value5");
+ res.put("2", "value6");
+ }
+
+ public static String evaluate(String key) {
+ if (key == null) {
+ return null;
+ }
+ return res.get(key);
+ }
+ }
+ ```
+
+ ```java
+ public class FunctionUdf {
+ public String evaluate(String key) {
+ String value = DictLibrary.evaluate(key);
+ return value;
+ }
+ }
+ ```
-## Delete UDF
+## Usage Notes
-When you no longer need UDF functions, you can delete a UDF function by the
following command, you can refer to `DROP FUNCTION`.
+1. Complex data types (HLL, Bitmap) are not supported.
-## Example
-Examples of Java UDF are provided in the `samples/doris-demo/java-udf-demo/`
directory. See the `README.md` in each directory for details on how to use it,
Check it out
[here](https://github.com/apache/incubator-doris/tree/master/samples/doris-demo/java-udf-demo)
+2. Users are currently allowed to specify the maximum JVM heap size. The
configuration item is the `-Xmx` part of `JAVA_OPTS` in `be.conf`. The default
is 1024m. If you need to aggregate data, it is recommended to increase this
value to enhance performance and reduce the risk of memory overflow.
-## Instructions
-1. Complex data types (HLL, bitmap) are not supported.
-2. Currently, users are allowed to specify the maximum heap size of the JVM
themselves. The configuration item is jvm_ max_ heap_ size. The configuration
item is in the global configuration file 'be.conf' under the installation
directory of the BE. The default value is 1024M. If data aggregation is
required, it is recommended to increase the value to improve performance and
reduce the risk of memory overflow.
-3. The udf of char type needs to use the String type when creating a function.
-4. Due to the problem that the jvm loads classes with the same name, do not
use multiple classes with the same name as udf implementations at the same
time. If you want to update the udf of a class with the same name, you need to
restart be to reload the classpath.
+3. For Char type UDFs, use the String type when creating the function.
+4. Due to issues with JVM loading classes with the same name, do not use
multiple classes with the same name as UDF implementations simultaneously. If
you want to update a UDF with a class of the same name, you need to restart BE
to reload the classpath.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/udf/java-user-defined-function.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/udf/java-user-defined-function.md
index 5259b8a30d4..7bcc682c1e1 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/udf/java-user-defined-function.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/udf/java-user-defined-function.md
@@ -24,33 +24,74 @@ specific language governing permissions and limitations
under the License.
-->
-
-
-
:::tip
Java UDF 功能自 Doris 1.2 版本开始支持
:::
## Java UDF 介绍
-Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java 语言进行自定义函数的执行。相比于 Native 的 UDF
实现，Java UDF 有如下优势和限制：
-**1. 优势**
+Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java 语言进行自定义函数的执行。
-* 兼容性：使用 Java UDF 可以兼容不同的 Doris 版本，所以在进行 Doris 版本升级时，Java UDF
不需要进行额外的迁移操作。与此同时，Java UDF 同样遵循了和 Hive/Spark 等引擎同样的编程规范，使得用户可以直接将 Hive/Spark 的
UDF jar 包迁移至 Doris 使用。
+Doris 支持使用 JAVA 编写 UDF、UDAF 和 UDTF。下文如无特殊说明，使用 UDF 统称所有用户自定义函数。
-* 安全：Java UDF 执行失败或崩溃仅会导致 JVM 报错，而不会导致 Doris 进程崩溃。
+## 创建 UDF
-* 灵活：Java UDF 中用户通过把第三方依赖打进用户 jar 包，而不需要额外处理引入的三方库。
+实现的 jar 包可以放在本地也可以存放在远程服务端通过 HTTP 下载，但必须让每个 FE 和 BE 节点都能获取到 jar 包。
-**2. 使用限制**
+否则将会返回错误状态信息 `Couldn't open file ......`。
-* 性能：相比于 Native UDF，Java UDF 会带来额外的 JNI 开销，不过通过批式执行的方式，我们已经尽可能的将 JNI 开销降到最低。
+更多语法帮助可参阅 [CREATE
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Create/CREATE-FUNCTION.md).
-* 向量化引擎：Java UDF 当前只支持向量化引擎。
+### UDF
+```sql
+CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
+ "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
+ "symbol"="org.apache.doris.udf.AddOne",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+### UDAF
-**3. 类型对应关系**
+```sql
+CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+
+### UDTF
+
+:::tip
+UDTF 自 Doris 3.0 版本开始支持
+:::
+
+```sql
+CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string>
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+
+## 使用 UDF
+
+用户使用 UDF 必须拥有对应数据库的 `SELECT` 权限。
+
+UDF 的使用与普通的函数方式一致，唯一的区别在于，内置函数的作用域是全局的，而 UDF 的作用域是 DB 内部。
+
+当链接 session 位于数据内部时，直接使用 UDF 名字会在当前 DB 内部查找对应的 UDF。否则用户需要显示的指定 UDF 的数据库名字，例如
`dbName.funcName`。
+
+## 删除 UDF
+
+当你不再需要 UDF 函数时，你可以通过下述命令来删除一个 UDF 函数，可以参考 [DROP
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Drop/DROP-FUNCTION.md)
+
+## 类型对应关系
|Type|UDF Argument Type|
|----|---------|
@@ -66,48 +107,32 @@ Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java
|Datetime|LocalDateTime|
|String|String|
|Decimal|BigDecimal|
-|```struct<Type...>```|```ArrayList<Object>```|
-|```array<Type>```|```ArrayList<Type>```|
-|```map<Type1,Type2>```|```HashMap<Type1,Type2>```|
+|```array<Type>```|```ArrayList<Type>``` 支持嵌套 |
+|```map<Type1,Type2>```|```HashMap<Type1,Type2>``` 支持嵌套 |
+|```struct<Type...>```|```ArrayList<Object>``` 3.0.0 版本开始支持 |
:::tip
-array/map类型可以嵌套其它类型，例如 Doris: ```array<array<int>>```对应 JAVA UDF Argument
Type: ```ArrayList<ArrayList<Integer>>```, 其他依此类推
+`array/map/struct` 类型可以嵌套其它类型，例如 Doris: ```array<array<int>>```对应 JAVA UDF
Argument Type: ```ArrayList<ArrayList<Integer>>```, 其他依此类推
:::
:::caution Warning
在创建函数的时候，不要使用 `varchar` 代替 `string`，否则函数可能执行失败。
:::
-## 编写 UDF 函数
+## UDF 的编写
本小节主要介绍如何开发一个 Java UDF。在 `samples/doris-demo/java-udf-demo/`
下提供了示例，可供参考，查看点击[这里](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
使用 Java 代码编写 UDF，UDF 的主入口必须为 `evaluate` 函数。这一点与 Hive 等其他引擎保持一致。在本示例中，我们编写了
`AddOne` UDF 来完成对整型输入进行加一的操作。
-值得一提的是，本例不只是 Doris 支持的 Java UDF，同时还是 Hive 支持的 UDF，也就是说，对于用户来讲，Hive UDF
是可以直接迁移至 Doris 的。
-## 创建 UDF
-
-```sql
-CREATE FUNCTION
-name ([,...])
-[RETURNS] rettype
-PROPERTIES (["key"="value"][,...])
-```
-说明：
-
-1. PROPERTIES 中`symbol`表示的是包含 UDF 类的类名，这个参数是必须设定的。
-
-2. PROPERTIES 中`file`表示的包含用户 UDF 的 jar 包，这个参数是必须设定的。
-
-3. PROPERTIES 中`type`表示的 UDF 调用类型，默认为 Native，使用 Java UDF 时传 JAVA_UDF。
+值得一提的是，本例不只是 Doris 支持的 Java UDF，同时还是 Hive 支持的 UDF，也就是说，对于用户来讲，Hive UDF
是可以直接迁移至 Doris 的。
-4. PROPERTIES 中`always_nullable`表示的 UDF 返回结果中是否有可能出现 NULL 值，是可选参数，默认值为 true。
+另外，如果定义的 UDF 中需要加载很大的资源文件，或者希望可以定义全局的 static 变量，可以参照文档下方的 static 变量加载方式。
-5. name: 一个 function 是要归属于某个 DB 的，name
的形式为`dbName`.`funcName`。当`dbName`没有明确指定的时候，就是使用当前 session 所在的 db 作为`dbName`。
-示例：
+### UDF
-```JAVA
+```java
public class AddOne extends UDF {
public Integer evaluate(Integer value) {
return value == null ? null : value + 1;
@@ -115,31 +140,13 @@ public class AddOne extends UDF {
}
```
-```sql
-CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
- "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
- "symbol"="org.apache.doris.udf.AddOne",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
-
-* "file"="http://IP:port/udf-code.jar";, 当在多机环境时，也可以使用 http 的方式下载 jar 包
-
-* "always_nullable"可选属性，如果在计算中对出现的 NULL 值有特殊处理，确定结果中不会返回 NULL，可以设为
false，这样在整个查询计算过程中性能可能更好些。
+### UDAF
-* 如果你是**本地路径**方式，这里数据库驱动依赖的 jar 包，**FE、BE 节点都要放置**
-
-:::tip
-实现的 jar 包可以放在本地也可以存放在远程服务端通过 HTTP 下载，但必须让每个 FE/BE 节点都能获取到 jar 包;
-- 否则将会返回错误状态信息"Couldn't open file ......".
-:::
+在使用 Java 代码编写 UDAF 时，有一些必须实现的函数 (标记 required) 和一个内部类 State，下面将以一个具体的实例来说明。
-## 编写 UDAF 函数
+#### 示例1
-
-在使用 Java 代码编写 UDAF 时，有一些必须实现的函数 (标记 required) 和一个内部类 State，下面将以一个具体的实例来说明
-下面的 SimpleDemo 将实现一个类似的 sum 的简单函数，输入参数 INT，输出参数是 INT
+下面的 SimpleDemo 将实现一个类似的 sum 的简单函数，输入参数 INT，输出参数是 INT。
```java
package org.apache.doris.udf.demo;
@@ -227,14 +234,7 @@ public class SimpleDemo {
```
-```sql
-CREATE AGGREGATE FUNCTION simple_sum(INT) RETURNS INT PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.SimpleDemo",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+#### 示例2
```java
package org.apache.doris.udf.demo;
@@ -350,19 +350,11 @@ public class MedianUDAF {
```
-```sql
-CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+### UDTF
-## 编写 UDTF 函数
-UDTF 和 UDF 函数一样，需要用户自主实现一个 `evaluate` 方法，但是 UDTF 函数的返回值必须是 Array 类型。
-另外 Doris 中表函数会因为 _outer 后缀有不同的表现，可查看[OUTER
组合器](../../sql-manual/sql-functions/table-functions/explode-numbers-outer)
+UDTF 和 UDF 函数一样，需要用户自主实现一个 `evaluate` 方法， 但是 UDTF 函数的返回值必须是 Array 类型。
+另外Doris中表函数会因为 `_outer` 后缀有不同的表现，可查看[OUTER
组合器](../sql-manual/sql-functions/table-functions/explode-numbers-outer.md)
```JAVA
public class UDTFStringTest {
@@ -376,34 +368,72 @@ public class UDTFStringTest {
}
```
-```sql
-CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string>
PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+## 最佳实践
+## static 变量加载
-## 使用 UDF
+当前在 Doris 中，执行一个 UDF 函数，eg: `select udf(col) from table`,
每一个并发instance会加载一次udf.jar包，在该instance结束时卸载掉udf.jar包。
+所以当 udf.jar 文件中需要加载一个几百 MB的文件时，会因为并发的原因，使得占据的内存急剧增大，容易OOM。
-用户使用 UDF 必须拥有对应数据库的 `SELECT` 权限。
+解决方法是可以将资源加载代码拆分开，单独生成一个 jar 包文件，其他包直接引用该资源jar包.
-UDF 的使用与普通的函数方式一致，唯一的区别在于，内置函数的作用域是全局的，而 UDF 的作用域是 DB 内部。当链接 session
位于数据内部时，直接使用 UDF 名字会在当前 DB 内部查找对应的 UDF。否则用户需要显示的指定 UDF 的数据库名字，例如
`dbName`.`funcName`。
+假设已经拆分为了 DictLibrary 和 FunctionUdf 两个文件。
-## 删除 UDF
+1. 单独编译 DictLibrary 文件，使其生成一个独立的 jar 包,这样可以得到一个资源文件 DictLibrary.jar:
-当你不再需要 UDF 函数时，你可以通过下述命令来删除一个 UDF 函数，可以参考 `DROP FUNCTION`。
+ ```shell
+ javac ./DictLibrary.java
+ jar -cf ./DictLibrary.jar ./DictLibrary.class
+ ```
-## 示例
-在`samples/doris-demo/java-udf-demo/`
目录中提供了具体示例。具体使用方法见每个目录下的`README.md`，查看点击[这里](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
+2. 然后编译 FunctionUdf 文件，可以直接引用上一步的到的资源包, 这样可以得到 udf 的 FunctionUdf.jar包。
+
+ ```shell
+ javac -cp ./DictLibrary.jar ./FunctionUdf.java
+ jar -cvf ./FunctionUdf.jar ./FunctionUdf.class
+ ```
+
+3. 经过上面两步之后，会得到两个 jar 包，由于想让资源 jar 包被所有的并发引用，所以需要将它放到 BE 的部署路径
`be/lib/java_extensions/java-udf `下面，BE重启之后就可以随着 JVM 的启动加载进来。
+
+4. 最后利用 `create function` 语句创建一个 UDF 函数，其中 file 的路径指向 FunctionUdf.jar 包,
这样资源包会随着 BE 启动而加载，停止而释放。FunctionUdf.jar 的加载与释放则是跟随 SQL 的执行周期。
+
+ ```java
+ public class DictLibrary {
+ private static HashMap<String, String> res = new HashMap<>();
+
+ static {
+ // suppose we built this dictionary from a certain local file.
+ res.put("key1", "value1");
+ res.put("key2", "value2");
+ res.put("key3", "value3");
+ res.put("0", "value4");
+ res.put("1", "value5");
+ res.put("2", "value6");
+ }
+
+ public static String evaluate(String key) {
+ if (key == null) {
+ return null;
+ }
+ return res.get(key);
+ }
+ }
+ ```
+
+ ```java
+ public class FunctionUdf {
+ public String evaluate(String key) {
+ String value = DictLibrary.evaluate(key);
+ return value;
+ }
+ }
+ ```
## 使用须知
1. 不支持复杂数据类型（HLL，Bitmap）。
-2. 当前允许用户自己指定 JVM 最大堆大小，配置项是 be.conf 中的 JAVA_OPTS 的 -Xmx 部分。默认
1024m，如果需要聚合数据，建议调大一些，增加性能，减少内存溢出风险。
+2. 当前允许用户自己指定JVM最大堆大小，配置项是 be.conf 中的 `JAVA_OPTS` 的 -Xmx 部分。默认
1024m，如果需要聚合数据，建议调大一些，增加性能，减少内存溢出风险。
3. Char 类型的 UDF 在 create function 时需要使用 String 类型。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/query/udf/java-user-defined-function.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/query/udf/java-user-defined-function.md
index 1da156242f4..7bcc682c1e1 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/query/udf/java-user-defined-function.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/query/udf/java-user-defined-function.md
@@ -24,33 +24,74 @@ specific language governing permissions and limitations
under the License.
-->
-
-
-
:::tip
Java UDF 功能自 Doris 1.2 版本开始支持
:::
## Java UDF 介绍
-Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java 语言进行自定义函数的执行。相比于 Native 的 UDF
实现，Java UDF 有如下优势和限制：
-**1. 优势**
+Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java 语言进行自定义函数的执行。
+
+Doris 支持使用 JAVA 编写 UDF、UDAF 和 UDTF。下文如无特殊说明，使用 UDF 统称所有用户自定义函数。
+
+## 创建 UDF
+
+实现的 jar 包可以放在本地也可以存放在远程服务端通过 HTTP 下载，但必须让每个 FE 和 BE 节点都能获取到 jar 包。
+
+否则将会返回错误状态信息 `Couldn't open file ......`。
+
+更多语法帮助可参阅 [CREATE
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Create/CREATE-FUNCTION.md).
+
+### UDF
+
+```sql
+CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
+ "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
+ "symbol"="org.apache.doris.udf.AddOne",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+
+### UDAF
+
+```sql
+CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+
+### UDTF
-* 兼容性：使用 Java UDF 可以兼容不同的 Doris 版本，所以在进行 Doris 版本升级时，Java UDF
不需要进行额外的迁移操作。与此同时，Java UDF 同样遵循了和 Hive/Spark 等引擎同样的编程规范，使得用户可以直接将 Hive/Spark 的
UDF jar 包迁移至 Doris 使用。
+:::tip
+UDTF 自 Doris 3.0 版本开始支持
+:::
-* 安全：Java UDF 执行失败或崩溃仅会导致 JVM 报错，而不会导致 Doris 进程崩溃。
+```sql
+CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string>
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
-* 灵活：Java UDF 中用户通过把第三方依赖打进用户 jar 包，而不需要额外处理引入的三方库。
+## 使用 UDF
-**2. 使用限制**
+用户使用 UDF 必须拥有对应数据库的 `SELECT` 权限。
-* 性能：相比于 Native UDF，Java UDF 会带来额外的 JNI 开销，不过通过批式执行的方式，我们已经尽可能的将 JNI 开销降到最低。
+UDF 的使用与普通的函数方式一致，唯一的区别在于，内置函数的作用域是全局的，而 UDF 的作用域是 DB 内部。
-* 向量化引擎：Java UDF 当前只支持向量化引擎。
+当链接 session 位于数据内部时，直接使用 UDF 名字会在当前 DB 内部查找对应的 UDF。否则用户需要显示的指定 UDF 的数据库名字，例如
`dbName.funcName`。
+## 删除 UDF
+当你不再需要 UDF 函数时，你可以通过下述命令来删除一个 UDF 函数，可以参考 [DROP
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Drop/DROP-FUNCTION.md)
-**3. 类型对应关系**
+## 类型对应关系
|Type|UDF Argument Type|
|----|---------|
@@ -66,65 +107,46 @@ Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java
|Datetime|LocalDateTime|
|String|String|
|Decimal|BigDecimal|
-|```array<Type>```|```ArrayList<Type>```|
-|```map<Type1,Type2>```|```HashMap<Type1,Type2>```|
+|```array<Type>```|```ArrayList<Type>``` 支持嵌套 |
+|```map<Type1,Type2>```|```HashMap<Type1,Type2>``` 支持嵌套 |
+|```struct<Type...>```|```ArrayList<Object>``` 3.0.0 版本开始支持 |
:::tip
-array/map类型可以嵌套其它类型，例如 Doris: ```array<array<int>>```对应 JAVA UDF Argument
Type: ```ArrayList<ArrayList<Integer>>```, 其他依此类推
+`array/map/struct` 类型可以嵌套其它类型，例如 Doris: ```array<array<int>>```对应 JAVA UDF
Argument Type: ```ArrayList<ArrayList<Integer>>```, 其他依此类推
:::
:::caution Warning
在创建函数的时候，不要使用 `varchar` 代替 `string`，否则函数可能执行失败。
:::
-## 编写 UDF 函数
+## UDF 的编写
本小节主要介绍如何开发一个 Java UDF。在 `samples/doris-demo/java-udf-demo/`
下提供了示例，可供参考，查看点击[这里](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
使用 Java 代码编写 UDF，UDF 的主入口必须为 `evaluate` 函数。这一点与 Hive 等其他引擎保持一致。在本示例中，我们编写了
`AddOne` UDF 来完成对整型输入进行加一的操作。
-值得一提的是，本例不只是 Doris 支持的 Java UDF，同时还是 Hive 支持的 UDF，也就是说，对于用户来讲，Hive UDF
是可以直接迁移至 Doris 的。
-
-## 创建 UDF
-
-```sql
-CREATE FUNCTION
-name ([,...])
-[RETURNS] rettype
-PROPERTIES (["key"="value"][,...])
-```
-说明：
-
-1. PROPERTIES 中`symbol`表示的是包含 UDF 类的类名，这个参数是必须设定的。
-2. PROPERTIES 中`file`表示的包含用户 UDF 的 jar 包，这个参数是必须设定的。
+值得一提的是，本例不只是 Doris 支持的 Java UDF，同时还是 Hive 支持的 UDF，也就是说，对于用户来讲，Hive UDF
是可以直接迁移至 Doris 的。
-3. PROPERTIES 中`type`表示的 UDF 调用类型，默认为 Native，使用 Java UDF 时传 JAVA_UDF。
+另外，如果定义的 UDF 中需要加载很大的资源文件，或者希望可以定义全局的 static 变量，可以参照文档下方的 static 变量加载方式。
-4. PROPERTIES 中`always_nullable`表示的 UDF 返回结果中是否有可能出现 NULL 值，是可选参数，默认值为 true。
-5. name: 一个 function 是要归属于某个 DB 的，name
的形式为`dbName`.`funcName`。当`dbName`没有明确指定的时候，就是使用当前 session 所在的 db 作为`dbName`。
+### UDF
-示例：
-```sql
-CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
- "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
- "symbol"="org.apache.doris.udf.AddOne",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
+```java
+public class AddOne extends UDF {
+ public Integer evaluate(Integer value) {
+ return value == null ? null : value + 1;
+ }
+}
```
-* "file"="http://IP:port/udf-code.jar";, 当在多机环境时，也可以使用 http 的方式下载 jar 包
-
-* "always_nullable"可选属性，如果在计算中对出现的 NULL 值有特殊处理，确定结果中不会返回 NULL，可以设为
false，这样在整个查询计算过程中性能可能更好些。
+### UDAF
-* 如果你是**本地路径**方式，这里数据库驱动依赖的 jar 包，**FE、BE 节点都要放置**
+在使用 Java 代码编写 UDAF 时，有一些必须实现的函数 (标记 required) 和一个内部类 State，下面将以一个具体的实例来说明。
-## 编写 UDAF 函数
+#### 示例1
-
-在使用 Java 代码编写 UDAF 时，有一些必须实现的函数 (标记 required) 和一个内部类 State，下面将以一个具体的实例来说明
-下面的 SimpleDemo 将实现一个类似的 sum 的简单函数，输入参数 INT，输出参数是 INT
+下面的 SimpleDemo 将实现一个类似的 sum 的简单函数，输入参数 INT，输出参数是 INT。
```java
package org.apache.doris.udf.demo;
@@ -212,14 +234,7 @@ public class SimpleDemo {
```
-```sql
-CREATE AGGREGATE FUNCTION simple_sum(INT) RETURNS INT PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.SimpleDemo",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+#### 示例2
```java
package org.apache.doris.udf.demo;
@@ -335,42 +350,90 @@ public class MedianUDAF {
```
-```sql
-CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
+### UDTF
+
+UDTF 和 UDF 函数一样，需要用户自主实现一个 `evaluate` 方法， 但是 UDTF 函数的返回值必须是 Array 类型。
+
+另外Doris中表函数会因为 `_outer` 后缀有不同的表现，可查看[OUTER
组合器](../sql-manual/sql-functions/table-functions/explode-numbers-outer.md)
+
+```JAVA
+public class UDTFStringTest {
+ public ArrayList<String> evaluate(String value, String separator) {
+ if (value == null || separator == null) {
+ return null;
+ } else {
+ return new ArrayList<>(Arrays.asList(value.split(separator)));
+ }
+ }
+}
```
-:::tip
-实现的 jar 包可以放在本地也可以存放在远程服务端通过 HTTP 下载，但必须让每个 BE 节点都能获取到 jar 包;
+## 最佳实践
-- 否则将会返回错误状态信息"Couldn't open file ......".
+## static 变量加载
-- 目前还暂不支持 UDTF
-:::
+当前在 Doris 中，执行一个 UDF 函数，eg: `select udf(col) from table`,
每一个并发instance会加载一次udf.jar包，在该instance结束时卸载掉udf.jar包。
+所以当 udf.jar 文件中需要加载一个几百 MB的文件时，会因为并发的原因，使得占据的内存急剧增大，容易OOM。
+解决方法是可以将资源加载代码拆分开，单独生成一个 jar 包文件，其他包直接引用该资源jar包.
-## 使用 UDF
+假设已经拆分为了 DictLibrary 和 FunctionUdf 两个文件。
-用户使用 UDF 必须拥有对应数据库的 `SELECT` 权限。
+1. 单独编译 DictLibrary 文件，使其生成一个独立的 jar 包,这样可以得到一个资源文件 DictLibrary.jar:
-UDF 的使用与普通的函数方式一致，唯一的区别在于，内置函数的作用域是全局的，而 UDF 的作用域是 DB 内部。当链接 session
位于数据内部时，直接使用 UDF 名字会在当前 DB 内部查找对应的 UDF。否则用户需要显示的指定 UDF 的数据库名字，例如
`dbName`.`funcName`。
+ ```shell
+ javac ./DictLibrary.java
+ jar -cf ./DictLibrary.jar ./DictLibrary.class
+ ```
-## 删除 UDF
+2. 然后编译 FunctionUdf 文件，可以直接引用上一步的到的资源包, 这样可以得到 udf 的 FunctionUdf.jar包。
+
+ ```shell
+ javac -cp ./DictLibrary.jar ./FunctionUdf.java
+ jar -cvf ./FunctionUdf.jar ./FunctionUdf.class
+ ```
+
+3. 经过上面两步之后，会得到两个 jar 包，由于想让资源 jar 包被所有的并发引用，所以需要将它放到 BE 的部署路径
`be/lib/java_extensions/java-udf `下面，BE重启之后就可以随着 JVM 的启动加载进来。
+
+4. 最后利用 `create function` 语句创建一个 UDF 函数，其中 file 的路径指向 FunctionUdf.jar 包,
这样资源包会随着 BE 启动而加载，停止而释放。FunctionUdf.jar 的加载与释放则是跟随 SQL 的执行周期。
+
+ ```java
+ public class DictLibrary {
+ private static HashMap<String, String> res = new HashMap<>();
+
+ static {
+ // suppose we built this dictionary from a certain local file.
+ res.put("key1", "value1");
+ res.put("key2", "value2");
+ res.put("key3", "value3");
+ res.put("0", "value4");
+ res.put("1", "value5");
+ res.put("2", "value6");
+ }
-当你不再需要 UDF 函数时，你可以通过下述命令来删除一个 UDF 函数，可以参考 `DROP FUNCTION`。
+ public static String evaluate(String key) {
+ if (key == null) {
+ return null;
+ }
+ return res.get(key);
+ }
+ }
+ ```
-## 示例
-在`samples/doris-demo/java-udf-demo/`
目录中提供了具体示例。具体使用方法见每个目录下的`README.md`，查看点击[这里](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
+ ```java
+ public class FunctionUdf {
+ public String evaluate(String key) {
+ String value = DictLibrary.evaluate(key);
+ return value;
+ }
+ }
+ ```
## 使用须知
1. 不支持复杂数据类型（HLL，Bitmap）。
-2. 当前允许用户自己指定 JVM 最大堆大小，配置项是 jvm_max_heap_size。配置项在 BE 安装目录下的 be.conf 全局配置中，默认
512M，如果需要聚合数据，建议调大一些，增加性能，减少内存溢出风险。
+2. 当前允许用户自己指定JVM最大堆大小，配置项是 be.conf 中的 `JAVA_OPTS` 的 -Xmx 部分。默认
1024m，如果需要聚合数据，建议调大一些，增加性能，减少内存溢出风险。
3. Char 类型的 UDF 在 create function 时需要使用 String 类型。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/query/udf/java-user-defined-function.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/query/udf/java-user-defined-function.md
index 6845ad284a5..7bcc682c1e1 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/query/udf/java-user-defined-function.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/query/udf/java-user-defined-function.md
@@ -24,33 +24,74 @@ specific language governing permissions and limitations
under the License.
-->
-
-
-
:::tip
Java UDF 功能自 Doris 1.2 版本开始支持
:::
## Java UDF 介绍
-Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java 语言进行自定义函数的执行。相比于 Native 的 UDF
实现，Java UDF 有如下优势和限制：
-**1. 优势**
+Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java 语言进行自定义函数的执行。
-* 兼容性：使用 Java UDF 可以兼容不同的 Doris 版本，所以在进行 Doris 版本升级时，Java UDF
不需要进行额外的迁移操作。与此同时，Java UDF 同样遵循了和 Hive/Spark 等引擎同样的编程规范，使得用户可以直接将 Hive/Spark 的
UDF jar 包迁移至 Doris 使用。
+Doris 支持使用 JAVA 编写 UDF、UDAF 和 UDTF。下文如无特殊说明，使用 UDF 统称所有用户自定义函数。
-* 安全：Java UDF 执行失败或崩溃仅会导致 JVM 报错，而不会导致 Doris 进程崩溃。
+## 创建 UDF
-* 灵活：Java UDF 中用户通过把第三方依赖打进用户 jar 包，而不需要额外处理引入的三方库。
+实现的 jar 包可以放在本地也可以存放在远程服务端通过 HTTP 下载，但必须让每个 FE 和 BE 节点都能获取到 jar 包。
-**2. 使用限制**
+否则将会返回错误状态信息 `Couldn't open file ......`。
-* 性能：相比于 Native UDF，Java UDF 会带来额外的 JNI 开销，不过通过批式执行的方式，我们已经尽可能的将 JNI 开销降到最低。
+更多语法帮助可参阅 [CREATE
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Create/CREATE-FUNCTION.md).
-* 向量化引擎：Java UDF 当前只支持向量化引擎。
+### UDF
+```sql
+CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
+ "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
+ "symbol"="org.apache.doris.udf.AddOne",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+### UDAF
-**3. 类型对应关系**
+```sql
+CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+
+### UDTF
+
+:::tip
+UDTF 自 Doris 3.0 版本开始支持
+:::
+
+```sql
+CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string>
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
+
+## 使用 UDF
+
+用户使用 UDF 必须拥有对应数据库的 `SELECT` 权限。
+
+UDF 的使用与普通的函数方式一致，唯一的区别在于，内置函数的作用域是全局的，而 UDF 的作用域是 DB 内部。
+
+当链接 session 位于数据内部时，直接使用 UDF 名字会在当前 DB 内部查找对应的 UDF。否则用户需要显示的指定 UDF 的数据库名字，例如
`dbName.funcName`。
+
+## 删除 UDF
+
+当你不再需要 UDF 函数时，你可以通过下述命令来删除一个 UDF 函数，可以参考 [DROP
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Drop/DROP-FUNCTION.md)
+
+## 类型对应关系
|Type|UDF Argument Type|
|----|---------|
@@ -66,47 +107,32 @@ Java UDF 为用户提供 UDF 编写的 Java 接口，以方便用户使用 Java
|Datetime|LocalDateTime|
|String|String|
|Decimal|BigDecimal|
-|```array<Type>```|```ArrayList<Type>```|
-|```map<Type1,Type2>```|```HashMap<Type1,Type2>```|
+|```array<Type>```|```ArrayList<Type>``` 支持嵌套 |
+|```map<Type1,Type2>```|```HashMap<Type1,Type2>``` 支持嵌套 |
+|```struct<Type...>```|```ArrayList<Object>``` 3.0.0 版本开始支持 |
:::tip
-array/map类型可以嵌套其它类型，例如 Doris: ```array<array<int>>```对应 JAVA UDF Argument
Type: ```ArrayList<ArrayList<Integer>>```, 其他依此类推
+`array/map/struct` 类型可以嵌套其它类型，例如 Doris: ```array<array<int>>```对应 JAVA UDF
Argument Type: ```ArrayList<ArrayList<Integer>>```, 其他依此类推
:::
:::caution Warning
在创建函数的时候，不要使用 `varchar` 代替 `string`，否则函数可能执行失败。
:::
-## 编写 UDF 函数
+## UDF 的编写
本小节主要介绍如何开发一个 Java UDF。在 `samples/doris-demo/java-udf-demo/`
下提供了示例，可供参考，查看点击[这里](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
使用 Java 代码编写 UDF，UDF 的主入口必须为 `evaluate` 函数。这一点与 Hive 等其他引擎保持一致。在本示例中，我们编写了
`AddOne` UDF 来完成对整型输入进行加一的操作。
-值得一提的是，本例不只是 Doris 支持的 Java UDF，同时还是 Hive 支持的 UDF，也就是说，对于用户来讲，Hive UDF
是可以直接迁移至 Doris 的。
-
-## 创建 UDF
-
-```sql
-CREATE FUNCTION
-name ([,...])
-[RETURNS] rettype
-PROPERTIES (["key"="value"][,...])
-```
-说明：
-
-1. PROPERTIES 中`symbol`表示的是包含 UDF 类的类名，这个参数是必须设定的。
-
-2. PROPERTIES 中`file`表示的包含用户 UDF 的 jar 包，这个参数是必须设定的。
-3. PROPERTIES 中`type`表示的 UDF 调用类型，默认为 Native，使用 Java UDF 时传 JAVA_UDF。
+值得一提的是，本例不只是 Doris 支持的 Java UDF，同时还是 Hive 支持的 UDF，也就是说，对于用户来讲，Hive UDF
是可以直接迁移至 Doris 的。
-4. PROPERTIES 中`always_nullable`表示的 UDF 返回结果中是否有可能出现 NULL 值，是可选参数，默认值为 true。
+另外，如果定义的 UDF 中需要加载很大的资源文件，或者希望可以定义全局的 static 变量，可以参照文档下方的 static 变量加载方式。
-5. name: 一个 function 是要归属于某个 DB 的，name
的形式为`dbName`.`funcName`。当`dbName`没有明确指定的时候，就是使用当前 session 所在的 db 作为`dbName`。
-示例：
+### UDF
-```JAVA
+```java
public class AddOne extends UDF {
public Integer evaluate(Integer value) {
return value == null ? null : value + 1;
@@ -114,26 +140,13 @@ public class AddOne extends UDF {
}
```
-```sql
-CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
- "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
- "symbol"="org.apache.doris.udf.AddOne",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
-
-* "file"="http://IP:port/udf-code.jar";, 当在多机环境时，也可以使用 http 的方式下载 jar 包
+### UDAF
-* "always_nullable"可选属性，如果在计算中对出现的 NULL 值有特殊处理，确定结果中不会返回 NULL，可以设为
false，这样在整个查询计算过程中性能可能更好些。
+在使用 Java 代码编写 UDAF 时，有一些必须实现的函数 (标记 required) 和一个内部类 State，下面将以一个具体的实例来说明。
-* 如果你是**本地路径**方式，这里数据库驱动依赖的 jar 包，**FE、BE 节点都要放置**
+#### 示例1
-## 编写 UDAF 函数
-
-
-在使用 Java 代码编写 UDAF 时，有一些必须实现的函数 (标记 required) 和一个内部类 State，下面将以一个具体的实例来说明
-下面的 SimpleDemo 将实现一个类似的 sum 的简单函数，输入参数 INT，输出参数是 INT
+下面的 SimpleDemo 将实现一个类似的 sum 的简单函数，输入参数 INT，输出参数是 INT。
```java
package org.apache.doris.udf.demo;
@@ -221,14 +234,7 @@ public class SimpleDemo {
```
-```sql
-CREATE AGGREGATE FUNCTION simple_sum(INT) RETURNS INT PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.SimpleDemo",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+#### 示例2
```java
package org.apache.doris.udf.demo;
@@ -344,43 +350,90 @@ public class MedianUDAF {
```
-```sql
-CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
+### UDTF
+
+UDTF 和 UDF 函数一样，需要用户自主实现一个 `evaluate` 方法， 但是 UDTF 函数的返回值必须是 Array 类型。
+
+另外Doris中表函数会因为 `_outer` 后缀有不同的表现，可查看[OUTER
组合器](../sql-manual/sql-functions/table-functions/explode-numbers-outer.md)
+
+```JAVA
+public class UDTFStringTest {
+ public ArrayList<String> evaluate(String value, String separator) {
+ if (value == null || separator == null) {
+ return null;
+ } else {
+ return new ArrayList<>(Arrays.asList(value.split(separator)));
+ }
+ }
+}
```
-:::tip
-实现的 jar 包可以放在本地也可以存放在远程服务端通过 HTTP 下载，但必须让每个 BE 节点都能获取到 jar 包;
+## 最佳实践
-- 否则将会返回错误状态信息"Couldn't open file ......".
+## static 变量加载
-- 目前还暂不支持 UDTF
+当前在 Doris 中，执行一个 UDF 函数，eg: `select udf(col) from table`,
每一个并发instance会加载一次udf.jar包，在该instance结束时卸载掉udf.jar包。
+所以当 udf.jar 文件中需要加载一个几百 MB的文件时，会因为并发的原因，使得占据的内存急剧增大，容易OOM。
- :::
+解决方法是可以将资源加载代码拆分开，单独生成一个 jar 包文件，其他包直接引用该资源jar包.
+假设已经拆分为了 DictLibrary 和 FunctionUdf 两个文件。
-## 使用 UDF
+1. 单独编译 DictLibrary 文件，使其生成一个独立的 jar 包,这样可以得到一个资源文件 DictLibrary.jar:
-用户使用 UDF 必须拥有对应数据库的 `SELECT` 权限。
+ ```shell
+ javac ./DictLibrary.java
+ jar -cf ./DictLibrary.jar ./DictLibrary.class
+ ```
-UDF 的使用与普通的函数方式一致，唯一的区别在于，内置函数的作用域是全局的，而 UDF 的作用域是 DB 内部。当链接 session
位于数据内部时，直接使用 UDF 名字会在当前 DB 内部查找对应的 UDF。否则用户需要显示的指定 UDF 的数据库名字，例如
`dbName`.`funcName`。
+2. 然后编译 FunctionUdf 文件，可以直接引用上一步的到的资源包, 这样可以得到 udf 的 FunctionUdf.jar包。
-## 删除 UDF
+ ```shell
+ javac -cp ./DictLibrary.jar ./FunctionUdf.java
+ jar -cvf ./FunctionUdf.jar ./FunctionUdf.class
+ ```
+
+3. 经过上面两步之后，会得到两个 jar 包，由于想让资源 jar 包被所有的并发引用，所以需要将它放到 BE 的部署路径
`be/lib/java_extensions/java-udf `下面，BE重启之后就可以随着 JVM 的启动加载进来。
+
+4. 最后利用 `create function` 语句创建一个 UDF 函数，其中 file 的路径指向 FunctionUdf.jar 包,
这样资源包会随着 BE 启动而加载，停止而释放。FunctionUdf.jar 的加载与释放则是跟随 SQL 的执行周期。
+
+ ```java
+ public class DictLibrary {
+ private static HashMap<String, String> res = new HashMap<>();
+
+ static {
+ // suppose we built this dictionary from a certain local file.
+ res.put("key1", "value1");
+ res.put("key2", "value2");
+ res.put("key3", "value3");
+ res.put("0", "value4");
+ res.put("1", "value5");
+ res.put("2", "value6");
+ }
-当你不再需要 UDF 函数时，你可以通过下述命令来删除一个 UDF 函数，可以参考 `DROP FUNCTION`。
+ public static String evaluate(String key) {
+ if (key == null) {
+ return null;
+ }
+ return res.get(key);
+ }
+ }
+ ```
-## 示例
-在`samples/doris-demo/java-udf-demo/`
目录中提供了具体示例。具体使用方法见每个目录下的`README.md`，查看点击[这里](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
+ ```java
+ public class FunctionUdf {
+ public String evaluate(String key) {
+ String value = DictLibrary.evaluate(key);
+ return value;
+ }
+ }
+ ```
## 使用须知
1. 不支持复杂数据类型（HLL，Bitmap）。
-2. 当前允许用户自己指定 JVM 最大堆大小，配置项是 be.conf 中的 JAVA_OPTS 的 -Xmx 部分。默认
1024m，如果需要聚合数据，建议调大一些，增加性能，减少内存溢出风险。
+2. 当前允许用户自己指定JVM最大堆大小，配置项是 be.conf 中的 `JAVA_OPTS` 的 -Xmx 部分。默认
1024m，如果需要聚合数据，建议调大一些，增加性能，减少内存溢出风险。
3. Char 类型的 UDF 在 create function 时需要使用 String 类型。
diff --git a/versioned_docs/version-2.0/query/udf/java-user-defined-function.md
b/versioned_docs/version-2.0/query/udf/java-user-defined-function.md
index a11fcb42f01..78618dec4e1 100644
--- a/versioned_docs/version-2.0/query/udf/java-user-defined-function.md
+++ b/versioned_docs/version-2.0/query/udf/java-user-defined-function.md
@@ -24,100 +24,141 @@ specific language governing permissions and limitations
under the License.
-->
-# Java UDF
-
-<version since="1.2.0">
-
-Java UDF
-
-</version>
-
-Java UDF provides users with a Java interface written in UDF to facilitate the
execution of user-defined functions in Java language. Compared with native UDF
implementation, Java UDF has the following advantages and limitations:
-1. The advantages
-* Compatibility: Using Java UDF can be compatible with different Doris
versions, so when upgrading Doris version, Java UDF does not need additional
migration. At the same time, Java UDF also follows the same programming
specifications as hive / spark and other engines, so that users can directly
move Hive / Spark UDF jar to Doris.
-* Security: The failure or crash of Java UDF execution will only cause the JVM
to report an error, not the Doris process to crash.
-* Flexibility: In Java UDF, users can package the third-party dependencies
together in the user jar.
+:::tip
+Java UDF is supported starting from Doris version 1.2.
+:::
-2. Restrictions on use
-* Performance: Compared with native UDF, Java UDF will bring additional JNI
overhead, but through batch execution, we have minimized the JNI overhead as
much as possible.
-* Vectorized engine: Java UDF is only supported on vectorized engine now.
+## Introduction to Java UDF
-### Type correspondence
+Java UDF provides users with a Java interface for writing UDFs, making it
convenient for users to execute custom functions using the Java language.
-|Type|UDF Argument Type|
-|----|---------|
-|Bool|Boolean|
-|TinyInt|Byte|
-|SmallInt|Short|
-|Int|Integer|
-|BigInt|Long|
-|LargeInt|BigInteger|
-|Float|Float|
-|Double|Double|
-|Date|LocalDate|
-|Datetime|LocalDateTime|
-|String|String|
-|Decimal|BigDecimal|
-|```array<Type>```|```ArrayList<Type>```|
-|```map<Type1,Type2>```|```HashMap<Type1,Type2>```|
+Doris supports writing UDFs, UDAFs, and UDTFs using JAVA. Unless otherwise
specified, UDF is used as a general term for all user-defined functions in the
following text.
-* Array/Map types can nested other types, Eg: In Doris:
```array<array<int>>``` corresponds to JAVA UDF Argument Type:
```ArrayList<ArrayList<Integer>>```, and so on.
+## Creating UDF
-:::caution Warning
-When creating a function, don't use `varchar` instead of `string`, otherwise
the function may fail to execute.
-:::
+The implemented jar package can be placed locally or stored on a remote server
for download via HTTP, but each FE and BE node must be able to access the jar
package.
-## Write UDF functions
+Otherwise, an error message `Couldn't open file ......` will be returned.
-This section mainly introduces how to develop a Java UDF. Samples for the Java
version are provided under `samples/doris-demo/java-udf-demo/` for your
reference, Check it out
[here](https://github.com/apache/incubator-doris/tree/master/samples/doris-demo/java-udf-demo)
+For more syntax help, refer to [CREATE
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Create/CREATE-FUNCTION.md).
-To use Java UDF, the main entry of UDF must be the `evaluate` function. This
is consistent with other engines such as Hive. In the example of `AddOne`, we
have completed the operation of adding an integer as the UDF.
+### UDF
-It is worth mentioning that this example is not only the Java UDF supported by
Doris, but also the UDF supported by Hive, that's to say, for users, Hive UDF
can be directly migrated to Doris.
+```sql
+CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
+ "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
+ "symbol"="org.apache.doris.udf.AddOne",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
-## Create UDF
+### UDAF
```sql
-CREATE FUNCTION
-name ([,...])
-[RETURNS] rettype
-PROPERTIES (["key"="value"][,...])
+CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
```
-Instructions:
-1. `symbol` in properties represents the class name containing UDF classes.
This parameter must be set.
-2. The jar package containing UDF represented by `file` in properties must be
set.
-3. The UDF call type represented by `type` in properties is native by default.
When using java UDF, it is transferred to `Java_UDF`.
-4. In PROPERTIES `always_nullable` indicates whether there may be a NULL value
in the UDF return result. It is an optional parameter. The default value is
true.
-5. `name`: A function belongs to a DB and name is of the
form`dbName`.`funcName`. When `dbName` is not explicitly specified, the db of
the current session is used`dbName`.
+### UDTF
+
+:::tip
+UDTF is supported starting from Doris version 3.0.
+:::
-Sample：
```sql
-CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
- "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
- "symbol"="org.apache.doris.udf.AddOne",
+CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string>
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
"always_nullable"="true",
"type"="JAVA_UDF"
);
```
-* "file"=" http://IP:port/udf -code. Jar ", you can also use http to download
jar packages in a multi machine environment.
-* The "always_nullable" is optional attribute, if there is special treatment
for the NULL value in the calculation, it is determined that the result will
not return NULL, and it can be set to false, so that the performance may be
better in the whole calculation process.
+## Using UDF
-* If you use the local path method, the jar package that the database driver
depends on, the FE and BE nodes must be placed here
-## Create UDAF
-<br/>
-When using Java code to write UDAF, there are some functions that must be
implemented (mark required) and an inner class State, which will be explained
with a specific example below.
-The following SimpleDemo will implement a simple function similar to sum, the
input parameter is INT, and the output parameter is INT
+To utilize UDFs, users must possess the `SELECT` privilege for the
corresponding database.
-```JAVA
+The usage of UDFs is identical to standard functions, with the primary
distinction being that built-in functions have a global scope, while UDFs are
scoped within the DB.
+
+When the session is linked within the database, directly using the UDF name
will search for the corresponding UDF within the current DB. Otherwise, users
must explicitly specify the UDF's database name, for example, `dbName.funcName`.
+
+## Dropping UDF
+
+If a UDF is no longer needed, it can be dropped using the following command,
as detailed in [DROP
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Drop/DROP-FUNCTION.md).
+
+## Type Correspondence
+
+| Type | UDF Argument Type |
+|-----------------------|------------------------------|
+| Bool | Boolean |
+| TinyInt | Byte |
+| SmallInt | Short |
+| Int | Integer |
+| BigInt | Long |
+| LargeInt | BigInteger |
+| Float | Float |
+| Double | Double |
+| Date | LocalDate |
+| Datetime | LocalDateTime |
+| String | String |
+| Decimal | BigDecimal |
+| `array<Type>` | `ArrayList<Type>` |
+| `map<Type1,Type2>` | `HashMap<Type1,Type2>` |
+| `struct<Type...>` | `ArrayList<Object>` (from version 3.0.0) |
+
+:::tip
+`array/map/struct` types can be nested with other types. For instance, Doris:
`array<array<int>>` corresponds to JAVA UDF Argument Type:
`ArrayList<ArrayList<Integer>>`. Other types follow the same pattern.
+:::
+
+:::caution Warning
+When creating functions, avoid using `varchar` in place of `string`, as this
may cause the function to fail.
+:::
+
+## Writing UDF
+
+This section mainly introduces how to develop a Java UDF. Examples are
provided in `samples/doris-demo/java-udf-demo/` for reference. Click
[here](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
to view them.
+
+When writing a UDF in Java, the main entry point must be the `evaluate`
function. This is consistent with other engines like Hive. In this example, we
write an `AddOne` UDF to perform an increment operation on integer inputs.
+
+It is worth mentioning that this example not only supports Java UDFs in Doris
but is also a UDF supported by Hive. This means that Hive UDFs can be directly
migrated to Doris.
+
+Additionally, if the UDF being defined needs to load large resource files or
if you want to define global static variables, you can refer to the static
variable loading method described at the bottom of the document.
+
+### UDF
+
+```java
+public class AddOne extends UDF {
+ public Integer evaluate(Integer value) {
+ return value == null ? null : value + 1;
+ }
+}
+```
+
+### UDAF
+
+When writing a UDAF using Java code, there are some required functions (marked
as required) and an inner class State that must be implemented. Below is a
specific example to illustrate.
+
+#### Example 1
+
+The following SimpleDemo will implement a simple function similar to sum, with
the input parameter being INT and the output parameter being INT.
+
+```java
package org.apache.doris.udf.demo;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
+import java.util.logging.Logger;
public class SimpleDemo {
+
+ Logger log = Logger.getLogger("SimpleDemo");
+
//Need an inner class to store data
/*required*/
public static class State {
@@ -192,16 +233,9 @@ public class SimpleDemo {
```
-```sql
-CREATE AGGREGATE FUNCTION simple_sum(INT) RETURNS INT PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.SimpleDemo",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+#### Example 2
-```JAVA
+```java
package org.apache.doris.udf.demo;
@@ -211,30 +245,30 @@ import java.math.BigDecimal;
import java.util.Arrays;
import java.util.logging.Logger;
-/*UDAF for calculating the median*/
+/*UDAF 计算中位数*/
public class MedianUDAF {
Logger log = Logger.getLogger("MedianUDAF");
- // State storage
+ //状态存储
public static class State {
- // Precision of the result
+ //返回结果的精度
int scale = 0;
- // Whether this is the first time to execute add() for the data under
a certain aggregation condition of a certain tablet
+ //是否是某一个 tablet 下的某个聚合条件下的数据第一次执行 add 方法
boolean isFirst = true;
- //Data storage
+ //数据存储
public StringBuilder stringBuilder;
}
- //State initialization
+ //状态初始化
public State create() {
State state = new State();
- //Pre-initialize based on the amount of data to be aggregated for each
aggregation condition under each tablet, for improved performance
+ //根据每个 tablet 下的聚合条件需要聚合的数据量大小，预先初始化，增加性能
state.stringBuilder = new StringBuilder(1000);
return state;
}
- // Handle the data for each unit under each aggregation condition for each
tablet
+ //处理执行单位处理各自 tablet 下的各自聚合条件下的每个数据
public void add(State state, Double val, int scale) {
try {
if (val != null && state.isFirst) {
@@ -244,24 +278,24 @@ public class MedianUDAF {
state.stringBuilder.append(val).append(",");
}
} catch (Exception e) {
- // If it is not guaranteed that there will be no exceptions, it is
recommended to maximize the exception capture for each method, as the
processing of java-thrown exceptions is currently not supported
- log.info("Exception encountered while retrieving data: " +
e.getMessage());
+ //如果不能保证一定不会异常，建议每个方法都最大化捕获异常，因为目前不支持处理 java 抛出的异常
+ log.info("获取数据异常：" + e.getMessage());
}
}
- // Output the data after processing for aggregation
+ //处理数据完需要输出等待聚合
public void serialize(State state, DataOutputStream out) {
try {
- // Only DataOutputStream is currently provided, if object
serialization is required, consider methods such as concatenating strings,
converting to json, serializing to byte arrays, etc.
- // If you want to serialize the State object, you may need to
implement the serialization interface for the inner class State yourself
- // In the end, it will be transmitted through DataOutputStream
+ //目前暂时只提供 DataOutputStream，如果需要序列化对象可以考虑拼接字符串，转换 json，序列化成字节数组等方式
+ //如果要序列化 State 对象，可能需要自己将 State 内部类实现序列化接口
+ //最终都是要通过 DataOutputStream 传输
out.writeUTF(state.stringBuilder.toString());
} catch (Exception e) {
- log.info("Exception encountered while serializing data:" +
e.getMessage());
+ log.info("序列化异常：" + e.getMessage());
}
}
- // Retrieve the data output by each data processing unit
+ //获取处理数据执行单位输出的数据
public void deserialize(State state, DataInputStream in) {
try {
String string = in.readUTF();
@@ -269,24 +303,24 @@ public class MedianUDAF {
StringBuilder stringBuilder = new
StringBuilder(string.substring(2));
state.stringBuilder = stringBuilder;
} catch (Exception e) {
- log.info("Exception encountered while deserializing data: " +
e.getMessage());
+ log.info("反序列化异常：" + e.getMessage());
}
}
- // Merge the processing results of data under a certain key according to
the aggregation condition, where state1 is the initialized instance for the
first merge of each key
+ //聚合执行单位按照聚合条件合并某一个键下数据的处理结果 ,每个键第一次合并时，state1 参数是初始化的实例
public void merge(State state1, State state2) {
try {
state1.scale = state2.scale;
state1.stringBuilder.append(state2.stringBuilder.toString());
} catch (Exception e) {
- log.info("Exception encountered while merging results: " +
e.getMessage());
+ log.info("合并结果异常：" + e.getMessage());
}
}
- // Aggregate the data for each key after merging and output the final
result
+ //对每个键合并后的数据进行并输出最终结果
public Double getValue(State state) {
try {
- String[] strings = state.stringBuilder.toString( ).split(",");
+ String[] strings = state.stringBuilder.toString().split(",");
double[] doubles = new double[strings.length + 1];
doubles =
Arrays.stream(strings).mapToDouble(Double::parseDouble).toArray();
@@ -302,12 +336,12 @@ public class MedianUDAF {
BigDecimal decimal = new BigDecimal(value);
return decimal.setScale(state.scale,
BigDecimal.ROUND_HALF_UP).doubleValue();
} catch (Exception e) {
- log.info("Exception encountered while calculating result：" +
e.getMessage());
+ log.info("计算异常：" + e.getMessage());
}
return 0.0;
}
- //This method is executed after each processing unit is completed
+ //每个执行单位执行完都会执行
public void destroy(State state) {
}
@@ -315,39 +349,90 @@ public class MedianUDAF {
```
-```sql
-CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
+### UDTF
+
+Similar to UDFs, UDTFs require users to implement an `evaluate` method.
However, the return value of a UDTF must be of the Array type.
+
+Additionally, table functions in Doris may exhibit different behaviors due to
the `_outer` suffix. For more details, refer to [OUTER
combinator](../sql-manual/sql-functions/table-functions/explode-numbers-outer.md).
+
+```JAVA
+public class UDTFStringTest {
+ public ArrayList<String> evaluate(String value, String separator) {
+ if (value == null || separator == null) {
+ return null;
+ } else {
+ return new ArrayList<>(Arrays.asList(value.split(separator)));
+ }
+ }
+}
```
+## Best Practices
+
+## Loading static variables
+
+Currently, in Doris, executing a UDF function, e.g., `select udf(col) from
table`, will load the udf.jar package for each concurrent instance, and unload
the udf.jar package when the instance ends. If the udf.jar file needs to load a
file of several hundred MBs, the memory usage will increase sharply due to
concurrency, potentially leading to OOM (Out of Memory).
+
+The solution is to split the resource loading code, generate a separate jar
package, and have other packages directly reference this resource jar package.
-* The implemented jar package can be stored at local or in a remote server and
downloaded via http, And each BE node must be able to obtain the jar package;
-Otherwise, the error status message "Couldn't open file..." will be returned
+Assume the files have been split into DictLibrary and FunctionUdf.
-Currently, UDTF are not supported.
+1. Compile the DictLibrary file separately to generate an independent jar
package, resulting in a resource file DictLibrary.jar:
-<br/>
+ ```shell
+ javac ./DictLibrary.java
+ jar -cf ./DictLibrary.jar ./DictLibrary.class
+ ```
-## Use UDF
+2. Then compile the FunctionUdf file, directly referencing the resource
package from the previous step, resulting in the FunctionUdf.jar package:
-Users must have the `SELECT` permission of the corresponding database to use
UDF/UDAF.
+ ```shell
+ javac -cp ./DictLibrary.jar ./FunctionUdf.java
+ jar -cvf ./FunctionUdf.jar ./FunctionUdf.class
+ ```
-The use of UDF is consistent with ordinary function methods. The only
difference is that the scope of built-in functions is global, and the scope of
UDF is internal to DB. When the link session is inside the data, directly using
the UDF name will find the corresponding UDF inside the current DB. Otherwise,
the user needs to display the specified UDF database name, such as
`dbName`.`funcName`.
+3. After the above two steps, you will get two jar packages. To allow the
resource jar package to be referenced by all concurrent instances, place it in
the BE deployment path `be/lib/java_extensions/java-udf`. After restarting BE,
it will be loaded with the JVM startup.
+
+4. Finally, use the `create function` statement to create a UDF function, with
the file path pointing to the FunctionUdf.jar package. This way, the resource
package will be loaded and released with the BE startup and shutdown. The
FunctionUdf.jar will be loaded and released with the SQL execution cycle.
+
+ ```java
+ public class DictLibrary {
+ private static HashMap<String, String> res = new HashMap<>();
+
+ static {
+ // suppose we built this dictionary from a certain local file.
+ res.put("key1", "value1");
+ res.put("key2", "value2");
+ res.put("key3", "value3");
+ res.put("0", "value4");
+ res.put("1", "value5");
+ res.put("2", "value6");
+ }
+
+ public static String evaluate(String key) {
+ if (key == null) {
+ return null;
+ }
+ return res.get(key);
+ }
+ }
+ ```
+
+ ```java
+ public class FunctionUdf {
+ public String evaluate(String key) {
+ String value = DictLibrary.evaluate(key);
+ return value;
+ }
+ }
+ ```
-## Delete UDF
+## Usage Notes
-When you no longer need UDF functions, you can delete a UDF function by the
following command, you can refer to `DROP FUNCTION`.
+1. Complex data types (HLL, Bitmap) are not supported.
-## Example
-Examples of Java UDF are provided in the `samples/doris-demo/java-udf-demo/`
directory. See the `README.md` in each directory for details on how to use it,
Check it out
[here](https://github.com/apache/incubator-doris/tree/master/samples/doris-demo/java-udf-demo)
+2. Users are currently allowed to specify the maximum JVM heap size. The
configuration item is the `-Xmx` part of `JAVA_OPTS` in `be.conf`. The default
is 1024m. If you need to aggregate data, it is recommended to increase this
value to enhance performance and reduce the risk of memory overflow.
-## Instructions
-1. Complex data types (HLL, bitmap) are not supported.
-2. Currently, users are allowed to specify the maximum heap size of the JVM
themselves. The configuration item is jvm_ max_ heap_ size. The configuration
item is in the global configuration file 'be.conf' under the installation
directory of the BE. The default value is 512M. If data aggregation is
required, it is recommended to increase the value to improve performance and
reduce the risk of memory overflow.
-3. The udf of char type needs to use the String type when creating a function.
-4. Due to the problem that the jvm loads classes with the same name, do not
use multiple classes with the same name as udf implementations at the same
time. If you want to update the udf of a class with the same name, you need to
restart be to reload the classpath.
+3. For Char type UDFs, use the String type when creating the function.
+4. Due to issues with JVM loading classes with the same name, do not use
multiple classes with the same name as UDF implementations simultaneously. If
you want to update a UDF with a class of the same name, you need to restart BE
to reload the classpath.
diff --git a/versioned_docs/version-2.1/query/udf/java-user-defined-function.md
b/versioned_docs/version-2.1/query/udf/java-user-defined-function.md
index 88565bcf985..78618dec4e1 100644
--- a/versioned_docs/version-2.1/query/udf/java-user-defined-function.md
+++ b/versioned_docs/version-2.1/query/udf/java-user-defined-function.md
@@ -24,76 +24,114 @@ specific language governing permissions and limitations
under the License.
-->
-# Java UDF
-
-<version since="1.2.0">
+:::tip
+Java UDF is supported starting from Doris version 1.2.
+:::
-Java UDF
+## Introduction to Java UDF
-</version>
+Java UDF provides users with a Java interface for writing UDFs, making it
convenient for users to execute custom functions using the Java language.
-Java UDF provides users with a Java interface written in UDF to facilitate the
execution of user-defined functions in Java language. Compared with native UDF
implementation, Java UDF has the following advantages and limitations:
-1. The advantages
-* Compatibility: Using Java UDF can be compatible with different Doris
versions, so when upgrading Doris version, Java UDF does not need additional
migration. At the same time, Java UDF also follows the same programming
specifications as hive / spark and other engines, so that users can directly
move Hive / Spark UDF jar to Doris.
-* Security: The failure or crash of Java UDF execution will only cause the JVM
to report an error, not the Doris process to crash.
-* Flexibility: In Java UDF, users can package the third-party dependencies
together in the user jar.
+Doris supports writing UDFs, UDAFs, and UDTFs using JAVA. Unless otherwise
specified, UDF is used as a general term for all user-defined functions in the
following text.
-2. Restrictions on use
-* Performance: Compared with native UDF, Java UDF will bring additional JNI
overhead, but through batch execution, we have minimized the JNI overhead as
much as possible.
-* Vectorized engine: Java UDF is only supported on vectorized engine now.
+## Creating UDF
-### Type correspondence
+The implemented jar package can be placed locally or stored on a remote server
for download via HTTP, but each FE and BE node must be able to access the jar
package.
-|Type|UDF Argument Type|
-|----|---------|
-|Bool|Boolean|
-|TinyInt|Byte|
-|SmallInt|Short|
-|Int|Integer|
-|BigInt|Long|
-|LargeInt|BigInteger|
-|Float|Float|
-|Double|Double|
-|Date|LocalDate|
-|Datetime|LocalDateTime|
-|String|String|
-|Decimal|BigDecimal|
-|```array<Type>```|```ArrayList<Type>```|
-|```map<Type1,Type2>```|```HashMap<Type1,Type2>```|
+Otherwise, an error message `Couldn't open file ......` will be returned.
-* Array/Map types can nested other types, Eg: In Doris:
```array<array<int>>``` corresponds to JAVA UDF Argument Type:
```ArrayList<ArrayList<Integer>>```, and so on.
+For more syntax help, refer to [CREATE
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Create/CREATE-FUNCTION.md).
-:::caution Warning
-When creating a function, don't use `varchar` instead of `string`, otherwise
the function may fail to execute.
-:::
+### UDF
-## Write UDF functions
+```sql
+CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
+ "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
+ "symbol"="org.apache.doris.udf.AddOne",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
-This section mainly introduces how to develop a Java UDF. Samples for the Java
version are provided under `samples/doris-demo/java-udf-demo/` for your
reference, Check it out
[here](https://github.com/apache/incubator-doris/tree/master/samples/doris-demo/java-udf-demo)
+### UDAF
-To use Java UDF, the main entry of UDF must be the `evaluate` function. This
is consistent with other engines such as Hive. In the example of `AddOne`, we
have completed the operation of adding an integer as the UDF.
+```sql
+CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
+```
-It is worth mentioning that this example is not only the Java UDF supported by
Doris, but also the UDF supported by Hive, that's to say, for users, Hive UDF
can be directly migrated to Doris.
+### UDTF
-## Create UDF
+:::tip
+UDTF is supported starting from Doris version 3.0.
+:::
```sql
-CREATE FUNCTION
-name ([,...])
-[RETURNS] rettype
-PROPERTIES (["key"="value"][,...])
+CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string>
PROPERTIES (
+ "file"="file:///pathTo/java-udaf.jar",
+ "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
+ "always_nullable"="true",
+ "type"="JAVA_UDF"
+);
```
-Instructions:
-1. `symbol` in properties represents the class name containing UDF classes.
This parameter must be set.
-2. The jar package containing UDF represented by `file` in properties must be
set.
-3. The UDF call type represented by `type` in properties is native by default.
When using java UDF, it is transferred to `Java_UDF`.
-4. In PROPERTIES `always_nullable` indicates whether there may be a NULL value
in the UDF return result. It is an optional parameter. The default value is
true.
-5. `name`: A function belongs to a DB and name is of the
form`dbName`.`funcName`. When `dbName` is not explicitly specified, the db of
the current session is used`dbName`.
+## Using UDF
-Sample：
+To utilize UDFs, users must possess the `SELECT` privilege for the
corresponding database.
-```JAVA
+The usage of UDFs is identical to standard functions, with the primary
distinction being that built-in functions have a global scope, while UDFs are
scoped within the DB.
+
+When the session is linked within the database, directly using the UDF name
will search for the corresponding UDF within the current DB. Otherwise, users
must explicitly specify the UDF's database name, for example, `dbName.funcName`.
+
+## Dropping UDF
+
+If a UDF is no longer needed, it can be dropped using the following command,
as detailed in [DROP
FUNCTION](../sql-manual/sql-statements/Data-Definition-Statements/Drop/DROP-FUNCTION.md).
+
+## Type Correspondence
+
+| Type | UDF Argument Type |
+|-----------------------|------------------------------|
+| Bool | Boolean |
+| TinyInt | Byte |
+| SmallInt | Short |
+| Int | Integer |
+| BigInt | Long |
+| LargeInt | BigInteger |
+| Float | Float |
+| Double | Double |
+| Date | LocalDate |
+| Datetime | LocalDateTime |
+| String | String |
+| Decimal | BigDecimal |
+| `array<Type>` | `ArrayList<Type>` |
+| `map<Type1,Type2>` | `HashMap<Type1,Type2>` |
+| `struct<Type...>` | `ArrayList<Object>` (from version 3.0.0) |
+
+:::tip
+`array/map/struct` types can be nested with other types. For instance, Doris:
`array<array<int>>` corresponds to JAVA UDF Argument Type:
`ArrayList<ArrayList<Integer>>`. Other types follow the same pattern.
+:::
+
+:::caution Warning
+When creating functions, avoid using `varchar` in place of `string`, as this
may cause the function to fail.
+:::
+
+## Writing UDF
+
+This section mainly introduces how to develop a Java UDF. Examples are
provided in `samples/doris-demo/java-udf-demo/` for reference. Click
[here](https://github.com/apache/doris/tree/master/samples/doris-demo/java-udf-demo)
to view them.
+
+When writing a UDF in Java, the main entry point must be the `evaluate`
function. This is consistent with other engines like Hive. In this example, we
write an `AddOne` UDF to perform an increment operation on integer inputs.
+
+It is worth mentioning that this example not only supports Java UDFs in Doris
but is also a UDF supported by Hive. This means that Hive UDFs can be directly
migrated to Doris.
+
+Additionally, if the UDF being defined needs to load large resource files or
if you want to define global static variables, you can refer to the static
variable loading method described at the bottom of the document.
+
+### UDF
+
+```java
public class AddOne extends UDF {
public Integer evaluate(Integer value) {
return value == null ? null : value + 1;
@@ -101,32 +139,26 @@ public class AddOne extends UDF {
}
```
-```sql
-CREATE FUNCTION java_udf_add_one(int) RETURNS int PROPERTIES (
- "file"="file:///path/to/java-udf-demo-jar-with-dependencies.jar",
- "symbol"="org.apache.doris.udf.AddOne",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
-* "file"=" http://IP:port/udf -code. Jar ", you can also use http to download
jar packages in a multi machine environment.
+### UDAF
-* The "always_nullable" is optional attribute, if there is special treatment
for the NULL value in the calculation, it is determined that the result will
not return NULL, and it can be set to false, so that the performance may be
better in the whole calculation process.
+When writing a UDAF using Java code, there are some required functions (marked
as required) and an inner class State that must be implemented. Below is a
specific example to illustrate.
-* If you use the local path method, the jar package that the database driver
depends on, the FE and BE nodes must be placed here
-## Create UDAF
-<br/>
-When using Java code to write UDAF, there are some functions that must be
implemented (mark required) and an inner class State, which will be explained
with a specific example below.
-The following SimpleDemo will implement a simple function similar to sum, the
input parameter is INT, and the output parameter is INT
+#### Example 1
-```JAVA
+The following SimpleDemo will implement a simple function similar to sum, with
the input parameter being INT and the output parameter being INT.
+
+```java
package org.apache.doris.udf.demo;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
+import java.util.logging.Logger;
public class SimpleDemo {
+
+ Logger log = Logger.getLogger("SimpleDemo");
+
//Need an inner class to store data
/*required*/
public static class State {
@@ -201,16 +233,9 @@ public class SimpleDemo {
```
-```sql
-CREATE AGGREGATE FUNCTION simple_sum(INT) RETURNS INT PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.SimpleDemo",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
-```
+#### Example 2
-```JAVA
+```java
package org.apache.doris.udf.demo;
@@ -220,30 +245,30 @@ import java.math.BigDecimal;
import java.util.Arrays;
import java.util.logging.Logger;
-/*UDAF for calculating the median*/
+/*UDAF 计算中位数*/
public class MedianUDAF {
Logger log = Logger.getLogger("MedianUDAF");
- // State storage
+ //状态存储
public static class State {
- // Precision of the result
+ //返回结果的精度
int scale = 0;
- // Whether this is the first time to execute add() for the data under
a certain aggregation condition of a certain tablet
+ //是否是某一个 tablet 下的某个聚合条件下的数据第一次执行 add 方法
boolean isFirst = true;
- //Data storage
+ //数据存储
public StringBuilder stringBuilder;
}
- //State initialization
+ //状态初始化
public State create() {
State state = new State();
- //Pre-initialize based on the amount of data to be aggregated for each
aggregation condition under each tablet, for improved performance
+ //根据每个 tablet 下的聚合条件需要聚合的数据量大小，预先初始化，增加性能
state.stringBuilder = new StringBuilder(1000);
return state;
}
- // Handle the data for each unit under each aggregation condition for each
tablet
+ //处理执行单位处理各自 tablet 下的各自聚合条件下的每个数据
public void add(State state, Double val, int scale) {
try {
if (val != null && state.isFirst) {
@@ -253,24 +278,24 @@ public class MedianUDAF {
state.stringBuilder.append(val).append(",");
}
} catch (Exception e) {
- // If it is not guaranteed that there will be no exceptions, it is
recommended to maximize the exception capture for each method, as the
processing of java-thrown exceptions is currently not supported
- log.info("Exception encountered while retrieving data: " +
e.getMessage());
+ //如果不能保证一定不会异常，建议每个方法都最大化捕获异常，因为目前不支持处理 java 抛出的异常
+ log.info("获取数据异常：" + e.getMessage());
}
}
- // Output the data after processing for aggregation
+ //处理数据完需要输出等待聚合
public void serialize(State state, DataOutputStream out) {
try {
- // Only DataOutputStream is currently provided, if object
serialization is required, consider methods such as concatenating strings,
converting to json, serializing to byte arrays, etc.
- // If you want to serialize the State object, you may need to
implement the serialization interface for the inner class State yourself
- // In the end, it will be transmitted through DataOutputStream
+ //目前暂时只提供 DataOutputStream，如果需要序列化对象可以考虑拼接字符串，转换 json，序列化成字节数组等方式
+ //如果要序列化 State 对象，可能需要自己将 State 内部类实现序列化接口
+ //最终都是要通过 DataOutputStream 传输
out.writeUTF(state.stringBuilder.toString());
} catch (Exception e) {
- log.info("Exception encountered while serializing data:" +
e.getMessage());
+ log.info("序列化异常：" + e.getMessage());
}
}
- // Retrieve the data output by each data processing unit
+ //获取处理数据执行单位输出的数据
public void deserialize(State state, DataInputStream in) {
try {
String string = in.readUTF();
@@ -278,24 +303,24 @@ public class MedianUDAF {
StringBuilder stringBuilder = new
StringBuilder(string.substring(2));
state.stringBuilder = stringBuilder;
} catch (Exception e) {
- log.info("Exception encountered while deserializing data: " +
e.getMessage());
+ log.info("反序列化异常：" + e.getMessage());
}
}
- // Merge the processing results of data under a certain key according to
the aggregation condition, where state1 is the initialized instance for the
first merge of each key
+ //聚合执行单位按照聚合条件合并某一个键下数据的处理结果 ,每个键第一次合并时，state1 参数是初始化的实例
public void merge(State state1, State state2) {
try {
state1.scale = state2.scale;
state1.stringBuilder.append(state2.stringBuilder.toString());
} catch (Exception e) {
- log.info("Exception encountered while merging results: " +
e.getMessage());
+ log.info("合并结果异常：" + e.getMessage());
}
}
- // Aggregate the data for each key after merging and output the final
result
+ //对每个键合并后的数据进行并输出最终结果
public Double getValue(State state) {
try {
- String[] strings = state.stringBuilder.toString( ).split(",");
+ String[] strings = state.stringBuilder.toString().split(",");
double[] doubles = new double[strings.length + 1];
doubles =
Arrays.stream(strings).mapToDouble(Double::parseDouble).toArray();
@@ -311,12 +336,12 @@ public class MedianUDAF {
BigDecimal decimal = new BigDecimal(value);
return decimal.setScale(state.scale,
BigDecimal.ROUND_HALF_UP).doubleValue();
} catch (Exception e) {
- log.info("Exception encountered while calculating result：" +
e.getMessage());
+ log.info("计算异常：" + e.getMessage());
}
return 0.0;
}
- //This method is executed after each processing unit is completed
+ //每个执行单位执行完都会执行
public void destroy(State state) {
}
@@ -324,37 +349,90 @@ public class MedianUDAF {
```
-```sql
-CREATE AGGREGATE FUNCTION middle_quantiles(DOUBLE,INT) RETURNS DOUBLE
PROPERTIES (
- "file"="file:///pathTo/java-udaf.jar",
- "symbol"="org.apache.doris.udf.demo.MiddleNumberUDAF",
- "always_nullable"="true",
- "type"="JAVA_UDF"
-);
+### UDTF
+
+Similar to UDFs, UDTFs require users to implement an `evaluate` method.
However, the return value of a UDTF must be of the Array type.
+
+Additionally, table functions in Doris may exhibit different behaviors due to
the `_outer` suffix. For more details, refer to [OUTER
combinator](../sql-manual/sql-functions/table-functions/explode-numbers-outer.md).
+
+```JAVA
+public class UDTFStringTest {
+ public ArrayList<String> evaluate(String value, String separator) {
+ if (value == null || separator == null) {
+ return null;
+ } else {
+ return new ArrayList<>(Arrays.asList(value.split(separator)));
+ }
+ }
+}
```
+## Best Practices
+
+## Loading static variables
+
+Currently, in Doris, executing a UDF function, e.g., `select udf(col) from
table`, will load the udf.jar package for each concurrent instance, and unload
the udf.jar package when the instance ends. If the udf.jar file needs to load a
file of several hundred MBs, the memory usage will increase sharply due to
concurrency, potentially leading to OOM (Out of Memory).
+
+The solution is to split the resource loading code, generate a separate jar
package, and have other packages directly reference this resource jar package.
+
+Assume the files have been split into DictLibrary and FunctionUdf.
-* The implemented jar package can be stored at local or in a remote server and
downloaded via http, And each BE node must be able to obtain the jar package;
-Otherwise, the error status message "Couldn't open file..." will be returned
+1. Compile the DictLibrary file separately to generate an independent jar
package, resulting in a resource file DictLibrary.jar:
-Currently, UDTF are not supported.
+ ```shell
+ javac ./DictLibrary.java
+ jar -cf ./DictLibrary.jar ./DictLibrary.class
+ ```
-## Use UDF
+2. Then compile the FunctionUdf file, directly referencing the resource
package from the previous step, resulting in the FunctionUdf.jar package:
-Users must have the `SELECT` permission of the corresponding database to use
UDF/UDAF.
+ ```shell
+ javac -cp ./DictLibrary.jar ./FunctionUdf.java
+ jar -cvf ./FunctionUdf.jar ./FunctionUdf.class
+ ```
-The use of UDF is consistent with ordinary function methods. The only
difference is that the scope of built-in functions is global, and the scope of
UDF is internal to DB. When the link session is inside the data, directly using
the UDF name will find the corresponding UDF inside the current DB. Otherwise,
the user needs to display the specified UDF database name, such as
`dbName`.`funcName`.
+3. After the above two steps, you will get two jar packages. To allow the
resource jar package to be referenced by all concurrent instances, place it in
the BE deployment path `be/lib/java_extensions/java-udf`. After restarting BE,
it will be loaded with the JVM startup.
+
+4. Finally, use the `create function` statement to create a UDF function, with
the file path pointing to the FunctionUdf.jar package. This way, the resource
package will be loaded and released with the BE startup and shutdown. The
FunctionUdf.jar will be loaded and released with the SQL execution cycle.
+
+ ```java
+ public class DictLibrary {
+ private static HashMap<String, String> res = new HashMap<>();
+
+ static {
+ // suppose we built this dictionary from a certain local file.
+ res.put("key1", "value1");
+ res.put("key2", "value2");
+ res.put("key3", "value3");
+ res.put("0", "value4");
+ res.put("1", "value5");
+ res.put("2", "value6");
+ }
+
+ public static String evaluate(String key) {
+ if (key == null) {
+ return null;
+ }
+ return res.get(key);
+ }
+ }
+ ```
+
+ ```java
+ public class FunctionUdf {
+ public String evaluate(String key) {
+ String value = DictLibrary.evaluate(key);
+ return value;
+ }
+ }
+ ```
-## Delete UDF
+## Usage Notes
-When you no longer need UDF functions, you can delete a UDF function by the
following command, you can refer to `DROP FUNCTION`.
+1. Complex data types (HLL, Bitmap) are not supported.
-## Example
-Examples of Java UDF are provided in the `samples/doris-demo/java-udf-demo/`
directory. See the `README.md` in each directory for details on how to use it,
Check it out
[here](https://github.com/apache/incubator-doris/tree/master/samples/doris-demo/java-udf-demo)
+2. Users are currently allowed to specify the maximum JVM heap size. The
configuration item is the `-Xmx` part of `JAVA_OPTS` in `be.conf`. The default
is 1024m. If you need to aggregate data, it is recommended to increase this
value to enhance performance and reduce the risk of memory overflow.
-## Instructions
-1. Complex data types (HLL, bitmap) are not supported.
-2. Currently, users are allowed to specify the maximum heap size of the JVM
themselves. The configuration item is jvm_ max_ heap_ size. The configuration
item is in the global configuration file 'be.conf' under the installation
directory of the BE. The default value is 1024M. If data aggregation is
required, it is recommended to increase the value to improve performance and
reduce the risk of memory overflow.
-3. The udf of char type needs to use the String type when creating a function.
-4. Due to the problem that the jvm loads classes with the same name, do not
use multiple classes with the same name as udf implementations at the same
time. If you want to update the udf of a class with the same name, you need to
restart be to reload the classpath.
+3. For Char type UDFs, use the String type when creating the function.
+4. Due to issues with JVM loading classes with the same name, do not use
multiple classes with the same name as UDF implementations simultaneously. If
you want to update a UDF with a class of the same name, you need to restart BE
to reload the classpath.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
