[
https://issues.apache.org/jira/browse/FLINK-40471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126553#comment-18126553
]
ChenZhuo commented on FLINK-40471:
----------------------------------
Thanks for the clarification, Dian Fu. The implementation is ready for review:
https://github.com/apache/flink/pull/29470
It supports concurrency for general, async, pandas and Arrow scalar UDFs, and
batch_size for pandas/Arrow UDFs. Unsupported batch_size settings raise an
exception; omitted options preserve existing behavior.
I do not have permission to edit issue fields. Could a committer add the
following Release Notes?
Python DataFrame udf() now accepts per-UDF concurrency to configure Python
stage parallelism and batch_size to configure pandas/Arrow batch sizes.
Omitting these options preserves existing defaults.
> Add per-UDF concurrency and batch size support to DataFrame API
> ---------------------------------------------------------------
>
> Key: FLINK-40471
> URL: https://issues.apache.org/jira/browse/FLINK-40471
> Project: Flink
> Issue Type: Sub-task
> Reporter: Liu Liu
> Assignee: ChenZhuo
> Priority: Major
> Labels: pull-request-available
>
> Support specifying per-UDF concurrency and batch size in UDF creation:
> {code:java}
> def udf(
> func=None,
> *,
> return_dtype: DataType = None,
> deterministic: bool = True,
> name: str = None,
> concurrency: Optional[int] = None, // per-UDF concurrency
> batch_size: Optional[int] = None, // per-UDF batch size
> ) -> DataFrameUDFWrapper {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
