[
https://issues.apache.org/jira/browse/FLINK-40293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiqing Yang updated FLINK-40293:
---------------------------------
Description:
This is the second of four sub-tasks implementing FLIP-485.
FLIP:
https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics
Vote thread: https://lists.apache.org/thread/symqpswsohl2s5wmtkcw0jjp1w5dot0n
Add the two configuration options and the first end-to-end slice of the
feature: synchronous scalar and table UDF calls instrumented at code generation.
Configuration, both {{@PublicEvolving}} additions to ExecutionConfigOptions:
* {{table.exec.udf-metric-enabled}}, default false, enables the feature.
* {{table.exec.udf-metric.sample-interval}}, default 100, measures every Nth
invocation.
Instrumentation:
* Wrap the generated {{eval}} call site for synchronous scalar and table UDFs
in BridgingFunctionGenUtil with sampled timing and exception counting via the
UdfMetrics helper.
* Cache one handle per (operator, udf name) in CodeGeneratorContext, so
repeated call sites of the same function within one operator share a handle.
Without sharing, the second registration is dropped and only one call site's
timings survive.
* Meter only user functions on the modern BridgingSqlFunction stack, through an
opt-in name passed from BridgingSqlFunctionCallGen. Lookup-join, ML-predict,
vector-search, legacy CallGens, and PROCESS_TABLE functions are not metered.
The instrumentation is emitted at code generation only when the feature is
enabled, so the generated operator is byte-identical to today when it is
disabled. When enabled, the non-sampled path costs a single integer increment.
Exceptions are counted on every invocation and are not sampled.
Covered by a new UdfMetricsITCase using InMemoryReporter: sync scalar and table
functions, metric naming and scope, exception counting, the enabled/disabled
gate, a lookup join not being metered, handle sharing across repeated call
sites versus separate handles for distinct functions, and udfProcessingTime
reflecting a real induced delay.
was:
This is the second of four sub-tasks implementing FLIP-485.
FLIP:
https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics
Vote thread: https://lists.apache.org/thread/symqpswsohl2s5wmtkcw0jjp1w5dot0n
Add the two configuration options and the first end-to-end slice of the feature:
synchronous scalar and table UDF calls instrumented at code generation.
Configuration, both {{@PublicEvolving}} additions to ExecutionConfigOptions:
* {{table.exec.udf-metric-enabled}}, default false, enables the feature.
* {{table.exec.udf-metric.sample-interval}}, default 100, measures every Nth
invocation.
Instrumentation:
* Wrap the generated {{eval}} call site for synchronous scalar and table UDFs in
BridgingFunctionGenUtil with sampled timing and exception counting via the
UdfMetrics
helper.
* Cache one handle per (operator, udf name) in CodeGeneratorContext, so
repeated call
sites of the same function within one operator share a handle. Without
sharing, the
second registration is dropped and only one call site's timings survive.
* Meter only user functions on the modern BridgingSqlFunction stack, through an
opt-in
name passed from BridgingSqlFunctionCallGen. Lookup-join, ML-predict,
vector-search,
legacy CallGens, and PROCESS_TABLE functions are not metered.
The instrumentation is emitted at code generation only when the feature is
enabled, so
the generated operator is byte-identical to today when it is disabled. When
enabled, the
non-sampled path costs a single integer increment. Exceptions are counted on
every
invocation and are not sampled.
Covered by a new UdfMetricsITCase using InMemoryReporter: sync scalar and table
functions, metric naming and scope, exception counting, the enabled/disabled
gate, a
lookup join not being metered, handle sharing across repeated call sites versus
separate
handles for distinct functions, and udfProcessingTime reflecting a real induced
delay.
> Add UDF metric config options and instrument sync scalar and table UDF calls
> ----------------------------------------------------------------------------
>
> Key: FLINK-40293
> URL: https://issues.apache.org/jira/browse/FLINK-40293
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Planner
> Reporter: Weiqing Yang
> Priority: Major
>
> This is the second of four sub-tasks implementing FLIP-485.
> FLIP:
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics
> Vote thread: https://lists.apache.org/thread/symqpswsohl2s5wmtkcw0jjp1w5dot0n
> Add the two configuration options and the first end-to-end slice of the
> feature: synchronous scalar and table UDF calls instrumented at code
> generation.
> Configuration, both {{@PublicEvolving}} additions to ExecutionConfigOptions:
> * {{table.exec.udf-metric-enabled}}, default false, enables the feature.
> * {{table.exec.udf-metric.sample-interval}}, default 100, measures every Nth
> invocation.
> Instrumentation:
> * Wrap the generated {{eval}} call site for synchronous scalar and table UDFs
> in BridgingFunctionGenUtil with sampled timing and exception counting via the
> UdfMetrics helper.
> * Cache one handle per (operator, udf name) in CodeGeneratorContext, so
> repeated call sites of the same function within one operator share a handle.
> Without sharing, the second registration is dropped and only one call site's
> timings survive.
> * Meter only user functions on the modern BridgingSqlFunction stack, through
> an opt-in name passed from BridgingSqlFunctionCallGen. Lookup-join,
> ML-predict, vector-search, legacy CallGens, and PROCESS_TABLE functions are
> not metered.
> The instrumentation is emitted at code generation only when the feature is
> enabled, so the generated operator is byte-identical to today when it is
> disabled. When enabled, the non-sampled path costs a single integer
> increment. Exceptions are counted on every invocation and are not sampled.
> Covered by a new UdfMetricsITCase using InMemoryReporter: sync scalar and
> table functions, metric naming and scope, exception counting, the
> enabled/disabled gate, a lookup join not being metered, handle sharing across
> repeated call sites versus separate handles for distinct functions, and
> udfProcessingTime reflecting a real induced delay.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
