[
https://issues.apache.org/jira/browse/SPARK-59986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alyssa Gorbaneva updated SPARK-59986:
-------------------------------------
Description:
h2. Motivation
StreamingQueryProgress JSON reports query identifiers, batch timings, source,
sink, state operator, and observed metrics, but it does not report the Trigger
selected for the query. Consumers of progress events cannot determine whether a
query uses ProcessingTime, Continuous, RealTime, AvailableNow, or OneTime from
the progress payload.
h2. Proposed change
Add an optional top-level trigger object to StreamingQueryProgress JSON and
preserve it when QueryProgressEvent JSON and status-store records are
serialized and deserialized.
The object contains the selected trigger type and any relevant configuration:
{code:json}
"trigger": {"type": "ProcessingTime", "intervalMs": 1000}
"trigger": {"type": "RealTime", "batchDurationMs": 300000}
"trigger": {"type": "AvailableNow"}
{code}
The type field is the discriminator for the remaining fields. The configuration
names match the existing Trigger APIs and execution semantics. ProcessingTime
uses intervalMs to schedule batch starts, and Continuous uses intervalMs to
schedule checkpoints. RealTime uses batchDurationMs as each real-time scan's
deadline; because consecutive real-time batches start immediately, this
duration also influences batch cadence. AvailableNow and OneTime do not have a
duration field. Consumers should read type first and then the configuration
field defined for that trigger type.
The field should:
* support all built-in trigger types;
* provide stable, queryable type and configuration fields;
* remain backward-compatible with progress and event records that do not
contain it.
was:
h2. Motivation
StreamingQueryProgress JSON reports query identifiers, batch timings, source,
sink, state operator, and observed metrics, but it does not report the Trigger
selected for the query. Consumers of progress events cannot determine whether a
query uses ProcessingTime, Continuous, RealTime, AvailableNow, or OneTime from
the progress payload.
h2. Proposed change
Add an optional top-level trigger object to StreamingQueryProgress JSON and
preserve it when QueryProgressEvent JSON and status-store records are
serialized and deserialized.
The object contains the selected trigger type and any relevant configuration:
{code:json}
"trigger": {"type": "ProcessingTime", "intervalMs": 1000}
"trigger": {"type": "RealTime", "batchDurationMs": 300000}
"trigger": {"type": "AvailableNow"}
{code}
The type field is the discriminator for the remaining fields. ProcessingTime
uses intervalMs for its scheduling interval, and Continuous uses intervalMs for
its checkpoint interval. RealTime uses batchDurationMs because the value
configures each real-time batch's duration rather than a scheduling interval.
AvailableNow and OneTime do not have a duration field. Consumers should read
type first and then the configuration field defined for that trigger type.
The field should:
* support all built-in trigger types;
* provide stable, queryable type and configuration fields;
* remain backward-compatible with progress and event records that do not
contain it.
> Expose trigger metadata in StreamingQueryProgress JSON
> ------------------------------------------------------
>
> Key: SPARK-59986
> URL: https://issues.apache.org/jira/browse/SPARK-59986
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Alyssa Gorbaneva
> Priority: Major
>
> h2. Motivation
> StreamingQueryProgress JSON reports query identifiers, batch timings, source,
> sink, state operator, and observed metrics, but it does not report the
> Trigger selected for the query. Consumers of progress events cannot determine
> whether a query uses ProcessingTime, Continuous, RealTime, AvailableNow, or
> OneTime from the progress payload.
> h2. Proposed change
> Add an optional top-level trigger object to StreamingQueryProgress JSON and
> preserve it when QueryProgressEvent JSON and status-store records are
> serialized and deserialized.
> The object contains the selected trigger type and any relevant configuration:
> {code:json}
> "trigger": {"type": "ProcessingTime", "intervalMs": 1000}
> "trigger": {"type": "RealTime", "batchDurationMs": 300000}
> "trigger": {"type": "AvailableNow"}
> {code}
> The type field is the discriminator for the remaining fields. The
> configuration names match the existing Trigger APIs and execution semantics.
> ProcessingTime uses intervalMs to schedule batch starts, and Continuous uses
> intervalMs to schedule checkpoints. RealTime uses batchDurationMs as each
> real-time scan's deadline; because consecutive real-time batches start
> immediately, this duration also influences batch cadence. AvailableNow and
> OneTime do not have a duration field. Consumers should read type first and
> then the configuration field defined for that trigger type.
> The field should:
> * support all built-in trigger types;
> * provide stable, queryable type and configuration fields;
> * remain backward-compatible with progress and event records that do not
> contain it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
