Alyssa Gorbaneva created SPARK-59986:
----------------------------------------
Summary: Expose trigger metadata in StreamingQueryProgress JSON
Key: SPARK-59986
URL: https://issues.apache.org/jira/browse/SPARK-59986
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Alyssa Gorbaneva
h2. Motivation
StreamingQueryProgress JSON reports query identifiers, batch timings, source,
sink, state operator, and observed metrics, but it does not report the Trigger
selected for the query. Consumers of progress events cannot determine whether a
query uses ProcessingTime, Continuous, RealTime, AvailableNow, or OneTime from
the progress payload.
h2. Proposed change
Add an optional top-level trigger field to StreamingQueryProgress JSON and
preserve it when QueryProgressEvent JSON and status-store records are
serialized and deserialized.
The field should:
* include the selected trigger type and relevant configuration;
* support all built-in trigger types;
* remain backward-compatible with progress and event records that do not
contain it.
Two representations are under consideration:
Structured:
{code:json}
"trigger": {"type": "ProcessingTime", "intervalMs": 1000}
"trigger": {"type": "AvailableNow"}
{code}
Compact scalar:
{code:json}
"trigger": "ProcessingTime(1000)"
"trigger": "AvailableNow"
{code}
The final representation should provide a stable contract for programmatic
consumers while remaining consistent with existing StreamingQueryProgress JSON.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
