Jevon Cowell created SPARK-51731:
------------------------------------
Summary: Introduce a new callback "onQueryTriggerStart" to
StreamingQueryListener
Key: SPARK-51731
URL: https://issues.apache.org/jira/browse/SPARK-51731
Project: Spark
Issue Type: Improvement
Components: Connect, Java API, PySpark, Spark Core, Structured
Streaming
Affects Versions: 4.1.0
Reporter: Jevon Cowell
Currently, {{{}StreamingQueryListener{}}}s does not notify users when a query
is triggered; it only provides updates when a query starts, progresses, becomes
idle, or terminates. By introducing a a new event callback that is emitted
whenever a query is triggered, users will be informed of this occurrence,
allowing for a variety of use cases.
In my use case, I plan to use this new event along with {{QueryProgress}} and
{{QueryIdle}} to develop a Maintenance Job Orchestration System. This system
will programmatically submit and execute Spark commands, determining the
eligibility of maintenance job executions based on the activity status of
streaming queries.
I've already started an initial implementation on my own
[fork|https://github.com/JevonCowell/spark/pull/1] but before proceeding with
writing test to verify functionality, I want to ensure this is something apache
folks are interested in having added in the first place.
Note that for a feature like this I'm:
* Ensuring there is backwards compatibility so that any folks with their own
custom SparkQueryListener won't break upon upgrading.
* Disabling by default to not add unnecessary load to folks who are just
upgrading their dependencies
* Aiming to set a sane interval to determine how often an event like this
should be emitted per query
I tired my best to see if a request like this had come up before but was unable
to. If it has please link me back to that ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
