Re: Inquiry in regards to a New onQuery Method for StreamingQueryListener

2025-04-06 Thread Jevon Cowell
I’ve been thinking about this quite a bit today and what an implementation on the spark side would look like. After some deliberation I concluded: We should instead have an `onQueryTriggerStart` method that is published every time a MicroBatch is triggered This should of course be disabled by d

Re: Inquiry in regards to a New onQuery Method for StreamingQueryListener

2025-04-04 Thread Jevon Cowell
Hey Jungtaek!Wanted to update the mailing list on my current approach in case others wanted something similar.I created an asynchronous poller iterates though all active queries and checks of the isTriggering boolean value is true.Here’s an example code snippet: ```javapublic static void checkAndUp

Re: Inquiry in regards to a New onQuery Method for StreamingQueryListener

2025-03-27 Thread Jungtaek Lim
Hi Jevon, > From testing, I see that `onQueryIdle` does not trigger when a query is waiting for the next trigger interval. Yeah it's based on trigger - if no trigger has been triggered, the event cannot be sent. > I wanted to get thoughts on whether it’s worth implementing a new QueryListener me