Re: Inquiry in regards to a New onQuery Method for StreamingQueryListener

Hey Jungtaek!Wanted to update the mailing list on my current approach in case others wanted something similar.I created an asynchronous poller iterates though all active queries and checks of the isTriggering boolean value is true.Here’s an example code snippet: ```javapublic static void checkAndUp

Re: Java coding with spark API

Hey Tim! What are you aiming to achieve exactly? Regards, Jevon C > On Apr 4, 2025, at 3:54 AM, tim wade wrote: > > Hello > > I am just newbie to spark. I am programming with Java mainly, knowing scala > very bit. > > Can I just write code with java to talk to Spark's java API for submit

Announcing the Community Over Code 2025 Streaming Track

Hi all, Community Over Code , the ASF conference, will be held in Minneapolis, Minnesota, September 11-14, 2025. The call for presentations is open now through April 21, 2025. I am one of the co-chairs for th

Kubeflow Spark-Operator

Hello folks, My colleague has posted this issue on Github: https://github.com/kubeflow/spark-operator/issues/2491 I'm wondering whether anyone here is using the kubeflow, Spark-Operator and could provide any insight into what's happening here. I know he's been stumped for a few days and it doesn'

Re: Executors not getting released dynamically once task is over

Hi Shivang, This sounds like classic Spark-on-Kubernetes behavior: - Executors do not necessarily shut down immediately after finishing their tasks, unless: - Dynamic resource allocation is enabled. - And Spark knows it can safely scale down - The Driver pod manages the whole lifecycle. If you

Correctness Issue: UNIX_SECONDS() mismatch with TO_UTC_TIMESTAMP() result in Spark 3.5.1

Hi Spark Dev Team, I believe I've encountered a potential bug in spark 3.5.1 concerning the UNIX_SECONDS function when used with TO_UTC_TIMESTAMP. When converting a timestamp from a specific timezone (e.g., 'Europe/Amsterdam') to UTC and then getting its Unix seconds, the result seems incorrect.

Executors not getting released dynamically once task is over

Hi Team, We are using spark java 3.5.3 and we have a requirement to run a batch of millions of transactions. i.e. I am running batch of 1M with 640 executors and each executor is having 8 cores and 16 GB memory running under kubernetes cluster. But our conversation is that one executor is don

Java coding with spark API

Hello I am just newbie to spark. I am programming with Java mainly, knowing scala very bit. Can I just write code with java to talk to Spark's java API for submitting jobs? (the main job is a stru-streaming job). Thanks. -