Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1 (non-binding) - this will be really useful for latency sensitive workloads Thanks, Anish On Sun, Jun 1, 2025 at 11:02 PM Sakthi wrote: > +1 (non-binding) > > On Sun, Jun 1, 2025 at 10:51 PM Gengliang Wang wrote: > >> +1 >> >> On Sun, Jun 1, 2025 at 10:20 PM Denny Lee wrote: >> >>> +1 (non

[DISCUSS][MINOR] Fix broken link in spark-website for SS Programming Guide

Hi, We have a broken link for the latest docs for the 4.0 release. This page: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html has a hyperlink that points to the contents of the Structured Streaming guide. But it seems this link is broken and points back to the mai

Re: [VOTE] Release Spark 4.0.0 (RC6)

Hi Adam, Thanks for reporting the issue. @Eric Marnadi has a PR for the fix here - https://github.com/apache/spark/pull/50924 @Wenchen Fan - would be good to include this in 4.0 RC Thanks, Anish On Fri, May 16, 2025 at 8:05 AM Adam Binford wrote: > I think I found a bug when trying to use t

Re: Seek for consensus on landing Spark Connect implementation for transformWithState in Spark 4.0.0

+1 - Would be great to get this into the Spark 4.0 release. Thanks, Anish On Mon, Mar 3, 2025 at 9:35 PM Jungtaek Lim wrote: > Hi dev, > > We are going to introduce a new API named `transformWithState` for > streaming query, which allows users to perform more complex stateful > operation in use

Re: [DISCUSS] Ongoing projects for Spark 4.0

Hi, We are working on the new arbitrary state API support for streaming called transformWithState - (epic here - https://issues.apache.org/jira/browse/SPARK-46815). We have few PRs left and we would like to get these into Spark 4.0. However, if we can't get them in within the deadline, we will tar

Re: Structured Streaming Programming Guide Improvements + Versionless Docs Update

+1 - I think it would be super helpful to split this single long doc. Couple of points that might be useful: - If we could create an individual page per operator with some concrete examples, that would be great - As Jungtaek mentioned, separating the API reference from the conceptual portions migh

Re: Enhanced Console Sink for Structured Streaming

Hi Neil, Thanks for putting this together. +1 to the proposal of enhancing the console sink further. I think it will help new users understand some of the streaming/micro-batch semantics a bit better in Spark. Agree with not having verbose mode enabled by default. I think step 1 described above s

Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

Thanks Jungtaek for creating the Vote thread. +1 (non-binding) from my side too. Thanks, Anish On Tue, Jan 9, 2024 at 6:09 AM Jungtaek Lim wrote: > Starting with my +1 (non-binding). Thanks! > > On Tue, Jan 9, 2024 at 9:37 AM Jungtaek Lim > wrote: > >> Hi all, >> >> I'd like to start the vote

Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

from non-US during the holiday. We can either address feedback >> altogether after the holiday (Anish is in the US) or I can answer if the >> feedback is more about the question. Thanks! >> >> On Thu, Nov 23, 2023 at 5:27 AM Anish Shrigondekar < >> anish.shrigonde...

[DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

Hi dev, I would like to start a discussion on "Structured Streaming - Arbitrary State API v2". This proposal aims to address a bunch of limitations we see today using mapGroupsWithState/flatMapGroupsWithState operator. The detailed set of limitations is described in the SPIP doc. We propose to su

Re: [VOTE] SPIP: State Data Source - Reader

+1 (non-binding) Thanks, Anish On Mon, Oct 23, 2023 at 5:01 PM Wenchen Fan wrote: > +1 > > On Mon, Oct 23, 2023 at 4:03 PM Jungtaek Lim > wrote: > >> Starting with my +1 (non-binding). Thanks! >> >> On Mon, Oct 23, 2023 at 1:23 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> H

Re: [DISCUSS] SPIP: State Data Source - Reader

Hi Jungtaek, Thanks for putting this together. +1 from me and looks good overall. Posted some minor comments/questions to the doc. Thanks, Anish On Mon, Oct 16, 2023 at 11:25 AM Bartosz Konieczny wrote: > Thank you, Jungtaek, for your answers! It's clear now. > > +1 for me. It seems like a pre

Re: [DISCUSS] Deprecate DStream in 3.4

+1 on the Dstreams deprecation proposal On Fri, Jan 13, 2023 at 10:47 AM Jerry Peng wrote: > +1 in general for marking the DStreams API as deprecated > > Jungtaek, can you please provide / elaborate on the concrete actions you > intend on taking for the depreciation process? > > Best, > > Jerry