Re: [VOTE] Release Spark 3.3.0 (RC5)

I agree with Jungtaek, -1 from me because of the issue of Kafka source throwing an error with an incorrect error message that was introduced recently. This may mislead users and cause unnecessary confusion. On Wed, Jun 8, 2022 at 12:04 AM Jungtaek Lim wrote: > Apologize for late participation.

[DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

Hi all, I would like to start the discussion for a SPIP, Asynchronous Offset Management in Structured Streaming. The high level summary of the SPIP is that currently in Structured Streaming we perform a couple of offset management operations for progress tracking purposes synchronously on the cri

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

> ps. Eventually we could probably discuss retiring continuous mode if the > new approach gets accepted and eventually considered as a stable one after > several minor releases. That's just me. > > On Wed, Nov 23, 2022 at 5:16 AM Jerry Peng > wrote: > >> Hi all, >>

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

d move it to deprecated ? >> >> I am much more comfortable not supporting this SPIP for CM if it was >> deprecated. >> >> Thoughts ? >> >> Regards, >> Mridul >> >> >> >> >> On Wed, Nov 23, 2022 at 1:16 AM Jerry Pen

Re: [VOTE][RESULT][SPIP] Asynchronous Offset Management in Structured Streaming

Thanks Jungtaek for shepherding this effort! On Sun, Dec 4, 2022 at 6:25 PM Jungtaek Lim wrote: > The vote passes with 7 +1s (5 binding +1s). > Thanks to all who reviews the SPIP doc and votes! > > (* = binding) > +1: > - Jungtaek Lim > - Xingbo Jiang > - Mridul Muralidharan (*) > - Hyukjin Kwon

Re: [DISCUSS] Deprecate DStream in 3.4

+1 in general for marking the DStreams API as deprecated Jungtaek, can you please provide / elaborate on the concrete actions you intend on taking for the depreciation process? Best, Jerry On Thu, Jan 12, 2023 at 11:16 PM L. C. Hsieh wrote: > +1 > > On Thu, Jan 12, 2023 at 10:39 PM Jungtaek L

Re: Enhanced Console Sink for Structured Streaming

I am generally a +1 on this as we can use this information in our docs to demonstrate certains concepts to potential users. I am in agreement with other reviewers that we should keep the existing default behavior of the console sink. This new style of output should be enabled behind a flag. As f

[DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

Hi all, I want to start a discussion thread for the SPIP titled “Real-Time Mode in Apache Spark Structured Streaming” that I've been working on with Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [JIRA ] [Doc

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh < >>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>> >>>

Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

Thank you all! Glad to see this much interest and support for this initiative! On Wed, Jun 4, 2025 at 1:27 PM L. C. Hsieh wrote: > Hi all, > > Thanks all for participating and your support! The vote has been passed. > I'll send out the result in a separate thread. > > On Mon, Jun 2, 2025 at 7:5

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/ "Apache Flink 2.0.0: A new Era of *Real-Time* Data Processing" Thus, what the term "rea-time" implies in this should not be confusing for folks in this area. On Thu, May 29, 2025 at 10:22 PM Jerry

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

vide context as the code snippet below is a >> direct copy >> > >>>>>>>>> from the SPIP itself. >> > >>>>>>>>> >> > >>>>>>>>> Thanks, >> > >>>>>>>>> Denny >> &

Re: [DISCUSS][MINOR] Fix broken link in spark-website for SS Programming Guide

+1 for fixing this immediately. Anish, thanks for pointing this issue out! On Fri, May 30, 2025 at 12:12 AM Jungtaek Lim wrote: > I’m +1 to fix this in website for 4.0.0 immediately. > > I got some inputs about this and they were unable to figure out the > correct page url. I’m mostly sure it w

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

ere but what does your >> > >>>>>>>>> definition here have to do with the SPIP? Perhaps add >> comments directly >> > >>>>>>>>> to the SPIP to provide context as the code snippet below is a >> di

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

ument. Technical clarity and > accuracy should be non-negotiable. > > > On Thu, May 29, 2025 at 10:27 PM Jerry Peng > wrote: > >> Mark, >> >> As an example of my point if you go the the Apache Storm (another stream >> processing engine) website: >> &g

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

edin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > > > On Fri, 30 May 2025 at 07:57, Jerry Peng > wrote: > >> Mark, >> >> For real-time systems there is a concept of "soft" real-time and "hard&qu

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

ke the use of marketing-speak "real-time" that isn't really real-time in > engineering documents or API namespaces. > > On Thu, May 29, 2025 at 10:43 PM Jerry Peng > wrote: > >> Mark, >> >> I thought we are simply discussing the naming of the mode?