+1
By unifying batch and low-latency streaming in Spark, we can eliminate the
need for separate streaming engines, reducing system complexity and
operational cost. Excited to see this direction!

On Wed, May 28, 2025 at 9:08 AM Mich Talebzadeh <mich.talebza...@gmail.com>
wrote:

> Hi,
>
> My point about "in real time application or data, there is nothing as an
> answer which is supposed to be late and correct. The timeliness is part of
> the application. if I get the right answer too slowly it becomes useless or
> wrong" is actually fundamental to *why* we need this Spark Structured
> Streaming proposal.
>
> The proposal is precisely about enabling Spark to power applications
> where, as I define it, the *timeliness* of the answer is as critical as
> its *correctness*. Spark's current streaming engine, primarily operating
> on micro-batches, often delivers results that are technically "correct" but
> arrive too late to be truly useful for certain high-stakes, real-time
> scenarios. This makes them "useless or wrong" in a practical,
> business-critical sense.
>
> For example *in real-time fraud detection* and In *high-frequency
> trading,* market data or trade execution commands must be delivered with
> minimal latency. Even a slight delay can mean missed opportunities or
> significant financial losses, making a "correct" price update useless if
> it's not instantaneous. able for these demanding use cases, where a "late
> but correct" answer is simply not good enough. As a colliery it is a
> fundamental concept, so it has to be treated as such not as a comment.in
> SPIP
>
> Hope this clarifies the connection in practical terms
> Dr Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
>    view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Wed, 28 May 2025 at 16:32, Denny Lee <denny.g....@gmail.com> wrote:
>
>> Hey Mich,
>>
>> Sorry, I may be missing something here but what does your definition here
>> have to do with the SPIP?   Perhaps add comments directly to the SPIP to
>> provide context as the code snippet below is a direct copy from the SPIP
>> itself.
>>
>> Thanks,
>> Denny
>>
>>
>>
>>
>> On Wed, May 28, 2025 at 06:48 Mich Talebzadeh <mich.talebza...@gmail.com>
>> wrote:
>>
>>> just to add
>>>
>>> A stronger definition of real time. The engineering definition of real
>>> time is roughly fast enough to be interactive
>>>
>>> However, I put a stronger definition. In real time application or data,
>>> there is nothing as an answer which is supposed to be late and correct. The
>>> timeliness is part of the application.if I get the right answer too slowly
>>> it becomes useless or wrong
>>>
>>>
>>>
>>> Dr Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>>    view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, 28 May 2025 at 11:10, Mich Talebzadeh <mich.talebza...@gmail.com>
>>> wrote:
>>>
>>>> The current limitations in SSS come from micro-batching.If you are
>>>> going to reduce micro-batching, this reduction must be balanced against the
>>>> available processing capacity of the cluster to prevent back pressure and
>>>> instability. In the case of Continuous Processing mode, a specific
>>>> continuous trigger with a desired checkpoint interval quote
>>>>
>>>> "
>>>> df.writeStream
>>>>    .format("...")
>>>>    .option("...")
>>>>    .trigger(Trigger.RealTime(“300 Seconds”))    // new trigger type to
>>>> enable real-time Mode
>>>>    .start()
>>>> This Trigger.RealTime signals that the query should run in the new
>>>> ultra low-latency execution mode.  A time interval can also be specified,
>>>> e.g. “300 Seconds”, to indicate how long each micro-batch should run for.
>>>> "
>>>>
>>>> will inevitably depend on many factors. Not that simple
>>>> HTH
>>>>
>>>>
>>>> Dr Mich Talebzadeh,
>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>>
>>>>    view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, 28 May 2025 at 05:13, Jerry Peng <jerry.boyang.p...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I want to start a discussion thread for the SPIP titled “Real-Time
>>>>> Mode in Apache Spark Structured Streaming” that I've been working on with
>>>>> Siying Dong, Indrajit Roy, Chao Sun, Jungtaek Lim, and Michael Armbrust: [
>>>>> JIRA <https://issues.apache.org/jira/browse/SPARK-52330>] [Doc
>>>>> <https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?usp=sharing>
>>>>> ].
>>>>>
>>>>> The SPIP proposes a new execution mode called “Real-time Mode” in
>>>>> Spark Structured Streaming that significantly lowers end-to-end latency 
>>>>> for
>>>>> processing streams of data.
>>>>>
>>>>> A key principle of this proposal is compatibility. Our goal is to make
>>>>> Spark capable of handling streaming jobs that need results almost
>>>>> immediately (within O(100) milliseconds). We want to achieve this without
>>>>> changing the high-level DataFrame/Dataset API that users already use – so
>>>>> existing streaming queries can run in this new ultra-low-latency mode by
>>>>> simply turning it on, without rewriting their logic.
>>>>>
>>>>> In short, we’re trying to enable Spark to power real-time applications
>>>>> (like instant anomaly alerts or live personalization) that today cannot
>>>>> meet their latency requirements with Spark’s current streaming engine.
>>>>>
>>>>> We'd greatly appreciate your feedback, thoughts, and suggestions on
>>>>> this approach!
>>>>>
>>>>>

Reply via email to