+1

On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim <kabhwan.opensou...@gmail.com>
wrote:

> +1 looking forward to seeing this make progress!
>
> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <yangji...@apache.org> wrote:
>
>> +1
>>
>> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>> > +1
>> >
>> > I am actually pretty excited to have this. Happy to see this being
>> proposed.
>> >
>> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <sunc...@apache.org> wrote:
>> >
>> > > +1. Super excited about this effort!
>> > >
>> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao <huaxin.ga...@gmail.com>
>> wrote:
>> > >
>> > >> +1 I support this SPIP because it simplifies data pipeline
>> management and
>> > >> enhances error detection.
>> > >>
>> > >>
>> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal <dkbis...@gmail.com>
>> wrote:
>> > >>
>> > >>> Excited to see this heading toward open source — materialized views
>> and
>> > >>> other features will bring a lot of value.
>> > >>> +1 (non-binding)
>> > >>>
>> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza <sa...@apache.org>
>> wrote:
>> > >>>
>> > >>>> Hi Khalid – the CLI in the current proposal will need to be built
>> on
>> > >>>> top of internal APIs for constructing and launching pipeline
>> executions.
>> > >>>> We'll have the option to expose these in the future.
>> > >>>>
>> > >>>> It would be worthwhile to understand the use cases in more depth
>> before
>> > >>>> exposing these, because APIs are one-way doors and can be costly to
>> > >>>> maintain.
>> > >>>>
>> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>> > >>>> khalidmammad...@gmail.com> wrote:
>> > >>>>
>> > >>>>> Looks great!
>> > >>>>> QQ: will user able to run this pipeline from normal code? I.e.
>> can I
>> > >>>>> trigger a pipeline from *driver* code based on some condition
>> etc. or
>> > >>>>> it must be executed via separate shell command ?
>> > >>>>> As a background Databricks imposes similar limitation where as you
>> > >>>>> cannot run normal Spark code and DLT on the same cluster for some
>> reason
>> > >>>>> and forces to use two clusters increasing the cost and latency.
>> > >>>>>
>> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote:
>> > >>>>>
>> > >>>>>> Hi all – starting a discussion thread for a SPIP that I've been
>> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang:
>> [JIRA
>> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc
>> > >>>>>> <
>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>> >
>> > >>>>>> ].
>> > >>>>>>
>> > >>>>>> The SPIP proposes extending Spark's lazy, declarative execution
>> model
>> > >>>>>> beyond single queries, to pipelines that keep multiple datasets
>> up to date.
>> > >>>>>> It introduces the ability to compose multiple transformations
>> into a single
>> > >>>>>> declarative dataflow graph.
>> > >>>>>>
>> > >>>>>> Declarative pipelines aim to simplify the development and
>> management
>> > >>>>>> of data pipelines, by  removing the need for manual
>> orchestration of
>> > >>>>>> dependencies and making it possible to catch many errors before
>> any
>> > >>>>>> execution steps are launched.
>> > >>>>>>
>> > >>>>>> Declarative pipelines can include both batch and streaming
>> > >>>>>> computations, leveraging Structured Streaming for stream
>> processing and new
>> > >>>>>> materialized view syntax for batch processing. Tight integration
>> with Spark
>> > >>>>>> SQL's analyzer enables deeper analysis and earlier error
>> detection than is
>> > >>>>>> achievable with more generic frameworks.
>> > >>>>>>
>> > >>>>>> Let us know what you think!
>> > >>>>>>
>> > >>>>>>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Reply via email to