+1 (non-binding) El jue, 10 abr 2025, 1:50, Burak Yavuz <brk...@gmail.com> escribió:
> +1 > > On Wed, Apr 9, 2025 at 4:33 PM Szehon Ho <szehon.apa...@gmail.com> wrote: > >> +1 really excited to finally see Materialized View finally make its way >> to Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg) >> already supporting it. >> >> Thanks >> Szehon >> >> On Wed, Apr 9, 2025 at 2:33 AM Martin Grund <mar...@databricks.com.invalid> >> wrote: >> >>> +1 >>> >>> On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>>> +1 >>>> >>>> Dr Mich Talebzadeh, >>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, 9 Apr 2025 at 08:07, Peter Toth <peter.t...@gmail.com> wrote: >>>> >>>>> +1 >>>>> >>>>> On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan <pan3...@gmail.com> wrote: >>>>> >>>>>> +1 (non-binding) >>>>>> >>>>>> Glad to see Spark SQL extended to streaming use cases. >>>>>> >>>>>> Thanks, >>>>>> Cheng Pan >>>>>> >>>>>> >>>>>> >>>>>> On Apr 9, 2025, at 14:43, Anton Okolnychyi <aokolnyc...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> +1 >>>>>> >>>>>> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee <qcsd2...@gmail.com> пише: >>>>>> >>>>>>> +1 I'm delighted that it will be open-sourced, enabling greater >>>>>>> integration with Iceberg/Delta to unlock more value. >>>>>>> >>>>>>> Jungtaek Lim <kabhwan.opensou...@gmail.com> 于2025年4月9日周三 10:47写道： >>>>>>> > >>>>>>> > +1 looking forward to seeing this make progress! >>>>>>> > >>>>>>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie <yangji...@apache.org> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> +1 >>>>>>> >> >>>>>>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote: >>>>>>> >> > +1 >>>>>>> >> > >>>>>>> >> > I am actually pretty excited to have this. Happy to see this >>>>>>> being proposed. >>>>>>> >> > >>>>>>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun <sunc...@apache.org> >>>>>>> wrote: >>>>>>> >> > >>>>>>> >> > > +1. Super excited about this effort! >>>>>>> >> > > >>>>>>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao < >>>>>>> huaxin.ga...@gmail.com> wrote: >>>>>>> >> > > >>>>>>> >> > >> +1 I support this SPIP because it simplifies data pipeline >>>>>>> management and >>>>>>> >> > >> enhances error detection. >>>>>>> >> > >> >>>>>>> >> > >> >>>>>>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal < >>>>>>> dkbis...@gmail.com> wrote: >>>>>>> >> > >> >>>>>>> >> > >>> Excited to see this heading toward open source — >>>>>>> materialized views and >>>>>>> >> > >>> other features will bring a lot of value. >>>>>>> >> > >>> +1 (non-binding) >>>>>>> >> > >>> >>>>>>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza < >>>>>>> sa...@apache.org> wrote: >>>>>>> >> > >>> >>>>>>> >> > >>>> Hi Khalid – the CLI in the current proposal will need to >>>>>>> be built on >>>>>>> >> > >>>> top of internal APIs for constructing and launching >>>>>>> pipeline executions. >>>>>>> >> > >>>> We'll have the option to expose these in the future. >>>>>>> >> > >>>> >>>>>>> >> > >>>> It would be worthwhile to understand the use cases in more >>>>>>> depth before >>>>>>> >> > >>>> exposing these, because APIs are one-way doors and can be >>>>>>> costly to >>>>>>> >> > >>>> maintain. >>>>>>> >> > >>>> >>>>>>> >> > >>>> On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov < >>>>>>> >> > >>>> khalidmammad...@gmail.com> wrote: >>>>>>> >> > >>>> >>>>>>> >> > >>>>> Looks great! >>>>>>> >> > >>>>> QQ: will user able to run this pipeline from normal code? >>>>>>> I.e. can I >>>>>>> >> > >>>>> trigger a pipeline from *driver* code based on some >>>>>>> condition etc. or >>>>>>> >> > >>>>> it must be executed via separate shell command ? >>>>>>> >> > >>>>> As a background Databricks imposes similar limitation >>>>>>> where as you >>>>>>> >> > >>>>> cannot run normal Spark code and DLT on the same cluster >>>>>>> for some reason >>>>>>> >> > >>>>> and forces to use two clusters increasing the cost and >>>>>>> latency. >>>>>>> >> > >>>>> >>>>>>> >> > >>>>> On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> >>>>>>> wrote: >>>>>>> >> > >>>>> >>>>>>> >> > >>>>>> Hi all – starting a discussion thread for a SPIP that >>>>>>> I've been >>>>>>> >> > >>>>>> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie >>>>>>> Yang: [JIRA >>>>>>> >> > >>>>>> <https://issues.apache.org/jira/browse/SPARK-51727>] >>>>>>> [Doc >>>>>>> >> > >>>>>> < >>>>>>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0 >>>>>>> > >>>>>>> >> > >>>>>> ]. >>>>>>> >> > >>>>>> >>>>>>> >> > >>>>>> The SPIP proposes extending Spark's lazy, declarative >>>>>>> execution model >>>>>>> >> > >>>>>> beyond single queries, to pipelines that keep multiple >>>>>>> datasets up to date. >>>>>>> >> > >>>>>> It introduces the ability to compose multiple >>>>>>> transformations into a single >>>>>>> >> > >>>>>> declarative dataflow graph. >>>>>>> >> > >>>>>> >>>>>>> >> > >>>>>> Declarative pipelines aim to simplify the development >>>>>>> and management >>>>>>> >> > >>>>>> of data pipelines, by removing the need for manual >>>>>>> orchestration of >>>>>>> >> > >>>>>> dependencies and making it possible to catch many errors >>>>>>> before any >>>>>>> >> > >>>>>> execution steps are launched. >>>>>>> >> > >>>>>> >>>>>>> >> > >>>>>> Declarative pipelines can include both batch and >>>>>>> streaming >>>>>>> >> > >>>>>> computations, leveraging Structured Streaming for stream >>>>>>> processing and new >>>>>>> >> > >>>>>> materialized view syntax for batch processing. Tight >>>>>>> integration with Spark >>>>>>> >> > >>>>>> SQL's analyzer enables deeper analysis and earlier error >>>>>>> detection than is >>>>>>> >> > >>>>>> achievable with more generic frameworks. >>>>>>> >> > >>>>>> >>>>>>> >> > >>>>>> Let us know what you think! >>>>>>> >> > >>>>>> >>>>>>> >> > >>>>>> >>>>>>> >> > >>>>>>> >> >>>>>>> >> >>>>>>> --------------------------------------------------------------------- >>>>>>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>> >> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>> >>>>>>> >>>>>>
