Re: [DISCUSS] SPIP: Declarative Pipelines

This sounds quite interesting. +1 to What Szheon said about excitement around MVs. Happy to collaborate. On Wed, Apr 9, 2025 at 5:29 PM Ángel Álvarez Pascua < angel.alvarez.pas...@gmail.com> wrote: > +1 (non-binding) > > El jue, 10 abr 2025, 1:50, Burak Yavuz escribió: > >> +1 >> >> On Wed, Apr

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 (non-binding) On April 9, 2025 7:29:40 AM GMT+02:00, Rishab Joshi wrote: >+1 Exciting. >Rishab Joshi > >On Tue, Apr 8, 2025, 10:04 PM Ruifeng Zheng wrote: > >> +1 >> >> On Wed, Apr 9, 2025 at 12:57 PM Denny Lee wrote: >> >>> +1 (non-binding) >>> >>> On Tue, Apr 8, 2025 at 9:53 PM Yuming Wan

Re: [DISCUSS] SPIP: Declarative Pipelines

Hi Khalid – the CLI in the current proposal will need to be built on top of internal APIs for constructing and launching pipeline executions. We'll have the option to expose these in the future. It would be worthwhile to understand the use cases in more depth before exposing these, because APIs ar

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 (non-binding) On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang wrote: > +1 > > On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim > wrote: > >> +1 looking forward to seeing this make progress! >> >> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie wrote: >> >>> +1 >>> >>> On 2025/04/09 01:07:57 Hyukjin Kwon wr

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 looking forward to seeing this make progress! On Wed, Apr 9, 2025 at 11:32 AM Yang Jie wrote: > +1 > > On 2025/04/09 01:07:57 Hyukjin Kwon wrote: > > +1 > > > > I am actually pretty excited to have this. Happy to see this being > proposed. > > > > On Wed, 9 Apr 2025 at 01:55, Chao Sun wrote:

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 4:33 PM Szehon Ho wrote: > +1 really excited to finally see Materialized View finally make its way to > Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg) > already supporting it. > > Thanks > Szehon > > On Wed, Apr 9, 2025 at 2:33 AM Martin Grund

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 (non-binding) El jue, 10 abr 2025, 1:50, Burak Yavuz escribió: > +1 > > On Wed, Apr 9, 2025 at 4:33 PM Szehon Ho wrote: > >> +1 really excited to finally see Materialized View finally make its way >> to Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg) >> already suppo

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 really excited to finally see Materialized View finally make its way to Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg) already supporting it. Thanks Szehon On Wed, Apr 9, 2025 at 2:33 AM Martin Grund wrote: > +1 > > On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh >

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 Kent Yao Sem 于2025年4月9日周三 14:08写道： > +1 (non-binding) > > > On April 9, 2025 7:29:40 AM GMT+02:00, Rishab Joshi > wrote: > >> +1 Exciting. >> Rishab Joshi >> >> On Tue, Apr 8, 2025, 10:04 PM Ruifeng Zheng wrote: >> >>> +1 >>> >>> On Wed, Apr 9, 2025 at 12:57 PM Denny Lee wrote: >>> +

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh wrote: > +1 > > Dr Mich Talebzadeh, > Architect | Data Science | Financial Crime | Forensic Analysis | GDPR > >view my Linkedin profile > > > > > > > On Wed, 9 Apr 2025 at 08:07, Pete

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile On Wed, 9 Apr 2025 at 08:07, Peter Toth wrote: > +1 > > On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan wrote: > >>

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan wrote: > +1 (non-binding) > > Glad to see Spark SQL extended to streaming use cases. > > Thanks, > Cheng Pan > > > > On Apr 9, 2025, at 14:43, Anton Okolnychyi wrote: > > +1 > > вт, 8 квіт. 2025 р. о 23:36 Jacky Lee пише: > >> +1 I'm delighted that i

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 (non-binding) Glad to see Spark SQL extended to streaming use cases. Thanks, Cheng Pan > On Apr 9, 2025, at 14:43, Anton Okolnychyi wrote: > > +1 > > вт, 8 квіт. 2025 р. о 23:36 Jacky Lee > пише: >> +1 I'm delighted that it will be open-sourced, enabling great

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim wrote: > +1 looking forward to seeing this make progress! > > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie wrote: > >> +1 >> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote: >> > +1 >> > >> > I am actually pretty excited to have this. Happy to see this b

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 вт, 8 квіт. 2025 р. о 23:36 Jacky Lee пише: > +1 I'm delighted that it will be open-sourced, enabling greater > integration with Iceberg/Delta to unlock more value. > > Jungtaek Lim 于2025年4月9日周三 10:47写道： > > > > +1 looking forward to seeing this make progress! > > > > On Wed, Apr 9, 2025 at

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 I'm delighted that it will be open-sourced, enabling greater integration with Iceberg/Delta to unlock more value. Jungtaek Lim 于2025年4月9日周三 10:47写道： > > +1 looking forward to seeing this make progress! > > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie wrote: >> >> +1 >> >> On 2025/04/09 01:07:57 H

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 Exciting. Rishab Joshi On Tue, Apr 8, 2025, 10:04 PM Ruifeng Zheng wrote: > +1 > > On Wed, Apr 9, 2025 at 12:57 PM Denny Lee wrote: > >> +1 (non-binding) >> >> On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang wrote: >> >>> +1 >>> >>> On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim < >>> kabhwan.open

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 12:57 PM Denny Lee wrote: > +1 (non-binding) > > On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang wrote: > >> +1 >> >> On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> +1 looking forward to seeing this make progress! >>> >>> On We

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 On 2025/04/09 01:07:57 Hyukjin Kwon wrote: > +1 > > I am actually pretty excited to have this. Happy to see this being proposed. > > On Wed, 9 Apr 2025 at 01:55, Chao Sun wrote: > > > +1. Super excited about this effort! > > > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao wrote: > > > >> +1

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 I am actually pretty excited to have this. Happy to see this being proposed. On Wed, 9 Apr 2025 at 01:55, Chao Sun wrote: > +1. Super excited about this effort! > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao wrote: > >> +1 I support this SPIP because it simplifies data pipeline management an

Re: [DISCUSS] SPIP: Declarative Pipelines

+1. Super excited about this effort! On Tue, Apr 8, 2025 at 9:47 AM huaxin gao wrote: > +1 I support this SPIP because it simplifies data pipeline management and > enhances error detection. > > > On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal wrote: > >> Excited to see this heading toward open sou

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 I support this SPIP because it simplifies data pipeline management and enhances error detection. On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal wrote: > Excited to see this heading toward open source — materialized views and > other features will bring a lot of value. > +1 (non-binding) > > On

Re: [DISCUSS] SPIP: Declarative Pipelines

Excited to see this heading toward open source — materialized views and other features will bring a lot of value. +1 (non-binding) On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza wrote: > Hi Khalid – the CLI in the current proposal will need to be built on top > of internal APIs for constructing and

Re: [DISCUSS] SPIP: Declarative Pipelines

There are many projects in the Spark ecosystem — like Deequ and Great Expectations — that are focused on expressing and enforcing data quality checks. In the more complex cases, these checks do not fit the scope of the checks that a typical data source may support (i.e. PK, FK, CHECK), so these

Re: [DISCUSS] SPIP: Declarative Pipelines

Looks great! QQ: will user able to run this pipeline from normal code? I.e. can I trigger a pipeline from *driver* code based on some condition etc. or it must be executed via separate shell command ? As a background Databricks imposes similar limitation where as you cannot run normal Spark code an