Re: Re: Increasing Shading & Relocating for 4.0

The higher the level of abstraction, the less control and insight you typically have into its internal workings. If the goal is to create users rather than developers, Spark Connect is the right API to achieve that purpose. El dom, 19 ene 2025, 13:10, Mich Talebzadeh escribió: > I believe by act

Re: Re: Increasing Shading & Relocating for 4.0

I believe by actively involving the user community, we can create a more user-centric and successful path for the future of Spark in this respect. At the moment, the discussion is confined to this dev group but we ought to gather feedback from trenches so we can gain a sense of exit barriers and ti

RE: Re: Increasing Shading & Relocating for 4.0

On 2025/01/18 22:35:59 Mich Talebzadeh wrote: > I think your view highlights the need for a shift towards more stable and > version-independent APIs. Spark Connect IMO is a key enabler of this shift, > allowing users and developers to build applications and libraries that are > more resilient to ch

Re: Re: Increasing Shading & Relocating for 4.0

What about introducing isolated class loaders, similar to the approach used by web servers? Perhaps OSGi bundles or something similar? El sáb, 18 ene 2025, 22:43, Holden Karau escribió: > I would say the short answer is "mostly not" and the longer answer is that > the connect APIs are explicitly

Re: Re: Increasing Shading & Relocating for 4.0

I would say the short answer is "mostly not" and the longer answer is that the connect APIs are explicitly not covering many, what we would call, "paved paths." Because we're more likely to have JAR conflicts with advanced users who are more likely to use some of the non-supported APIs. For example

Re: Re: Increasing Shading & Relocating for 4.0

BTW, one of many reasons Spark Connect was developed was to potentially simplify this process around shading (i.e. not need to do it). I’m wondering if utilizing Spark Connect could be a potential solution here? On Fri, Jan 17, 2025 at 12:27 Holden Karau wrote: > +1 I think this is great. If

Re: Re: Increasing Shading & Relocating for 4.0

+1 I think this is great. If you’ve got any shading you’d be open to upstreaming I’d be happy to review it. Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ Books (Learning Spark, High Performa

Re: Re: Increasing Shading & Relocating for 4.0

Thanks for sharing the insightful context! On Fri, Jan 17, 2025 at 11:47 AM Regina Lee wrote: > Hi, > > I’d like to share insights from our Spark team at LinkedIn. We recently > moved to a mostly shaded Spark 3 client internally. Our goal was to > minimize dependency conflicts that could hinder

RE: Re: Increasing Shading & Relocating for 4.0

Hi, I’d like to share insights from our Spark team at LinkedIn. We recently moved to a mostly shaded Spark 3 client internally. Our goal was to minimize dependency conflicts that could hinder Spark upgrades, especially given our previous efforts to migrate our users from Spark 2 to Spark 3, an