This External Cluster Manager is an amazing concept and I really like the separation.
Would it be possible to include a broader group and discuss an approach on how to make Spark more pluggable? It is a bit far fetched but we would be very much interested in working on this if this resonates well within the Spark community. On Fri, Feb 7, 2025 at 4:45 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Agreed, If the goal is to make Spark truly pluggable, the spark-submit > tool itself should be more flexible in handling different cluster managers > and their specific requirements. > > 1. Back in the days, Spark's initial development focused on a limited > set of cluster managers (Standalone, YARN). > 2. Implementing *a full pluggability for spark-submit *would require > redesign and implementation to handle the diverse requirements of different > cluster managers which I think will be a major project for itself > > HTH > > Dr Mich Talebzadeh, > Architect | Data Science | Financial Crime | Forensic Analysis | GDPR > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > > > On Fri, 7 Feb 2025 at 15:27, George J <georgej...@gmail.com> wrote: > >> To me, this seems like a gap in the "pluggable cluster manager" >> implementation. >> >> What is the value of making cluster managers pluggable, if spark-submit >> doesn't accept jobs on those cluster managers? >> >> It seems to me, for pluggable cluster managers to work, you would want >> some parts of spark-submit to be "pluggable" as well. >> >> Thanks! >> George Jahad >> G-Research Developer >> >
