Agreed, If the goal is to make Spark truly pluggable, the spark-submit tool itself should be more flexible in handling different cluster managers and their specific requirements.
1. Back in the days, Spark's initial development focused on a limited set of cluster managers (Standalone, YARN). 2. Implementing *a full pluggability for spark-submit *would require redesign and implementation to handle the diverse requirements of different cluster managers which I think will be a major project for itself HTH Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> On Fri, 7 Feb 2025 at 15:27, George J <georgej...@gmail.com> wrote: > To me, this seems like a gap in the "pluggable cluster manager" > implementation. > > What is the value of making cluster managers pluggable, if spark-submit > doesn't accept jobs on those cluster managers? > > It seems to me, for pluggable cluster managers to work, you would want > some parts of spark-submit to be "pluggable" as well. > > Thanks! > George Jahad > G-Research Developer >
