Hi everyone,

Apologies if this is a basic question—I’ve searched around but haven’t
found a clear answer.

I'm currently developing a Spark application using Scala, and I’m looking
for a way to include all the JARs typically bundled in a standard Spark
installation as a single provided dependency.

>From what I’ve seen, most examples add each Spark module individually (like
spark-core, spark-sql, spark-mllib, etc.) as separate provided dependencies.
However, since these are all included in the Spark runtime environment, I’m
wondering why there isn’t a more aggregated dependency—something like a
parent project or BOM (Bill of Materials) that pulls in all the commonly
included Spark libraries (along with compatible versions of Log4j, Guava,
Jackson, and so on) - which is being used in projects.

Is there a particular reason this approach isn’t commonly used? Does it
cause issues with transitive dependencies or version mismatches? If so -
I'm sure those can be addressed as well...


Thanks in advance for any insights!


Best regards,

Nimrod

Reply via email to