we run it without issues on hadoop 2.6 - 2.8 on top of my head. we however do some post-processing on the tarball: 1) we fix the ownership of the files inside the tar.gz file (should be uid/gid 0/0, otherwise untarring by root can lead to ownership by unknown user). 2) add avro-1.8.2.jar and jline-2.14.6.jar to jars folder. i believe these jars missing in provided profile is simply a mistake.
best, koert On Mon, May 20, 2019 at 3:37 PM Michael Heuer <heue...@gmail.com> wrote: > Hello, > > Which Hadoop version or versions are compatible with Spark 2.4.3 and Scala > 2.12? > > The binary distribution spark-2.4.3-bin-without-hadoop-scala-2.12.tgz is > missing avro-1.8.2.jar, so when attempting to run with Hadoop 2.7.7 there > are classpath conflicts at runtime, as Hadoop 2.7.7 includes avro-1.7.4.jar. > > https://issues.apache.org/jira/browse/SPARK-27781 > > michael >
