I solved this issue by using spark 2.3.1 jars copied from the HDP3.0
cluster. Thanks.
On Thu, Aug 30, 2018 at 10:18 AM Lian Jiang wrote:
> Per https://spark.apache.org/docs/latest/building-spark.html, spark 2.3.1
> is built with hadoop 2.6.X by default. This is why I see my fat jar
> includes ha
Per https://spark.apache.org/docs/latest/building-spark.html, spark 2.3.1
is built with hadoop 2.6.X by default. This is why I see my fat jar
includes hadoop 2.6.5 (instead of 3.1.0) jars. HftpFileSystem has been
removed in hadoop 3.
On https://spark.apache.org/downloads.html, I only see spark 2.3
