Hi Ashika, Hadoop 2.6 is now no longer supported, and since it has not been maintained in the last 2 years, it means it may have some security issues unpatched. Spark 3.0 onwards, we no longer support it, in other words, we have modified our codebase in a way that Hadoop 2.6 won't work. However, if you are determined, you can always apply a custom patch to spark codebase and support it. I would recommend moving to newer Hadoop.
Thanks, On Mon, Jul 20, 2020 at 8:41 AM Ashika Umanga <ashika.uma...@gmail.com> wrote: > Greetings, > > Hadoop 2.6 has been removed according to this ticket > https://issues.apache.org/jira/browse/SPARK-25016 > > We run our Spark cluster on K8s in standalone mode. > We access HDFS/Hive running on a Hadoop 2.6 cluster. > We've been using Spark 2.4.5 and planning on upgrading to Spark 3.0.0 > However, we dont have any control over the Hadoop cluster and it will > remain in 2.6 > > Is Spark 3.0 still compatible with HDFS/Hive running on Hadoop 2.6 ? > > Best Regards, >
