Hi Steve, Yes, there won't be a need for winutils.exe once we provide a JNI implementation under a common file system interface.
Thanks, --Gautham On Fri, 29 Apr 2022 at 00:54, Steve Loughran <ste...@cloudera.com> wrote: > > > On Sun, 20 Feb 2022 at 18:42, Gautham Banasandra <gaur...@apache.org> > wrote: > >> Hi all, >> >> I've been working on getting Hadoop to build on Windows for quite some >> time >> now. We're now at a stage where we can parallelize the effort and complete >> this sooner. I've outlined the parts that are remaining. Please get in >> touch with me if anyone wishes to join hands in realizing this goal. >> >> *Why do we need Hadoop to run on Windows?* >> Windows has a very large user base. The modern alternative softwares to >> Hadoop (like Kubernetes) are cross platform by design. We have to >> acknowledge the fact it isn't easy to get Hadoop running on Windows. The >> reason why we haven't seen much adoption of Hadoop on Windows is probably >> because of issues like compilation, requiring work-arounds every step of >> the way etc. If we were to nail these issues, I believe it would >> tremendously expand the usage of Hadoop. >> >> >> *Phase 3 : Resolving systemic issues* >> 1. [HADOOP-13223] winutils.exe is a bug nexus and should be killed with an >> axe. - ASF JIRA (apache.org) >> <https://issues.apache.org/jira/browse/HADOOP-13223> >> The Hadoop environment is modeled closer to that of Linux than Windows. >> Thus, we see a lot of functional gaps between running Hadoop on Linux v/s >> Windows, which have become the source of bugs when it comes to running >> Hadoop on Windows. One such issue is that of winutils.exe. We can aim to >> address issues like these in this phase. I plan to provide JNI >> implementation for each platform and unify these under a common file >> system >> interface. So that we get stack traces for exceptions thrown in these >> layers and mostly so that we don't have any disparity between the >> platforms. >> >> > i for one endorse this jira. > > given a lot of it is for fs permissions, maybe whatever you do can > downgrade, so that running spark local on a windows laptop becomes easy. > those people do not need the posix permissions model > >
