ok, so let's start: what the easy bits we do to make life easier for people
trying to use the hadoop file:// client on a windows box, which essentially
means "people trying to use spark standalone?"
torg.apache.hadoop.util.Shell is the obvious first target. i had a go at
getting it fail more meani
As Chris mentioned earlier, it would be wise to do this in pieces that can
be reviewed properly.
Bringing large refactorings in all at once, as Garret mentioned, is not
likely to just get a +1.
We do have a feature branch process and criteria and we could determine
specific criteria for such a cri
On 11/29/2022 8:16 AM, Gautham Banasandra wrote:
…
However, I don't see anyone stopping you from working on removing
winutils. I encourage you to put across a PR and I would be glad to review
the same.
That's not how it works. This is an intense undertaking. If I spend six
months with no incom
Hi Steve,
I feel we won't have the confidence with major changes until we have
a CI that runs the full suite of tests.
Thanks,
--Gautham
On Tue, 29 Nov 2022 at 16:12, Steve Loughran wrote:
> he's got a point.
>
> what is to stop us cutting it from trunk?
>
> On Mon, 28 Nov 2022 at 18:12, Garr
Hi Garret,
My responses hitherto indicate that I'm currently focusing on setting up
a CI for Windows for building Hadoop and getting the unit tests to run on
it.
I believe, without these, no matter the improvements, the build is bound
to be broken someday. I would thus like to prioritize setting u
he's got a point.
what is to stop us cutting it from trunk?
On Mon, 28 Nov 2022 at 18:12, Garret Wilson wrote:
> On 11/14/2022 12:04 PM, Gautham Banasandra wrote:
>
> …
> I would like to aim for winutils deprecation by the end of the first
> quarter of 2023.
>
> The key word is "deprecation".
On 11/14/2022 12:04 PM, Gautham Banasandra wrote:
…
I would like to aim for winutils deprecation by the end of the first
quarter of 2023.
The key word is "deprecation". Note that the sentence doesn't say the
references in the code to Winutils will be removed by this time. I
offered to remove
Hi Steve,
There are quite a few unit test failures on Windows. Some of them are major
ones.
I feel we need to fix those before we do a release. I'm trying to focus on
setting up the CI
so that we can incrementally fix the unit tests and avoid any possible
regressions.
Thanks,
--Gautham
On Tue, 1
you up for doing a winutils build on the 3.3.5 release? i'm going to do the
arm64 binaries
On Mon, 14 Nov 2022 at 18:08, Gautham Banasandra wrote:
> Also, I plan to do a Windows release once I setup the CI for Windows and
> after I get
> the major unit tests to pass. It would still contain winut
Also, I plan to do a Windows release once I setup the CI for Windows and
after I get
the major unit tests to pass. It would still contain winutils though.
However, we can do
another release after deprecating winutils.
Thanks,
--Gautham
On Mon, 14 Nov 2022 at 23:34, Gautham Banasandra wrote:
> H
Hi Iñigo,
I would like to aim for winutils deprecation by the end of the first
quarter of 2023.
It really depends on how fast I can wrap up with setting up CI for Windows.
Given
that this involves getting Yetus to work properly on Windows, I feel it's a
bit
ambitious. But if things fall into place
Somewhat related, I'd love to see the overall JNI footprint go down where
possible. For example, HADOOP-17895 proposes switching to the newer JDK
APIs for chmod.
Chris Nauroth
On Fri, Nov 11, 2022 at 10:50 AM Iñigo Goiri wrote:
> Gautham, thank you very much for the summary.
> Do you have a ti
Gautham, thank you very much for the summary.
Do you have a time-line for when we can get rid of winutils?
My idea was to get this and the YARN federation hardening work into a 3.4
release.
On Fri, Nov 11, 2022, 10:15 Gautham Banasandra wrote:
> Hi folks,
>
>
> What have we done so far?
>
Hi folks,
What have we done so far?
Inigo and I have been working for quite some time now on this topic,
but our efforts have mostly been oriented towards making Hadoop
cross-platform compatible. Our focus has been on streamlining the
process of building Hadoo
It's time to reach for the axe.
We haven't shipped eight version of Apache hadoop which builds and runs on
windows for a long long time. I the only people trying to use the library
is on windows Will have been people trying to use spark on their laptops
with "small" dataset of only a are few tens
Symlink support on the local file system is still used. One example I can
think of is YARN container launch [1].
I would welcome removal of winutils, as already described in various JIRA
issues. I think the biggest challenge we'll have is testing of a transition
from winutils to the newer Java API
>
>
>
> * Bare Naked Local File System v0.1.0 doesn't (yet) support symlinks
> or the sticky bit.
>
ok to not support symlinks. The symlinks of HDFS are not being maintained
and I am not aware of anything relying on it.
So I assume people don't need it.
Sticky bit would be useful, I guess.
17 matches
Mail list logo