Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/468#issuecomment-42102935
@pwendell Before I begin can I propose a refactoring of profiles that will
make this and similar issues easy to deal with? Probably it's for a different
PR, but will probably make this and similar changes easy.
We need profiles to deal with this. Profiles can be triggered explicitly
(e.g. `-Phadoop-2.3`) or by property values (`-Dhadoop.version=2.3.0`). It's
necessary to have things like `hadoop.version` be customizable, so it would be
nice to also trigger needed profiles from this. However, Maven lacks ability to
trigger on a range of property values; you can trigger on a particular value
like "2.3.0" but not "2.3.*" or "[2.3.0,2.4.0)" syntax.
So it seems necessary to use a series of named profiles. Those profiles can
set default version values, and those versions can be overridden. For example,
it's nice to have a `hadoop-2.3` profile set `hadoop.version=2.3.0` for you,
even though that can still be overridden.
(The SBT build can shadow these changes.)
After reading over the build and docs, I propose the following:
- Introduce a `hadoop-2.3` profile, similar to `hadoop-0.23`, to encompass
2.3+-specific build changes, and one for `hadoop-2.2` as well (see later)
- `hadoop.major.version` appears to be unused -- remove it?
- I believe `yarn.version` can be removed; use `hadoop.version` in its
place. Ideally these are always synced, no? All doc examples show
`yarn.version` matching `hadoop.version` and the distribution script uses
`SPARK_HADOOP_VERSION` for `yarn.version`. Now, the default Hadoop version is
1.0.4 and there is no such YARN version. But the `yarn-alpha` profile sets
`hadoop.version=0.23.7` to match the default `yarn.version=0.23.7` anyway. It
seems like Hadoop 1.x + YARN is not intended anyway, which seems corroborated
by the build documentation.
- So, YARN-related profiles should not set `hadoop.version`, and in fact
only serve to add the `yarn` child module
... and then the fix for this issue is trivial.
All of the build permutations listed in the documentation work under this
new arrangement. Anyone want to see a PR or have objections?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
