Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/468#issuecomment-41770038
@darose this can be patched downstream, but that would not fix this for any
other distro. Ideally, the dependency is set to 0.9.0 when built against Hadoop
2.3.0+. As we've seen in other cases, it's possible to manage this with more
profiles in the build -- a PITA, but certainly possible. (I don't know if this
helps the SBT build though, but presumably some similar logic about the Hadoop
version is possible there.)
The funny thing is that this dependency is only needed at runtime. (Really
should be declared as <scope>runtime</scope>) I am still not sure why
hadoop-client doesn't package it. However, I wonder if, in the context of a
Hadoop cluster, it's going to be on the classpath anyway? and then it would be
the right version in all cases. What if you change the scope, so jets3t is not
even in the assembly?
I actually bet that works, and is simple. However I think it means that s3
would no longer work if running Spark by itself, so is probably a non-starter.
So, a new hadoop2.3.0 profile? that we try to trigger based on well-known
hadoop.version values?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
