One key thing I forgot to mention is that I changed the avro version to 1.7.7
to get AVRO-1476.
I took a closer look at the jars, and what I noticed is that the assembly jars
that work do not have the org.apache.avro.mapreduce package packaged into the
assembly. For spark-1.0.1, org.apache.avro
Hmm. Fair enough. I hadn¹t given that answer much thought and on
reflection think you¹re right in that a profile would just be a bad hack.
On 8/4/14, 10:35, "Sean Owen" wrote:
>What would such a profile do though? In general building for a
>specific vendor version means setting hadoop.verison
What would such a profile do though? In general building for a
specific vendor version means setting hadoop.verison and/or
yarn.version. Any hard-coded value is unlikely to match what a
particular user needs. Setting protobuf versions and so on is already
done by the generic profiles.
In a similar
The profile does set it automatically:
https://github.com/apache/spark/blob/master/pom.xml#L1086
yarn.version should default to hadoop.version
It shouldn't hurt, and should work, to set to any other specific
version. If one HDP version works and another doesn't, are you sure
the repo has the desir
I don’t think there is an hwx profile, but there probably should be.
- Steve
From: Patrick Wendell
Date: Monday, August 4, 2014 at 10:08
To: Ron's Yahoo!
Cc: Ron's Yahoo! , Steve Nunez
, , "d...@spark.apache.org"
Subject: Re: Issues with HDP 2.4.0.2.1.3.0-563
Ah I
Ah I see, yeah you might need to set hadoop.version and yarn.version. I
thought he profile set this automatically.
On Mon, Aug 4, 2014 at 10:02 AM, Ron's Yahoo! wrote:
> I meant yarn and hadoop defaulted to 1.0.4 so the yarn build fails since
> 1.0.4 doesn't exist for yarn...
>
> Thanks,
> Ron
I meant yarn and hadoop defaulted to 1.0.4 so the yarn build fails since 1.0.4
doesn’t exist for yarn...
Thanks,
Ron
On Aug 4, 2014, at 10:01 AM, Ron's Yahoo! wrote:
> That failed since it defaulted the versions for yarn and hadoop
> I’ll give it a try with just 2.4.0 for both yarn and hadoop
That failed since it defaulted the versions for yarn and hadoop
I’ll give it a try with just 2.4.0 for both yarn and hadoop…
Thanks,
Ron
On Aug 4, 2014, at 9:44 AM, Patrick Wendell wrote:
> Can you try building without any of the special `hadoop.version` flags and
> just building only with -P
Can you try building without any of the special `hadoop.version` flags and
just building only with -Phadoop-2.4? In the past users have reported
issues trying to build random spot versions... I think HW is supposed to be
compatible with the normal 2.4.0 build.
On Mon, Aug 4, 2014 at 8:35 AM, Ron'
Thanks, I ensured that $SPARK_HOME/pom.xml had the HDP repository under the
repositories element. I also confirmed that if the build couldn’t find the
version, it would fail fast so it seems as if it’s able to get the versions it
needs to build the distribution.
I ran the following (generated fr
Provided you¹ve got the HWX repo in your pom.xml, you can build with this
line:
mvn -Pyarn -Phive -Phadoop-2.4 -Dhadoop.version=2.4.0.2.1.1.0-385
-DskipTests clean package
I haven¹t tried building a distro, but it should be similar.
- SteveN
On 8/4/14, 1:25, "Sean Owen" wrote:
>For a
For any Hadoop 2.4 distro, yes, set hadoop.version but also set
-Phadoop-2.4. http://spark.apache.org/docs/latest/building-with-maven.html
On Mon, Aug 4, 2014 at 9:15 AM, Patrick Wendell wrote:
> For hortonworks, I believe it should work to just link against the
> corresponding upstream version.
For hortonworks, I believe it should work to just link against the
corresponding upstream version. I.e. just set the Hadoop version to "2.4.0"
Does that work?
- Patrick
On Mon, Aug 4, 2014 at 12:13 AM, Ron's Yahoo!
wrote:
> Hi,
> Not sure whose issue this is, but if I run make-distribution
13 matches
Mail list logo
