Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

[Tom Graves]

Thanks for extending the voting.

Unfortunately I've found an issue with the spark-shell in yarn-client mode.  It 
doesn't work with secure HDFS unless you 
export SPARK_YARN_MODE=true before starting the shell, or if you happen to do 
something immediately with HDFS.  If you wait for the connection to the 
namenode to timeout it will fail. 

I think it was actually this way in the 0.9 release also so I thought I would 
send this and get peoples feedback to see if you want it fixed? 

Another option would be to document that you have to export 
SPARK_YARN_MODE=true for the shell.   The fix actually went in with the 
authentication changes I made in master but I never realized that change needed 
to apply to 0.9. 

https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0

See the SparkILoop diff.

Tom


On Monday, March 31, 2014 1:33 PM, Tathagata Das  
wrote:
 
Yes, lets extend the vote for two more days from now. So the vote is open till 
Wednesday, April 02, at 20:00 UTC

On that note, my +1


TD






On Mon, Mar 31, 2014 at 9:57 AM, Patrick Wendell  wrote:

Yeah good point. Let's just extend this vote another few days?
>
>
>
>On Mon, Mar 31, 2014 at 8:12 AM, Tom Graves  wrote:
>
>> I should probably pull this off into another thread, but going forward can
>> we try to not have the release votes end on a weekend? Since we only seem
>> to give 3 days, it makes it really hard for anyone who is offline for the
>> weekend to try it out.   Either that or extend the voting for more then 3
>> days.
>>
>> Tom
>> On Monday, March 31, 2014 12:50 AM, Patrick Wendell 
>> wrote:
>>
>> TD - I downloaded and did some local testing. Looks good to me!
>>
>> +1
>>
>> You should cast your own vote - at that point it's enough to pass.
>>
>> - Patrick
>>
>>
>>
>> On Sun, Mar 30, 2014 at 9:47 PM, prabeesh k  wrote:
>>
>> > +1
>> > tested on Ubuntu12.04 64bit
>> >
>> >
>> > On Mon, Mar 31, 2014 at 3:56 AM, Matei Zaharia > > >wrote:
>> >
>> > > +1 tested on Mac OS X.
>> > >
>> > > Matei
>> > >
>> > > On Mar 27, 2014, at 1:32 AM, Tathagata Das <
>> tathagata.das1...@gmail.com>
>> > > wrote:
>> > >
>> > > > Please vote on releasing the following candidate as Apache Spark
>> > version
>> > > 0.9.1
>> > > >
>> > > > A draft of the release notes along with the CHANGES.txt file is
>> > > > attached to this e-mail.
>> > > >
>> > > > The tag to be voted on is v0.9.1-rc3 (commit 4c43182b):
>> > > >
>> > >
>> >
>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=4c43182b6d1b0b7717423f386c0214fe93073208
>> > > >
>> > > > The release files, including signatures, digests, etc. can be found
>> at:
>> > > > http://people.apache.org/~tdas/spark-0.9.1-rc3/
>> > > >
>> > > > Release artifacts are signed with the following key:
>> > > > https://people.apache.org/keys/committer/tdas.asc
>> > > >
>> > > > The staging repository for this release can be found at:
>> > > >
>> > https://repository.apache.org/content/repositories/orgapachespark-1009/
>> > > >
>> > > > The documentation corresponding to this release can be found at:
>> > > > http://people.apache.org/~tdas/spark-0.9.1-rc3-docs/
>> > > >
>> > > > Please vote on releasing this package as Apache Spark 0.9.1!
>> > > >
>> > > > The vote is open until Sunday, March 30, at 10:00 UTC and passes if
>> > > > a majority of at least 3 +1 PMC votes are cast.
>> > > >
>> > > > [ ] +1 Release this package as Apache Spark 0.9.1
>> > > > [ ] -1 Do not release this package because ...
>> > > >
>> > > > To learn more about Apache Spark, please see
>> > > > http://spark.apache.org/
>> > > > 
>> > >
>> > >
>> >
>>
>

sbt-package-bin

[Evan Chan]

Hey folks,

We are in the middle of creating a Chef recipe for Spark.   As part of that
we want to create a Debian package for Spark.

What do folks think of adding the sbt-package-bin plugin to allow easy
creation of a Spark .deb file?  I believe it adds all dependency jars into
a single lib/ folder, so in some ways it's even easier to manage than the
assembly.

Also I'm not sure if there's an equivalent plugin for Maven.

thanks,
Evan


-- 
--
Evan Chan
Staff Engineer
e...@ooyala.com  |

Re: sbt-package-bin

[Evan Chan]

Also, I understand this is the last week / merge window for 1.0, so if
folks are interested I'd like to get in a PR quickly.

thanks,
Evan



On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:

> Hey folks,
>
> We are in the middle of creating a Chef recipe for Spark.   As part of
> that we want to create a Debian package for Spark.
>
> What do folks think of adding the sbt-package-bin plugin to allow easy
> creation of a Spark .deb file?  I believe it adds all dependency jars into
> a single lib/ folder, so in some ways it's even easier to manage than the
> assembly.
>
> Also I'm not sure if there's an equivalent plugin for Maven.
>
> thanks,
> Evan
>
>
> --
> --
>  Evan Chan
> Staff Engineer
> e...@ooyala.com  |
>
>  
> 
>
>


-- 
--
Evan Chan
Staff Engineer
e...@ooyala.com  |

Re: sbt-package-bin

[Mark Hamstra]

A basic Debian package can already be created from the Maven build: mvn
-Pdeb ...


On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:

> Also, I understand this is the last week / merge window for 1.0, so if
> folks are interested I'd like to get in a PR quickly.
>
> thanks,
> Evan
>
>
>
> On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
>
> > Hey folks,
> >
> > We are in the middle of creating a Chef recipe for Spark.   As part of
> > that we want to create a Debian package for Spark.
> >
> > What do folks think of adding the sbt-package-bin plugin to allow easy
> > creation of a Spark .deb file?  I believe it adds all dependency jars
> into
> > a single lib/ folder, so in some ways it's even easier to manage than the
> > assembly.
> >
> > Also I'm not sure if there's an equivalent plugin for Maven.
> >
> > thanks,
> > Evan
> >
> >
> > --
> > --
> >  Evan Chan
> > Staff Engineer
> > e...@ooyala.com  |
> >
> >  <
> http://www.linkedin.com/company/ooyala>
> >
> >
>
>
> --
> --
> Evan Chan
> Staff Engineer
> e...@ooyala.com  |
>
> 
> <
> http://www.twitter.com/ooyala>
>

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

[Patrick Wendell]

Tom,

Given this is a pretty straightforward workaround, what do yo think about
the following course of action:

(a) We can put the workaround in the docs for 0.9.1. We don't need to do a
new RC/vote for this since we can update the published docs independently.

(b) We try to get a fix in for this into the 0.9 branch so it can end up in
0.9.2. But this takes the fix off the critical path for this release.

- Patrick


On Tue, Apr 1, 2014 at 7:28 AM, Tom Graves  wrote:

> Thanks for extending the voting.
>
> Unfortunately I've found an issue with the spark-shell in yarn-client
> mode.  It doesn't work with secure HDFS unless you
> export SPARK_YARN_MODE=true before starting the shell, or if you happen to
> do something immediately with HDFS.  If you wait for the connection to the
> namenode to timeout it will fail.
>
> I think it was actually this way in the 0.9 release also so I thought I
> would send this and get peoples feedback to see if you want it fixed?
>
> Another option would be to document that you have to export
> SPARK_YARN_MODE=true for the shell.   The fix actually went in with the
> authentication changes I made in master but I never realized that change
> needed to apply to 0.9.
>
>
> https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
>
> See the SparkILoop diff.
>
> Tom
>
>
> On Monday, March 31, 2014 1:33 PM, Tathagata Das <
> tathagata.das1...@gmail.com> wrote:
>
> Yes, lets extend the vote for two more days from now. So the vote is open
> till Wednesday, April 02, at 20:00 UTC
>
> On that note, my +1
>
>
> TD
>
>
>
>
>
>
> On Mon, Mar 31, 2014 at 9:57 AM, Patrick Wendell 
> wrote:
>
> Yeah good point. Let's just extend this vote another few days?
> >
> >
> >
> >On Mon, Mar 31, 2014 at 8:12 AM, Tom Graves  wrote:
> >
> >> I should probably pull this off into another thread, but going forward
> can
> >> we try to not have the release votes end on a weekend? Since we only
> seem
> >> to give 3 days, it makes it really hard for anyone who is offline for
> the
> >> weekend to try it out.   Either that or extend the voting for more then
> 3
> >> days.
> >>
> >> Tom
> >> On Monday, March 31, 2014 12:50 AM, Patrick Wendell  >
> >> wrote:
> >>
> >> TD - I downloaded and did some local testing. Looks good to me!
> >>
> >> +1
> >>
> >> You should cast your own vote - at that point it's enough to pass.
> >>
> >> - Patrick
> >>
> >>
> >>
> >> On Sun, Mar 30, 2014 at 9:47 PM, prabeesh k 
> wrote:
> >>
> >> > +1
> >> > tested on Ubuntu12.04 64bit
> >> >
> >> >
> >> > On Mon, Mar 31, 2014 at 3:56 AM, Matei Zaharia <
> matei.zaha...@gmail.com
> >> > >wrote:
> >> >
> >> > > +1 tested on Mac OS X.
> >> > >
> >> > > Matei
> >> > >
> >> > > On Mar 27, 2014, at 1:32 AM, Tathagata Das <
> >> tathagata.das1...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Please vote on releasing the following candidate as Apache Spark
> >> > version
> >> > > 0.9.1
> >> > > >
> >> > > > A draft of the release notes along with the CHANGES.txt file is
> >> > > > attached to this e-mail.
> >> > > >
> >> > > > The tag to be voted on is v0.9.1-rc3 (commit 4c43182b):
> >> > > >
> >> > >
> >> >
> >>
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=4c43182b6d1b0b7717423f386c0214fe93073208
> >> > > >
> >> > > > The release files, including signatures, digests, etc. can be
> found
> >> at:
> >> > > > http://people.apache.org/~tdas/spark-0.9.1-rc3/
> >> > > >
> >> > > > Release artifacts are signed with the following key:
> >> > > > https://people.apache.org/keys/committer/tdas.asc
> >> > > >
> >> > > > The staging repository for this release can be found at:
> >> > > >
> >> >
> https://repository.apache.org/content/repositories/orgapachespark-1009/
> >> > > >
> >> > > > The documentation corresponding to this release can be found at:
> >> > > > http://people.apache.org/~tdas/spark-0.9.1-rc3-docs/
> >> > > >
> >> > > > Please vote on releasing this package as Apache Spark 0.9.1!
> >> > > >
> >> > > > The vote is open until Sunday, March 30, at 10:00 UTC and passes
> if
> >> > > > a majority of at least 3 +1 PMC votes are cast.
> >> > > >
> >> > > > [ ] +1 Release this package as Apache Spark 0.9.1
> >> > > > [ ] -1 Do not release this package because ...
> >> > > >
> >> > > > To learn more about Apache Spark, please see
> >> > > > http://spark.apache.org/
> >> > > > 
> >> > >
> >> > >
> >> >
> >>
> >
>

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

[Tom Graves]

No one else has reported seeing the issue so I think documenting it is fine.  

Tom


On Tuesday, April 1, 2014 1:40 PM, Patrick Wendell  wrote:
 
Tom,

Given this is a pretty straightforward workaround, what do yo think about
the following course of action:

(a) We can put the workaround in the docs for 0.9.1. We don't need to do a
new RC/vote for this since we can update the published docs independently.

(b) We try to get a fix in for this into the 0.9 branch so it can end up in
0.9.2. But this takes the fix off the critical path for this release.

- Patrick



On Tue, Apr 1, 2014 at 7:28 AM, Tom Graves  wrote:

> Thanks for extending the voting.
>
> Unfortunately I've found an issue with the spark-shell in yarn-client
> mode.  It doesn't work with secure HDFS unless you
> export SPARK_YARN_MODE=true before starting the shell, or if you happen to
> do something immediately with HDFS.  If you wait for the connection to the
> namenode to timeout it will fail.
>
> I think it was actually this way in the 0.9 release also so I thought I
> would send this and get peoples feedback to see if you want it fixed?
>
> Another option would be to document that you have to export
> SPARK_YARN_MODE=true for the shell.   The fix actually went in with the
> authentication changes I made in master but I never realized that change
> needed to apply to 0.9.
>
>
> https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
>
> See the SparkILoop diff.
>
> Tom
>
>
> On Monday, March 31, 2014 1:33 PM, Tathagata Das <
> tathagata.das1...@gmail.com> wrote:
>
> Yes, lets extend the vote for two more days from now. So the vote is open
> till Wednesday, April 02, at 20:00 UTC
>
> On that note, my +1
>
>
> TD
>
>
>
>
>
>
> On Mon, Mar 31, 2014 at 9:57 AM, Patrick Wendell 
> wrote:
>
> Yeah good point. Let's just extend this vote another few days?
> >
> >
> >
> >On Mon, Mar 31, 2014 at 8:12 AM, Tom Graves  wrote:
> >
> >> I should probably pull this off into another thread, but going forward
> can
> >> we try to not have the release votes end on a weekend? Since we only
> seem
> >> to give 3 days, it makes it really hard for anyone who is offline for
> the
> >> weekend to try it out.   Either that or extend the voting for more then
> 3
> >> days.
> >>
> >> Tom
> >> On Monday, March 31, 2014 12:50 AM, Patrick Wendell  >
> >> wrote:
> >>
> >> TD - I downloaded and did some local testing. Looks good to me!
> >>
> >> +1
> >>
> >> You should cast your own vote - at that point it's enough to pass.
> >>
> >> - Patrick
> >>
> >>
> >>
> >> On Sun, Mar 30, 2014 at 9:47 PM, prabeesh k 
> wrote:
> >>
> >> > +1
> >> > tested on Ubuntu12.04 64bit
> >> >
> >> >
> >> > On Mon, Mar 31, 2014 at 3:56 AM, Matei Zaharia <
> matei.zaha...@gmail.com
> >> > >wrote:
> >> >
> >> > > +1 tested on Mac OS X.
> >> > >
> >> > > Matei
> >> > >
> >> > > On Mar 27, 2014, at 1:32 AM, Tathagata Das <
> >> tathagata.das1...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Please vote on releasing the following candidate as Apache Spark
> >> > version
> >> > > 0.9.1
> >> > > >
> >> > > > A draft of the release notes along with the CHANGES.txt file is
> >> > > > attached to this e-mail.
> >> > > >
> >> > > > The tag to be voted on is v0.9.1-rc3 (commit 4c43182b):
> >> > > >
> >> > >
> >> >
> >>
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=4c43182b6d1b0b7717423f386c0214fe93073208
> >> > > >
> >> > > > The release files, including signatures, digests, etc. can be
> found
> >> at:
> >> > > > http://people.apache.org/~tdas/spark-0.9.1-rc3/
> >> > > >
> >> > > > Release artifacts are signed with the following key:
> >> > > > https://people.apache.org/keys/committer/tdas.asc
> >> > > >
> >> > > > The staging repository for this release can be found at:
> >> > > >
> >> >
> https://repository.apache.org/content/repositories/orgapachespark-1009/
> >> > > >
> >> > > > The documentation corresponding to this release can be found at:
> >> > > > http://people.apache.org/~tdas/spark-0.9.1-rc3-docs/
> >> > > >
> >> > > > Please vote on releasing this package as Apache Spark 0.9.1!
> >> > > >
> >> > > > The vote is open until Sunday, March 30, at 10:00 UTC and passes
> if
> >> > > > a majority of at least 3 +1 PMC votes are cast.
> >> > > >
> >> > > > [ ] +1 Release this package as Apache Spark 0.9.1
> >> > > > [ ] -1 Do not release this package because ...
> >> > > >
> >> > > > To learn more about Apache Spark, please see
> >> > > > http://spark.apache.org/
> >> > > > 
> >> > >
> >> > >
> >> >
> >>
> >
>

Re: sbt-package-bin

[Patrick Wendell]

And there is a deb target as well - ah didn't see Mark's email.


On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell  wrote:

> Ya there is already some fragmentation here. Maven has some "dist" targets
> and there is also ./make-distribution.sh.
>
>
> On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra wrote:
>
>> A basic Debian package can already be created from the Maven build: mvn
>> -Pdeb ...
>>
>>
>> On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
>>
>> > Also, I understand this is the last week / merge window for 1.0, so if
>> > folks are interested I'd like to get in a PR quickly.
>> >
>> > thanks,
>> > Evan
>> >
>> >
>> >
>> > On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
>> >
>> > > Hey folks,
>> > >
>> > > We are in the middle of creating a Chef recipe for Spark.   As part of
>> > > that we want to create a Debian package for Spark.
>> > >
>> > > What do folks think of adding the sbt-package-bin plugin to allow easy
>> > > creation of a Spark .deb file?  I believe it adds all dependency jars
>> > into
>> > > a single lib/ folder, so in some ways it's even easier to manage than
>> the
>> > > assembly.
>> > >
>> > > Also I'm not sure if there's an equivalent plugin for Maven.
>> > >
>> > > thanks,
>> > > Evan
>> > >
>> > >
>> > > --
>> > > --
>> > >  Evan Chan
>> > > Staff Engineer
>> > > e...@ooyala.com  |
>> > >
>> > >  <
>> > http://www.linkedin.com/company/ooyala>
>> > >
>> > >
>> >
>> >
>> > --
>> > --
>> > Evan Chan
>> > Staff Engineer
>> > e...@ooyala.com  |
>> >
>> > 
>> > > ><
>> > http://www.twitter.com/ooyala>
>> >
>>
>
>

Re: sbt-package-bin

[Patrick Wendell]

Ya there is already some fragmentation here. Maven has some "dist" targets
and there is also ./make-distribution.sh.


On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra wrote:

> A basic Debian package can already be created from the Maven build: mvn
> -Pdeb ...
>
>
> On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
>
> > Also, I understand this is the last week / merge window for 1.0, so if
> > folks are interested I'd like to get in a PR quickly.
> >
> > thanks,
> > Evan
> >
> >
> >
> > On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
> >
> > > Hey folks,
> > >
> > > We are in the middle of creating a Chef recipe for Spark.   As part of
> > > that we want to create a Debian package for Spark.
> > >
> > > What do folks think of adding the sbt-package-bin plugin to allow easy
> > > creation of a Spark .deb file?  I believe it adds all dependency jars
> > into
> > > a single lib/ folder, so in some ways it's even easier to manage than
> the
> > > assembly.
> > >
> > > Also I'm not sure if there's an equivalent plugin for Maven.
> > >
> > > thanks,
> > > Evan
> > >
> > >
> > > --
> > > --
> > >  Evan Chan
> > > Staff Engineer
> > > e...@ooyala.com  |
> > >
> > >  <
> > http://www.linkedin.com/company/ooyala>
> > >
> > >
> >
> >
> > --
> > --
> > Evan Chan
> > Staff Engineer
> > e...@ooyala.com  |
> >
> > 
> >  ><
> > http://www.twitter.com/ooyala>
> >
>

Re: sbt-package-bin

[Evan Chan]

Mark - sorry, would you mind expanding what the "" is?

Something like

mvn -Pdeb package

?

I get:

[ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of
its dependencies could not be resolved: Failed to read artifact descriptor
for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not find
artifact org.apache:apache:pom:13 -> [Help 1]


On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell  wrote:

> And there is a deb target as well - ah didn't see Mark's email.
>
>
> On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell 
> wrote:
>
> > Ya there is already some fragmentation here. Maven has some "dist"
> targets
> > and there is also ./make-distribution.sh.
> >
> >
> > On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra  >wrote:
> >
> >> A basic Debian package can already be created from the Maven build: mvn
> >> -Pdeb ...
> >>
> >>
> >> On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
> >>
> >> > Also, I understand this is the last week / merge window for 1.0, so if
> >> > folks are interested I'd like to get in a PR quickly.
> >> >
> >> > thanks,
> >> > Evan
> >> >
> >> >
> >> >
> >> > On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
> >> >
> >> > > Hey folks,
> >> > >
> >> > > We are in the middle of creating a Chef recipe for Spark.   As part
> of
> >> > > that we want to create a Debian package for Spark.
> >> > >
> >> > > What do folks think of adding the sbt-package-bin plugin to allow
> easy
> >> > > creation of a Spark .deb file?  I believe it adds all dependency
> jars
> >> > into
> >> > > a single lib/ folder, so in some ways it's even easier to manage
> than
> >> the
> >> > > assembly.
> >> > >
> >> > > Also I'm not sure if there's an equivalent plugin for Maven.
> >> > >
> >> > > thanks,
> >> > > Evan
> >> > >
> >> > >
> >> > > --
> >> > > --
> >> > >  Evan Chan
> >> > > Staff Engineer
> >> > > e...@ooyala.com  |
> >> > >
> >> > >  <
> >> > http://www.linkedin.com/company/ooyala> >
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> > --
> >> > Evan Chan
> >> > Staff Engineer
> >> > e...@ooyala.com  |
> >> >
> >> > 
> >> > <
> http://www.linkedin.com/company/ooyala
> >> ><
> >> > http://www.twitter.com/ooyala>
> >> >
> >>
> >
> >
>



-- 
--
Evan Chan
Staff Engineer
e...@ooyala.com  |

Re: sbt-package-bin

[Mark Hamstra]

What the "..." is kind of depends on what you're trying to accomplish.  You
could be setting Hadoop version and other stuff in there, but if you go too
much beyond a pretty basic build, you're probably also going to have to
modify the  of the jdeb plugin in assembly/pom.xml to
include/exclude just what you want/don't want in the Debian package.

Anyway, a typical build would look something like 'mvn -U -Pdeb -DskipTests
clean package', after which you should be able to find your .deb in
assembly/target.


On Tue, Apr 1, 2014 at 3:01 PM, Evan Chan  wrote:

> Mark - sorry, would you mind expanding what the "" is?
>
> Something like
>
> mvn -Pdeb package
>
> ?
>
> I get:
>
> [ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of
> its dependencies could not be resolved: Failed to read artifact descriptor
> for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not find
> artifact org.apache:apache:pom:13 -> [Help 1]
>
>
> On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell 
> wrote:
>
> > And there is a deb target as well - ah didn't see Mark's email.
> >
> >
> > On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell 
> > wrote:
> >
> > > Ya there is already some fragmentation here. Maven has some "dist"
> > targets
> > > and there is also ./make-distribution.sh.
> > >
> > >
> > > On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra  > >wrote:
> > >
> > >> A basic Debian package can already be created from the Maven build:
> mvn
> > >> -Pdeb ...
> > >>
> > >>
> > >> On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
> > >>
> > >> > Also, I understand this is the last week / merge window for 1.0, so
> if
> > >> > folks are interested I'd like to get in a PR quickly.
> > >> >
> > >> > thanks,
> > >> > Evan
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
> > >> >
> > >> > > Hey folks,
> > >> > >
> > >> > > We are in the middle of creating a Chef recipe for Spark.   As
> part
> > of
> > >> > > that we want to create a Debian package for Spark.
> > >> > >
> > >> > > What do folks think of adding the sbt-package-bin plugin to allow
> > easy
> > >> > > creation of a Spark .deb file?  I believe it adds all dependency
> > jars
> > >> > into
> > >> > > a single lib/ folder, so in some ways it's even easier to manage
> > than
> > >> the
> > >> > > assembly.
> > >> > >
> > >> > > Also I'm not sure if there's an equivalent plugin for Maven.
> > >> > >
> > >> > > thanks,
> > >> > > Evan
> > >> > >
> > >> > >
> > >> > > --
> > >> > > --
> > >> > >  Evan Chan
> > >> > > Staff Engineer
> > >> > > e...@ooyala.com  |
> > >> > >
> > >> > >  <
> > >> > http://www.linkedin.com/company/ooyala><
> http://www.twitter.com/ooyala
> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > --
> > >> > Evan Chan
> > >> > Staff Engineer
> > >> > e...@ooyala.com  |
> > >> >
> > >> > 
> > >> > <
> > http://www.linkedin.com/company/ooyala
> > >> ><
> > >> > http://www.twitter.com/ooyala>
> > >> >
> > >>
> > >
> > >
> >
>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> e...@ooyala.com  |
>
> 
> <
> http://www.twitter.com/ooyala>
>

Re: sbt-package-bin

[Mark Hamstra]

...or at least you could do that if the Maven build wasn't broken right now.


On Tue, Apr 1, 2014 at 6:01 PM, Mark Hamstra wrote:

> What the "..." is kind of depends on what you're trying to accomplish.
>  You could be setting Hadoop version and other stuff in there, but if you
> go too much beyond a pretty basic build, you're probably also going to have
> to modify the  of the jdeb plugin in assembly/pom.xml to
> include/exclude just what you want/don't want in the Debian package.
>
> Anyway, a typical build would look something like 'mvn -U -Pdeb
> -DskipTests clean package', after which you should be able to find your
> .deb in assembly/target.
>
>
> On Tue, Apr 1, 2014 at 3:01 PM, Evan Chan  wrote:
>
>> Mark - sorry, would you mind expanding what the "" is?
>>
>> Something like
>>
>> mvn -Pdeb package
>>
>> ?
>>
>> I get:
>>
>> [ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one
>> of
>> its dependencies could not be resolved: Failed to read artifact descriptor
>> for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not find
>> artifact org.apache:apache:pom:13 -> [Help 1]
>>
>>
>> On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell 
>> wrote:
>>
>> > And there is a deb target as well - ah didn't see Mark's email.
>> >
>> >
>> > On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell 
>> > wrote:
>> >
>> > > Ya there is already some fragmentation here. Maven has some "dist"
>> > targets
>> > > and there is also ./make-distribution.sh.
>> > >
>> > >
>> > > On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra <
>> m...@clearstorydata.com
>> > >wrote:
>> > >
>> > >> A basic Debian package can already be created from the Maven build:
>> mvn
>> > >> -Pdeb ...
>> > >>
>> > >>
>> > >> On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
>> > >>
>> > >> > Also, I understand this is the last week / merge window for 1.0,
>> so if
>> > >> > folks are interested I'd like to get in a PR quickly.
>> > >> >
>> > >> > thanks,
>> > >> > Evan
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
>> > >> >
>> > >> > > Hey folks,
>> > >> > >
>> > >> > > We are in the middle of creating a Chef recipe for Spark.   As
>> part
>> > of
>> > >> > > that we want to create a Debian package for Spark.
>> > >> > >
>> > >> > > What do folks think of adding the sbt-package-bin plugin to allow
>> > easy
>> > >> > > creation of a Spark .deb file?  I believe it adds all dependency
>> > jars
>> > >> > into
>> > >> > > a single lib/ folder, so in some ways it's even easier to manage
>> > than
>> > >> the
>> > >> > > assembly.
>> > >> > >
>> > >> > > Also I'm not sure if there's an equivalent plugin for Maven.
>> > >> > >
>> > >> > > thanks,
>> > >> > > Evan
>> > >> > >
>> > >> > >
>> > >> > > --
>> > >> > > --
>> > >> > >  Evan Chan
>> > >> > > Staff Engineer
>> > >> > > e...@ooyala.com  |
>> > >> > >
>> > >> > >  <
>> > >> > http://www.linkedin.com/company/ooyala><
>> http://www.twitter.com/ooyala
>> > >
>> > >> > >
>> > >> > >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > --
>> > >> > Evan Chan
>> > >> > Staff Engineer
>> > >> > e...@ooyala.com  |
>> > >> >
>> > >> > 
>> > >> > <
>> > http://www.linkedin.com/company/ooyala
>> > >> ><
>> > >> > http://www.twitter.com/ooyala>
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> --
>> Evan Chan
>> Staff Engineer
>> e...@ooyala.com  |
>>
>> 
>> <
>> http://www.twitter.com/ooyala>
>>
>
>

Re: sbt-package-bin

[Mark Hamstra]

Whoops!  Looks like it was just my brain that was broken.


On Tue, Apr 1, 2014 at 6:03 PM, Mark Hamstra wrote:

> ...or at least you could do that if the Maven build wasn't broken right
> now.
>
>
> On Tue, Apr 1, 2014 at 6:01 PM, Mark Hamstra wrote:
>
>> What the "..." is kind of depends on what you're trying to accomplish.
>>  You could be setting Hadoop version and other stuff in there, but if you
>> go too much beyond a pretty basic build, you're probably also going to have
>> to modify the  of the jdeb plugin in assembly/pom.xml to
>> include/exclude just what you want/don't want in the Debian package.
>>
>> Anyway, a typical build would look something like 'mvn -U -Pdeb
>> -DskipTests clean package', after which you should be able to find your
>> .deb in assembly/target.
>>
>>
>> On Tue, Apr 1, 2014 at 3:01 PM, Evan Chan  wrote:
>>
>>> Mark - sorry, would you mind expanding what the "" is?
>>>
>>> Something like
>>>
>>> mvn -Pdeb package
>>>
>>> ?
>>>
>>> I get:
>>>
>>> [ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one
>>> of
>>> its dependencies could not be resolved: Failed to read artifact
>>> descriptor
>>> for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not
>>> find
>>> artifact org.apache:apache:pom:13 -> [Help 1]
>>>
>>>
>>> On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell 
>>> wrote:
>>>
>>> > And there is a deb target as well - ah didn't see Mark's email.
>>> >
>>> >
>>> > On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell 
>>> > wrote:
>>> >
>>> > > Ya there is already some fragmentation here. Maven has some "dist"
>>> > targets
>>> > > and there is also ./make-distribution.sh.
>>> > >
>>> > >
>>> > > On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra <
>>> m...@clearstorydata.com
>>> > >wrote:
>>> > >
>>> > >> A basic Debian package can already be created from the Maven build:
>>> mvn
>>> > >> -Pdeb ...
>>> > >>
>>> > >>
>>> > >> On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
>>> > >>
>>> > >> > Also, I understand this is the last week / merge window for 1.0,
>>> so if
>>> > >> > folks are interested I'd like to get in a PR quickly.
>>> > >> >
>>> > >> > thanks,
>>> > >> > Evan
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan  wrote:
>>> > >> >
>>> > >> > > Hey folks,
>>> > >> > >
>>> > >> > > We are in the middle of creating a Chef recipe for Spark.   As
>>> part
>>> > of
>>> > >> > > that we want to create a Debian package for Spark.
>>> > >> > >
>>> > >> > > What do folks think of adding the sbt-package-bin plugin to
>>> allow
>>> > easy
>>> > >> > > creation of a Spark .deb file?  I believe it adds all dependency
>>> > jars
>>> > >> > into
>>> > >> > > a single lib/ folder, so in some ways it's even easier to manage
>>> > than
>>> > >> the
>>> > >> > > assembly.
>>> > >> > >
>>> > >> > > Also I'm not sure if there's an equivalent plugin for Maven.
>>> > >> > >
>>> > >> > > thanks,
>>> > >> > > Evan
>>> > >> > >
>>> > >> > >
>>> > >> > > --
>>> > >> > > --
>>> > >> > >  Evan Chan
>>> > >> > > Staff Engineer
>>> > >> > > e...@ooyala.com  |
>>> > >> > >
>>> > >> > >  <
>>> > >> > http://www.linkedin.com/company/ooyala><
>>> http://www.twitter.com/ooyala
>>> > >
>>> > >> > >
>>> > >> > >
>>> > >> >
>>> > >> >
>>> > >> > --
>>> > >> > --
>>> > >> > Evan Chan
>>> > >> > Staff Engineer
>>> > >> > e...@ooyala.com  |
>>> > >> >
>>> > >> > 
>>> > >> > <
>>> > http://www.linkedin.com/company/ooyala
>>> > >> ><
>>> > >> > http://www.twitter.com/ooyala>
>>> > >> >
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> --
>>> Evan Chan
>>> Staff Engineer
>>> e...@ooyala.com  |
>>>
>>> 
>>> >> >
>>>
>>
>>
>

Re: sbt-package-bin

[Lee Mighdoll]

> What do folks think of adding the sbt-package-bin plugin to allow easy
> creation of a Spark .deb file?  I believe it adds all dependency jars into
> a single lib/ folder, so in some ways it's even easier to manage than the
> assembly.
>

You might also check out the
sbt-native-packager.


Cheers,
Lee