Assembly build on spark 2.0.0

2016-08-26 Thread Srikanth Sampath 
Hi,
mvn assembly is creating a .tgz distribution.  How can I create a plain jar
archive?  I would like to create a spark-assembly-.jar
-Srikanth

Re: Insert non-null values from dataframe

2016-08-26 Thread Russell Spitzer 
Cassandra does not differentiate between null and empty, so when reading
from C* all empty values are reported as null. To avoid inserting nulls
(avoiding tombstones) see

https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md#globally-treating-all-nulls-as-unset

This will not prevent those columns from being read as null though, it will
only skip writing tombstones.

On Thu, Aug 25, 2016, 1:23 PM Selvam Raman  wrote:

> Hi ,
>
> Dataframe:
> colA colB colC colD colE
> 1 2 3 4 5
> 1 2 3 null null
> 1 null null  null 5
> null null  3 4 5
>
> I want to insert dataframe to nosql database, where null occupies
> values(Cassandra). so i have to insert the column which has non-null values
> in the row.
>
> Expected:
>
> Record 1: (1,2,3,4,5)
> Record 2:(1,2,3)
> Record 3:(1,5)
> Record 4:(3,4,5)
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>

Re: Assembly build on spark 2.0.0

2016-08-26 Thread Radoslaw Gruchalski 
mvn package might be the command you’re looking for.

–
Best regards,
Radek Gruchalski
ra...@gruchalski.com


On August 26, 2016 at 3:59:24 PM, Srikanth Sampath (
ssampath.apa...@gmail.com) wrote:

Hi,
mvn assembly is creating a .tgz distribution.  How can I create a plain jar
archive?  I would like to create a spark-assembly-.jar
-Srikanth

Mesos is now a maven module

2016-08-26 Thread Michael Gummelt 
Hello devs,

Much like YARN, Mesos has been refactored into a Maven module.  So when
building, you must add "-Pmesos" to enable Mesos support.

The pre-built distributions from Apache will continue to enable Mesos.

PR: https://github.com/apache/spark/pull/14637

Cheers

-- 
Michael Gummelt
Software Engineer
Mesosphere

Re: Mesos is now a maven module

2016-08-26 Thread Reynold Xin 
This is great!


On Fri, Aug 26, 2016 at 1:20 PM, Michael Gummelt 
wrote:

> Hello devs,
>
> Much like YARN, Mesos has been refactored into a Maven module.  So when
> building, you must add "-Pmesos" to enable Mesos support.
>
> The pre-built distributions from Apache will continue to enable Mesos.
>
> PR: https://github.com/apache/spark/pull/14637
>
> Cheers
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>

Re: Mesos is now a maven module

2016-08-26 Thread Jacek Laskowski 
Hi Michael,

Congrats!

BTW What I like about the change the most is that it uses the
pluggable interface for TaskScheduler and SchedulerBackend (as
introduced by YARN). Think Standalone should follow the steps. WDYT?

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Fri, Aug 26, 2016 at 10:20 PM, Michael Gummelt
 wrote:
> Hello devs,
>
> Much like YARN, Mesos has been refactored into a Maven module.  So when
> building, you must add "-Pmesos" to enable Mesos support.
>
> The pre-built distributions from Apache will continue to enable Mesos.
>
> PR: https://github.com/apache/spark/pull/14637
>
> Cheers
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Mesos is now a maven module

2016-08-26 Thread Michael Gummelt 
If it's separable, then sure.  Consistency is nice.

On Fri, Aug 26, 2016 at 2:14 PM, Jacek Laskowski  wrote:

> Hi Michael,
>
> Congrats!
>
> BTW What I like about the change the most is that it uses the
> pluggable interface for TaskScheduler and SchedulerBackend (as
> introduced by YARN). Think Standalone should follow the steps. WDYT?
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Fri, Aug 26, 2016 at 10:20 PM, Michael Gummelt
>  wrote:
> > Hello devs,
> >
> > Much like YARN, Mesos has been refactored into a Maven module.  So when
> > building, you must add "-Pmesos" to enable Mesos support.
> >
> > The pre-built distributions from Apache will continue to enable Mesos.
> >
> > PR: https://github.com/apache/spark/pull/14637
> >
> > Cheers
> >
> > --
> > Michael Gummelt
> > Software Engineer
> > Mesosphere
>



-- 
Michael Gummelt
Software Engineer
Mesosphere

Re: GraphFrames 0.2.0 released

2016-08-26 Thread Joseph Bradley 
This should do it:
https://github.com/graphframes/graphframes/releases/tag/release-0.2.0
Thanks for the reminder!
Joseph

On Wed, Aug 24, 2016 at 10:11 AM, Maciej Bryński  wrote:

> Hi,
> Do you plan to add tag for this release on github ?
> https://github.com/graphframes/graphframes/releases
>
> Regards,
> Maciek
>
> 2016-08-17 3:18 GMT+02:00 Jacek Laskowski :
>
>> Hi Tim,
>>
>> AWESOME. Thanks a lot for releasing it. That makes me even more eager
>> to see it in Spark's codebase (and replacing the current RDD-based
>> API)!
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Tue, Aug 16, 2016 at 9:32 AM, Tim Hunter 
>> wrote:
>> > Hello all,
>> > I have released version 0.2.0 of the GraphFrames package. Apart from a
>> few
>> > bug fixes, it is the first release published for Spark 2.0 and both
>> scala
>> > 2.10 and 2.11. Please let us know if you have any comment or questions.
>> >
>> > It is available as a Spark package:
>> > https://spark-packages.org/package/graphframes/graphframes
>> >
>> > The source code is available as always at
>> > https://github.com/graphframes/graphframes
>> >
>> >
>> > What is GraphFrames?
>> >
>> > GraphFrames is a DataFrame-based graph engine Spark. In addition to the
>> > algorithms available in GraphX, users can write highly expressive
>> queries by
>> > leveraging the DataFrame API, combined with a new API for motif
>> finding. The
>> > user also benefits from DataFrame performance optimizations within the
>> Spark
>> > SQL engine.
>> >
>> > Cheers
>> >
>> > Tim
>> >
>> >
>> >
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>
>
> --
> Maciek Bryński
>

Re: Assembly build on spark 2.0.0

2016-08-26 Thread Srikanth Sampath 
Hi,
Thanks Radek.  However mvn package does not build the uber jar.  I am
looking for an uber jar and not a distribution.  I have seen references to
the uber jar here


What I see in the spark 2.0 codeline (assembly/pom.xml) builds a
distribution. I



  bigtop-dist

  

  



  

org.apache.maven.plugins

maven-assembly-plugin



  

dist

package



  single





  

src/main/assembly/assembly.xml

  



  



  

...

  




In src/main/assembly/assembly.xml we see



  dist

  

tar.gz

dir

  

  false

.



On Sat, Aug 27, 2016 at 1:02 AM, Radoslaw Gruchalski 
wrote:

> mvn package might be the command you’re looking for.
>
> –
> Best regards,
> Radek Gruchalski
> ra...@gruchalski.com
>
>
> On August 26, 2016 at 3:59:24 PM, Srikanth Sampath (
> ssampath.apa...@gmail.com) wrote:
>
> Hi,
> mvn assembly is creating a .tgz distribution.  How can I create a plain
> jar archive?  I would like to create a spark-assembly-.jar
> -Srikanth
>
>