Persist DStream into a single file on HDFS

2017-09-28 Thread Mustafa Elbehery
Hi Folks, I am writing a pipeline which reads from Kafka, applying some transformations, then persist to HDFS. Obviously such operation is not supported to DStream, since the *DStream.save*(Path) *method, considers the Path as a directory, not a file. Also using *repartition(1).mode(Savemode.APPE

Re: Loading multiple packages while starting spark-shell

2016-03-30 Thread Mustafa Elbehery
upplying a > comma-delimited list of maven coordinates with --packages. > > On Wed, Mar 30, 2016 at 7:15 AM, Mustafa Elbehery < > elbeherymust...@gmail.com> wrote: > >> Hi Folks, >> >> I am trying to use two Spark packages while working from the shell .. >

Loading multiple packages while starting spark-shell

2016-03-30 Thread Mustafa Elbehery
Hi Folks, I am trying to use two Spark packages while working from the shell .. Unfortunately it accepts only one package as parameter and ignore the second. Any suggestion how to work around this ? Regards.

PySpark RDD with NumpyArray Structure

2015-12-06 Thread Mustafa Elbehery
Hi All, I would like to parallelize Python NumpyArray to apply scikit Learn algorithm on top of Spark. When I call *sc.parallelize() *I receive rdd of different structure. To be more precise, I am trying to have the following, X = [[ 0.49426097 1.45106697] [-1.42808099 -0.83706377] [ 0.338559

Re: SparkSQL JDBC to PostGIS

2015-11-05 Thread Mustafa Elbehery
> On Wed, Nov 4, 2015 at 1:46 PM, Mustafa Elbehery < > elbeherymust...@gmail.com> wrote: > >> Hi Folks, >> >> I am trying to connect from SparkShell to PostGIS Database. Simply >> PostGIS is a *spatial *extension for Postgresql, in order to support >> *ge

SparkSQL JDBC to PostGIS

2015-11-04 Thread Mustafa Elbehery
the PostGIS extension. I have tried to build the JDBC driver of the PostGIS, and use it, but it throws *No suitable driver found for jdbc:postgis.* Would like to know if anyone had the same problem before reporting the bug on JIRA. Regards. -- Mustafa Elbehery EIT ICT Labs Master School <h

Indexing Support

2015-10-18 Thread Mustafa Elbehery
in future releases ?!! Regards. -- Mustafa Elbehery EIT ICT Labs Master School <http://www.masterschool.eitictlabs.eu/home/> +49(0)15750363097 skype: mustafaelbehery87