Re: Issue with sbt failing on amplab succinct

2016-03-04 Thread Luciano Resende
b: 29 > Exception in thread "main" java.lang.NoClassDefFoundError: > edu/berkeley/cs/succinct/package$ > > > I suspect I have not got the Library dependencies corrects? > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Flume with Spark Streaming Sink

2016-03-20 Thread Luciano Resende
; 4.0 (TID 76, impact3.indigo.co.il): java.lang.NoClassDefFoundError: > org/apache/spark/streaming/flume/sink/SparkFlumeProtocol$Callback > at > org.apache.spark.streaming.flume.FlumePollingReceiver$$anonfun$onStart$1.apply(FlumePollingInputDStream.scala:84) > > >

Re: Setting up spark to run on two nodes

2016-03-21 Thread Luciano Resende
There is also sbin/star-all.sh and sbin/stop-all.sh which enables you to star/stop master and workers all together On Sunday, March 20, 2016, Akhil Das wrote: > You can simply execute the sbin/start-slaves.sh file to start up all slave > processes. Just make sure you have spark installed on the

Re: Spark on Mobile platforms

2016-04-07 Thread Luciano Resende
apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Spark support for Complex Event Processing (CEP)

2016-04-17 Thread Luciano Resende
V8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Spark build failure with com.oracle:ojdbc6:jar:11.2.0.1.0

2016-05-02 Thread Luciano Resende
- >>> >>> [INFO] BUILD FAILURE >>> >>> [INFO] >>> -------- >>> >>> [INFO] Total time: 03:53 min >>> >>> [INFO] Finished at: 2016-05-02T17:44:57-07:00 >>> >>> [INFO] Final Memory: 80M/1525M >>> >>> [INFO] >>> >>> >>> [ERROR] Failed to execute goal on project >>> spark-docker-integration-tests_2.11: Could not resolve dependencies for >>> project >>> org.apache.spark:spark-docker-integration-tests_2.11:jar:2.0.0-SNAPSHOT: >>> Failed to collect dependencies at com.oracle:ojdbc6:jar:11.2.0.1.0: Failed >>> to read artifact descriptor for com.oracle:ojdbc6:jar:11.2.0.1.0: Could not >>> transfer artifact com.oracle:ojdbc6:pom:11.2.0.1.0 from/to uber-artifactory >>> (http://artifactory.uber.internal:4587/artifactory/repo/): Connect to >>> artifactory.uber.internal:4587 [artifactory.uber.internal/10.162.11.61] >>> failed: Operation timed out -> [Help 1] >>> >>> > > > -- > Regards, > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Spark build failure with com.oracle:ojdbc6:jar:11.2.0.1.0

2016-05-09 Thread Luciano Resende
;s > maven release blog. > > > Just my inference based on what I see form git and JIRA, however, I do see > a fix required to patch pom.xml to apply the correct groupId and version # > for ojdbc6 driver. > > > Thoughts? > > > > Get Oracle JDBC drivers and UCP from

Re: sbt for Spark build with Scala 2.11

2016-05-12 Thread Luciano Resende
2.0.0-SNAPSHOT. > Does this mean, the only option is to put all the required jars in the lib > folder (unmanaged dependencies)? > > Regards, > Raghava. > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Suggestions on Lambda Architecture in Spark

2016-06-13 Thread Luciano Resende
This might be useful: https://spark-summit.org/east-2016/events/lambda-at-weather-scale/ On Monday, June 13, 2016, KhajaAsmath Mohammed wrote: > Hi, > > In my current project, we are planning to implement POC for lambda > architecture using spark streaming. My use case would be > > Kafka --> ba

[ANNOUNCE] Apache SystemML 0.10.0-incubating released

2016-06-15 Thread Luciano Resende
The Apache SystemML team is pleased to announce the release of Apache SystemML version 0.10.0-incubating. Apache SystemML provides declarative large-scale machine learning (ML) that aims at flexible specification of ML algorithms and automatic generation of hybrid runtime plans ranging from single

Re: Integration tests for Spark Streaming

2016-06-28 Thread Luciano Resende
er List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: 2.0.0 packages for twitter streaming, flume and other connectors

2016-08-05 Thread Luciano Resende
On Thursday, August 4, 2016, Sean Owen wrote: > You're looking for http://bahir.apache.org/ > > And we should have a release supporting Spark 2.0 in a few days... Thanks -- Luciano -- Sent from my Mobile device

Help testing the Spark Extensions for the Apache Bahir 2.0.0 release

2016-08-06 Thread Luciano Resende
ector Twitter DStream connector ZeroMQ DStream connector MQTT Structured Streaming Thanks in advance -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Help testing the Spark Extensions for the Apache Bahir 2.0.0 release

2016-08-07 Thread Luciano Resende
://lresende.blogspot.gr/2016/08/getting-started-with-apache-bahir-mqtt_4.html On Sun, Aug 7, 2016 at 11:24 AM, Sivakumaran S wrote: > Hi, > > How can I help? > > regards, > > Sivakumaran S > > On 06-Aug-2016, at 6:18 PM, Luciano Resende wrote: > > Apache Bahir is voting i

[ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Luciano Resende
The Apache Bahir PMC is pleased to announce the release of Apache Bahir 2.0.0 which is our first major release and provides the following extensions for Apache Spark 2.0.0 : Akka Streaming MQTT Streaming and Structured Streaming Twitter Streaming ZeroMQ Streaming For more information about Apach

Re: pyspark + kafka + streaming = NoSuchMethodError

2015-12-17 Thread Luciano Resende
) > at py4j.Gateway.invoke(Gateway.java:259) > at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) > at py4j.commands.CallCommand.execute(CallCommand.java:79) > at py4j.GatewayConnection.run(GatewayConnection.java:207) > at java.lang.Thread.run(Thread.java:745) > > Am I missing something? > > Thanks in advance > Chris M. > > > > > > -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: How to edit/delete a message posted in Apache Spark User List?

2016-02-05 Thread Luciano Resende
unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Imported CSV file content isn't identical to the original file

2016-02-07 Thread Luciano Resende
y of the table wasn’t shown correctly. I also used the > other way to read csv file, by sc.textFile(...).map(_.split(" ")) and > sqlContext.createDataFrame, and the result is the same. Can someone point > me out where I did it wrong? > > — > BR, > Todd Leo > ​ > -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Imported CSV file content isn't identical to the original file

2016-02-08 Thread Luciano Resende
ue is gone. >> >> — >> Cheers, >> Todd Leo >> >> >> ​ >> On Mon, Feb 8, 2016 at 2:12 PM Luciano Resende >> wrote: >> >>> I tried in both 1.5.0, 1.6.0 and 2.0.0 trunk and >>> com.databricks:spark-csv_2.10:1.3.0 wit

[ANNOUNCE] Apache SystemML 0.9.0-incubating released

2016-02-15 Thread Luciano Resende
The Apache SystemML team is pleased to announce the release of Apache SystemML version 0.9.0-incubating. This is the first release as an Apache project. Apache SystemML provides declarative large-scale machine learning (ML) that aims at flexible specification of ML algorithms and automatic generat

Re: Spark Integration Patterns

2016-02-28 Thread Luciano Resende
rking a process on a remote host to >>> execute a shell script seems like a lot of effort What are the recommended >>> ways to connect and query Spark from a remote client ? Thanks Thx ! >>> -- >>> View this message in context: Spark Integrat

Re: SF Spark Office Hours Experiment - Friday Afternoon

2015-10-21 Thread Luciano Resende
> Holden :) > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau > +1, Very good initiative !!! Will try to attend some as time permits. -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: spark streaming 1.51. uses very old version of twitter4j

2015-10-21 Thread Luciano Resende
easily by just including the > following lines in your pom.xml. > > > >org.twitter4j > twitter4j-core >[4.0,) > >... > > > -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Rule Engine for Spark

2015-12-10 Thread Luciano Resende
> On Wed, Nov 4, 2015 at 1:42 AM, Cassa L wrote: > >> Hi, >> Has anyone used rule engine with spark streaming? I have a case where >> data is streaming from Kafka and I need to apply some rules on it (instead >> of hard coding in a code). >> >> Thanks, >

Re: SparkR - calling as.vector() with rdd dataframe causes error

2015-09-17 Thread Luciano Resende
',link='ward',K=5) > > Error in as.vector(data) : > no method for coercing this S4 class to a vector > > > I get the same error if I try to directly call as.vector(rdd) as well. > > Is there a reason why this works for localDF and not rdd? Should I be > doing something else to coerce the object into a vector? > > Thanks, > Ellen > > > -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: SparkR - calling as.vector() with rdd dataframe causes error

2015-09-18 Thread Luciano Resende
gt;> >> Is there a reason why this works for localDF and not rdd? Should I be >> doing something else to coerce the object into a vector? >> >> Thanks, >> Ellen >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-calling-as-vector-with-rdd-dataframe-causes-error-tp24717.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional >> commands, e-mail: user-h...@spark.apache.org >> >> > -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: SparkR - calling as.vector() with rdd dataframe causes error

2015-09-22 Thread Luciano Resende
m is that when I call a clustering > algorithm like amap::hcluster(), I get an error from as.vector() that the > rdd cannot be coerced into a vector. > > On Fri, Sep 18, 2015 at 12:33 PM, Luciano Resende > wrote: > >> I see the thread with all the responses on the bottom

Re: Spark to Kafka communication encrypted ?

2016-08-31 Thread Luciano Resende
a.apache.org/documentation.html#security_ssl >> >> On Wed, Aug 31, 2016 at 2:03 AM, Eric Ho wrote: >> > I can't find in Spark 1.6.2's docs in how to turn encryption on for >> Spark to >> > Kafka communication ... I think that the Spark docs only tells you how

Re: Scala Vs Python

2016-09-05 Thread Luciano Resende
s written in java/scala. Take a look at Apache SystemML http://systemml.apache.org/ for more details. -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Need help to setup eclipse environment

2016-10-05 Thread Luciano Resende
le > only. > > Thanks in advance. > > Best Regards, > > Mahedi > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Need help to setup eclipse environment

2016-10-05 Thread Luciano Resende
se in eclipse. But it is >>> showing lots of error. I think I am importing in a wrong way. Is there any >>> documentation how to setup spark in eclipse? I want to work on a particular >>> module (ex. Streaming). I am wondering if I only need to import that module >>>

Re: Need help to setup eclipse environment

2016-10-05 Thread Luciano Resende
Yes and yes. On Wednesday, October 5, 2016, Md. Mahedi Kaysar wrote: > Should I import it as a maven project? Should I import all the > sub-projects together? > > On Wed, Oct 5, 2016 at 10:14 AM, Marco Mistroni > wrote: > >> Hi >> Havent tried myself but spark is a mvn project. Have u tried to

Re: [ANNOUNCE] Announcing Spark 2.0.1

2016-10-05 Thread Luciano Resende
of >>> Spark. We strongly recommend all 2.0.0 users to upgrade to this stable >>> release. >>> >>> To download Apache Spark 2.0.1, visit http://spark.apache.org/downlo >>> ads.html >>> >>> We would like to acknowledge all comm

Re: [ANNOUNCE] Announcing Spark 2.0.1

2016-10-06 Thread Luciano Resende
repositories/releases/ > org/apache/spark/spark-core_2.11/ > -- Not sure why they haven't synced to maven central yet > > Shivaram > > On Wed, Oct 5, 2016 at 8:37 PM, Luciano Resende > wrote: > > It usually don't take that long to be synced, I still don'

Re: When will spark 2.0.1 be available in maven repo?

2016-10-07 Thread Luciano Resende
me/sushrutikhar > <https://about.me/sushrutikhar?promo=email_sig> > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: JSON Arrays and Spark

2016-10-10 Thread Luciano Resende
_corrupt_record: string (nullable = true) > > Both are legit JSON structures… Do you think that #2 is a bug? > > jg > > > > > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Spark 1.2

2016-10-25 Thread Luciano Resende
Regards, > Ayan Guha > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

[ANNOUNCE] Apache Bahir 2.0.1

2016-10-27 Thread Luciano Resende
Apache Bahir and to download the release: http://bahir.apache.org -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Akka Stream as the source for Spark Streaming. Please advice...

2016-11-12 Thread Luciano Resende
If you are interested in Akka streaming, it is being maintained in Apache Bahir. For Akka there isn't a structured streaming version yet, but we would be interested in collaborating in the structured streaming version for sure. On Thu, Nov 10, 2016 at 8:46 AM shyla deshpande wrote: > I am using

[ANNOUNCE] Apache SystemML 0.11.0-incubating released

2016-11-13 Thread Luciano Resende
he ASF. -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Create a Column expression from a String

2016-11-19 Thread Luciano Resende
Are you looking for UDFs? https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-udfs.html On Sun, Nov 20, 2016 at 3:12 AM Stuart White wrote: > I'd like to allow for runtime-configured Column expressions in my > Spark SQL application. For example, if my application needs

Re: Regarding Connection Problem

2016-12-17 Thread Luciano Resende
T, then you can see an example here https://github.com/lresende/bahir-iot-demo -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

Re: Old version of Spark [v1.2.0]

2017-01-19 Thread Luciano Resende
s, > >> _ > >> Md. Rezaul Karim, BSc, MSc > >> PhD Researcher, INSIGHT Centre for Data Analytics > >> National University of Ireland, Galway > >> IDA Business Park, Dangan, Galway, Ireland > >> Web: http://www.reza-analytics.eu/index.html > > > > > > > > > > -- > > Best Regards, > > Ayan Guha > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/

[ANNOUNCE] Apache Bahir 2.1.1 Released

2017-07-17 Thread Luciano Resende
- ZeroMQ Streaming For more information about Apache Bahir and to download the latest release go to: http://bahir.apache.org The Apache Bahir streaming connectors are also available at: https://spark-packages.org/?q=bahir -- Luciano Resende http://twitter.com/lresende1975 http

[ANNOUNCE] Apache Bahir 2.2.0 Released

2017-09-06 Thread Luciano Resende
Apache Bahir provides extensions to multiple distributed analytic platforms, extending their reach with a diversity of streaming connectors and SQL data sources. The Apache Bahir community is pleased to announce the release of Apache Bahir 2.2.0 which provides the following extensions for Apache Sp

[ANNOUNCE] Apache Bahir 2.1.2 Released

2018-06-07 Thread Luciano Resende
-- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

[ANNOUNCE] Apache Bahir 2.2.1 Released

2018-06-27 Thread Luciano Resende
-- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/

[ANNOUNCE] Apache Toree 0.2.0-incubating Released

2018-08-15 Thread Luciano Resende
information about Apache Toree and go download the latest release go to: https://toree.incubator.apache.org/ For more information on how to use Apache Toree please visit our documentation page: https://toree.incubator.apache.org/docs/current/user/quick-start/ -- Luciano Resende http

[ANNOUNCE] Apache Bahir 2.1.3 Released

2018-11-13 Thread Luciano Resende
-- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

[ANNOUNCE] Apache Bahir 2.2.2 Released

2018-11-13 Thread Luciano Resende
-- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

[ANNOUNCE] Apache Toree 0.3.0-incubating Released

2018-11-13 Thread Luciano Resende
page: https://toree.incubator.apache.org/docs/current/user/quick-start/ -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/ - To unsubscribe e-mail: user-unsubscr

[ANNOUNCE] Apache Bahir 2.3.0 Released

2018-12-04 Thread Luciano Resende
Bahir extensions in your application please visit our documentation page http://bahir.apache.org/docs/spark/overview/ The Apache Bahir PMC -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/ - To

[ANNOUNCE] Apache Bahir 2.3.1 Released

2018-12-04 Thread Luciano Resende
Bahir extensions in your application please visit our documentation page http://bahir.apache.org/docs/spark/overview/ The Apache Bahir PMC -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/ - To

[ANNOUNCE] Apache Bahir 2.3.2 Released

2018-12-04 Thread Luciano Resende
Bahir extensions in your application please visit our documentation page http://bahir.apache.org/docs/spark/overview/ The Apache Bahir PMC -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/ - To

[ANNOUNCE] Apache Bahir 2.3.3 Released

2019-06-08 Thread Luciano Resende
Bahir extensions in your application please visit our documentation page https://bahir.apache.org/docs/spark/overview/ The Apache Bahir PMC -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com

[ANNOUNCE] Apache Bahir 2.2.3 Released

2019-06-08 Thread Luciano Resende
Bahir extensions in your application please visit our documentation page https://bahir.apache.org/docs/spark/overview/ The Apache Bahir PMC -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com