Fwd: Spark for Oracle sample code

2015-09-25 Thread Cui Lin
Hello, All, I found the examples for JDBC connection are mostly read the whole table and then do operations like joining. val jdbcDF = sqlContext.read.format("jdbc").options( Map("url" -> "jdbc:postgresql:dbserver", "dbtable" -> "schema.tablename")).load() Sometimes it is not practical sinc

Spark for Oracle sample code

2015-09-25 Thread Cui Lin
Hello, All, I found the examples for JDBC connection are mostly read the whole table and then do operations like joining. val jdbcDF = sqlContext.read.format("jdbc").options( Map("url" -> "jdbc:postgresql:dbserver", "dbtable" -> "schema.tablename")).load() Sometimes it is not practical sinc

Time-series prediction using spark

2015-11-02 Thread Cui Lin
Hello, all, I am wondering if anyone tried time series prediction using spark? Any good practice to suggest me? Thanks a lot! -- Best regards! Lin,Cui

Best way to import data from Oracle to Spark?

2015-09-08 Thread Cui Lin
What's the best way to import data from Oracle to Spark? Thanks! -- Best regards! Lin,Cui

How to add sparkSQL into a standalone application

2015-09-17 Thread Cui Lin
Hello, I got stuck in adding spark sql into my standalone application. The build.sbt is defined as: libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1" I got the following error when building the package: *[error] /data/workspace/test/src/main/scala/TestMain.scala:6: object sql

Re: How to add sparkSQL into a standalone application

2015-09-17 Thread Cui Lin
% "1.4.1" > > Though, I would consider using spark-hive and HiveContext, as the query > parser is more powerful and you'll have access to window functions and other > features. > > > On Thu, Sep 17, 2015 at 10:59 AM, Cui Lin wrote: > >> Hello, >>

What's the best practice to parse JSON using spark

2015-09-18 Thread Cui Lin
Hello,All, Parsing JSON's nested structure is easy if using Java or Python API. Where I can find the similar way to parse JSON file using spark? Another question is by using SparkSQL, how can i easily save the results into NOSQL DB? any examples? Thanks a lot! -- Best regards! Lin,Cui

Spark sql results can't be printed out to system console from spark streaming application

2015-03-03 Thread Cui Lin
).foreach(line => System.out.println(line)) > >val jsonMessage = sqlContext.jsonRDD(anotherPeopleRDD) > >jsonMessage.toArray().foreach(line => System.out.println(line)) > >jsonMessage.registerTempTable("peopl

Re: Spark sql results can't be printed out to system console from spark streaming application

2015-03-03 Thread Cui Lin
I firstly tried simply copy the codes below into spark application that has codes working with kafka streaming. No any result can be printed into system console, then I tried spark shell, and it works. I can’t figured out the reason. Best regards, Cui Lin From: Tobias Pfeiffer mailto:t

How to parse Json formatted Kafka message in spark streaming

2015-03-04 Thread Cui Lin
columns in the json formatted message. Thanks for your help! Best regards, Cui Lin

Re: How to parse Json formatted Kafka message in spark streaming

2015-03-05 Thread Cui Lin
columns. Is there any way to get column name instead of hard code “time” in this case? mapper.readValue[Map[String,Any]](x).get("time") Best regards, Cui Lin From: Helena Edelson mailto:helena.edel...@datastax.com>> Date: Thursday, March 5, 2015 at 7:02 AM To: Ted Yu mailto:yuz

Re: How to parse Json formatted Kafka message in spark streaming

2015-03-05 Thread Cui Lin
Hi, Ted, Thanks for your reply. I noticed from the below link partitions.size will not work for checking empty RDD in streams. It seems that the problem can be solved in spark 1.3 which is no way to download at this time? https://issues.apache.org/jira/browse/SPARK-5270 Best regards, Cui Lin

Re: General Purpose Spark Cluster Hardware Requirements?

2015-03-08 Thread Cui Lin
No woder I had out of memory issue before… I doubt if we really need such configuration on production level… Best regards, Cui Lin From: Krishna Sankar mailto:ksanka...@gmail.com>> Date: Sunday, March 8, 2015 at 3:27 PM To: Nasir Khan mailto:nasirkhan.onl...@gmail.com>&g

Spark Streaming input data source list

2015-03-09 Thread Cui Lin
Dear all, Could you send me a list for input data source that spark streaming could support? My list is HDFS, Kafka, textfile?… I am wondering if spark streaming could directly read data from certain port (443 e.g.) that my devices directly send to? Best regards, Cui Lin

Re: Spark Streaming input data source list

2015-03-09 Thread Cui Lin
Tathagata, Thanks for your quick response. The link is helpful to me. Do you know any API for streaming data from RMDB ? Best regards, Cui Lin From: Tathagata Das mailto:t...@databricks.com>> Date: Monday, March 9, 2015 at 11:28 AM To: Cui Lin mailto:cui@hds.com>&g

SocketTextStream not working from messages sent from other host

2015-03-10 Thread Cui Lin
sent from other host to that port were ignored. I can see all messages from “nc –l [port no.]", why such thing could happen? Best regards, Cui Lin

Web Service + Spark

2015-01-09 Thread Cui Lin
Hello, All, What’s the best practice on deploying/publishing spark-based scientific applications into a web service? Similar to Shiny on R. Thanks! Best regards, Cui Lin

Re: Web Service + Spark

2015-01-10 Thread Cui Lin
Thanks, Gaurav and Corey, Probably I didn’t make myself clear. I am looking for best Spark practice similar to Shiny for R, the analysis/visualziation results can be easily published to web server and shown from web browser. Or any dashboard for Spark? Best regards, Cui Lin From: gtinside