Spark 1.3.0 Build Failure

So, I am trying to build Spark 1.3.0 (standalone mode) on Windows 7 using Maven, but I'm getting a build failure. java -version java version "1.8.0_31" Java(TM) SE Runtime Environment (build 1.8.0_31-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode) Here is the command I am usi

Spark 1.3.0 missing dependency?

Upon executing these two lines of code: conf = new SparkConf().setAppName(appName).setMaster(master); sc = new JavaSparkContext(conf); I get the following error message: ERROR Configuration: Failed to set setXIncludeAware(true) for parser org.apache.xerces.jaxp.DocumentBuilderFac

Re: How to start master and workers on Windows

I'm in the same boat. What are the equivalent commands to stop the master and workers? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-start-master-and-workers-on-Windows-tp12669p22341.html Sent from the Apache Spark User List mailing list archive at

Spark 1.3.0 DataFrame count() method throwing java.io.EOFException

Note: I am running Spark on Windows 7 in standalone mode. In my app, I run the following: DataFrame df = sqlContext.sql("SELECT * FROM tbBER"); System.out.println("Count: " + df.count()); tbBER is registered as a temp table in my SQLContext. When I try to print the number of rows

What happened to the Row class in 1.3.0?

I am trying to call Row.create(object[]) similarly to what's shown in this programming guide , but the create() method is no longer recognized. I tried to look up the documentation for the Ro

Re: Microsoft SQL jdbc support from spark sql

I am having the same issue with my java application. String url = "jdbc:sqlserver://" + host + ":1433;DatabaseName=" + database + ";integratedSecurity=true"; String driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"; SparkConf conf = new SparkConf().setAppName(appName

Re: Microsoft SQL jdbc support from spark sql

I have found that it works if you place the sqljdbc41.jar directly in the following folder: YOUR_SPARK_HOME/core/target/jars/ So Spark will have the SQL Server jdbc driver when it computes its classpath. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Micr

Re: Microsoft SQL jdbc support from spark sql

I take it back. My solution only works when you set the master to "local". I get the same error when I try to run it on the cluster. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Microsoft-SQL-jdbc-support-from-spark-sql-tp22399p22525.html Sent from the Ap

Which version of Hive QL is Spark 1.3.0 using?

So I'm trying to store the results of a query into a DataFrame, but I get the following exception thrown: Exception in thread "main" java.lang.RuntimeException: [1.71] failure: ``*'' expected but `select' found SELECT DISTINCT OutSwitchID FROM wtbECRTemp WHERE OutSwtichID NOT IN (SELECT SwitchID

Unsupported types in org.apache.spark.sql.jdbc.JDBCRDD$.getCatalystType

So I am trying to pull data from an external database using JDBC Map options = new HashMap<>(); options.put("driver", driver); options.put("url", dburl); options.put("dbtable", "tmpTrunk"); DataFrame tbTrunkInfo = sqlContext.load("jdbc", options); And the

Updating a Column in a DataFrame

In my Java application, I want to update the values of a Column in a given DataFrame. However, I realize DataFrames are immutable, and therefore cannot be updated by conventional means. Is there a workaround for this sort of transformation? If so, can someone provide an example? -- View this mes