Unless you are directly concerned with the query optimization you needn't
modify catalyst or any of the core Spark SQL code.
You can simply create a new project with Spark SQL as a dependency and like
is done in MLLib Vectors (in 1.3, the newer versions have it for matrices as
well)
Use the @SQLU
I was having the following issue with creating a user defined type (PosData)
which I had naturally included in an object (SQLTests)
In the SQLUserDefinedType.java in the sql.catalyst/src/main
(https://github.com/apache/spark/blob/f9969098c8cb15e36c718b80c6cf5b534a6cf7c3/sql/catalyst/src/main/sca
Yes, this private is checked at compile time and my class is in a subpackage
of org.apache.spark.ui, so the visibility is not the issue, or at least not
as far as I can tell.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/JettyUtils-createServletHandler-Met
I have a very strange error in Spark 1.3 where at runtime in the
org.apache.spark.ui.JettyUtils object the method createServletHandler is not
found
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.spark.ui.JettyUtils$.createServletHandler(Ljava/lang/String;Ljavax/servlet/http/Htt
The JavaAPI uses FakeClassTag for all of the implicit class tags fed to RDDs
during creation, mapping, etc. I am working on a more generic Scala library
where I won't always have the type information beforehand. Is it safe /
accepted practice to use FakeClassTag in these situations as well? It was
That worked for me as well, I was using spark 1.0 compiled against Hadoop
1.0, switching to 1.0.1 compiled against hadoop 2
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/wholeTextFiles-not-working-with-HDFS-tp7490p10547.html
Sent from the Apache Spark User
I have the same issue
val a = sc.textFile("s3n://MyBucket/MyFolder/*.tif")
a.first
works perfectly fine, but
val d = sc.wholeTextFiles("s3n://MyBucket/MyFolder/*.tif") does not
work
d.first
Gives the following error message
java.io.FileNotFoundExceptio
