[GraphX] Cast error when comparing a vertex attribute after its type has changed

[Pierre-Alexandre Fonta]

Hi,

Testing in mapTriplets if a vertex attribute, which is defined as Integer in
first VertexRDD but has been changed after to Double by mapVertices, is
greater than a number throws "java.lang.ClassCastException:
java.lang.Integer cannot be cast to java.lang.Double".

If second elements of vertex attributes don't contain a zero there is no
error.

Replace "vertices: RDD[(Long, (Int, Int))]" by "vertices: RDD[(Long, (Int,
Double))]" in the code below solves the problem.

I am not sure if it's a lineage gestion issue or if it's normal. I am using
Spark 0.9.1.

Thanks for your help,

Pierre-Alexandre


import org.apache.spark._
import org.apache.spark.rdd.RDD
import org.apache.spark.graphx._

val vertices: RDD[(Long, (Int, Integer))] = sc.parallelize(Array(
  (1L, (4, 0)),
  (2L, (0, 0)),
  (3L, (7, 0))
))
val edges = sc.parallelize(Array(
  Edge(1L, 2L, 0),
  Edge(2L, 3L, 2),
  Edge(3L, 1L, 5)
))
val graph0 = Graph(vertices, edges)
val graph1 = graph0.mapVertices { case (vid, (n, _)) => (n, n.toDouble/3) }
val graph2 = graph1.mapTriplets(t => { if (t.srcAttr._2 > 0) 1 else 2 })
graph2.edges.foreach(println(_)) // ERROR


ERROR Executor: Exception in task ID 7
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Double
        at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:119)
        at scala.Tuple2._2$mcD$sp(Tuple2.scala:19)
        at
$line27.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:27)
        at
$line27.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:27)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at 
org.apache.spark.graphx.impl.EdgePartition.map(EdgePartition.scala:96)
        at
org.apache.spark.graphx.impl.GraphImpl$$anonfun$10.apply(GraphImpl.scala:148)
        at
org.apache.spark.graphx.impl.GraphImpl$$anonfun$10.apply(GraphImpl.scala:133)
        at
org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:85)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
        at org.apache.spark.graphx.EdgeRDD.compute(EdgeRDD.scala:48)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
        at org.apache.spark.scheduler.Task.run(Task.scala:53)
        at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:211)
        at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:45)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/GraphX-Cast-error-when-comparing-a-vertex-attribute-after-its-type-has-changed-tp4119.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.