Hi, all. I want to seek suggestions on how to do checkpoint more
efficiently, especially for iterative applications written by GraphX.
For iterative applications, the lineage of a job can be very long, which is
easy to cause statckoverflow error. A solution is to do checkpoint.
However, checkpoint is time-consuming and not easy for ordinary users to
perform (e.g., which RDDs need checkpoint and when to checkpoint them).
Moreover, to shorten the linage, iterative applications need to do
checkpoint frequently (e.g., every 10 iterations). As a result, checkpoint
is too heavy for iterative applications especially written by GraphX.
I'm wondering if there is an elegant way to solve the problem: shortening
the lineage and also saving the intermediate data/results in a lightweight
way.
Maybe we can develop a new API like checkpoint(StorageLevel), which has the
feature of both cache() and current checkpoint().
Examples:
The lineage is very long without checkpoint even in the first iteration in
GraphX job.
[Iter 1][DEBUG] (2) EdgeRDD[33] at RDD at EdgeRDD.scala:35
| EdgeRDD ZippedPartitionsRDD2[32] at zipPartitions at
ReplicatedVertexView.scala:114
| EdgeRDD MapPartitionsRDD[12] at mapPartitionsWithIndex at
EdgeRDD.scala:169
| MappedRDD[11] at map at Graph.scala:392
| MappedRDD[10] at distinct at KCoreCommonDebug.scala:115
| ShuffledRDD[9] at distinct at KCoreCommonDebug.scala:115
+-(2) MappedRDD[8] at distinct at KCoreCommonDebug.scala:115
| FilteredRDD[7] at filter at KCoreCommonDebug.scala:112
| MappedRDD[6] at map at KCoreCommonDebug.scala:102
| MappedRDD[5] at repartition at KCoreCommonDebug.scala:101
| CoalescedRDD[4] at repartition at KCoreCommonDebug.scala:101
| ShuffledRDD[3] at repartition at KCoreCommonDebug.scala:101
+-(2) MapPartitionsRDD[2] at repartition at KCoreCommonDebug.scala:101
| D:\graphData\verylarge.txt MappedRDD[1] at textFile at
KCoreCommonDebug.scala:100
| D:\graphData\verylarge.txt HadoopRDD[0] at textFile at
KCoreCommonDebug.scala:100
| ShuffledRDD[31] at partitionBy at ReplicatedVertexView.scala:112
+-(2) ReplicatedVertexView.updateVertices - shippedVerts false false
(broadcast) MapPartitionsRDD[30] at mapPartitions at VertexRDD.scala:347
| VertexRDD ZippedPartitionsRDD2[28] at zipPartitions at
VertexRDD.scala:174
| VertexRDD, VertexRDD MapPartitionsRDD[18] at mapPartitions at
VertexRDD.scala:441
| MapPartitionsRDD[17] at mapPartitions at VertexRDD.scala:457
| ShuffledRDD[16] at ShuffledRDD at RoutingTablePartition.scala:36
+-(2) VertexRDD.createRoutingTables - vid2pid (aggregation)
MapPartitionsRDD[15] at mapPartitions at VertexRDD.scala:452
| EdgeRDD MapPartitionsRDD[12] at mapPartitionsWithIndex at
EdgeRDD.scala:169
| MappedRDD[11] at map at Graph.scala:392
| MappedRDD[10] at distinct at KCoreCommonDebug.scala:115
| ShuffledRDD[9] at distinct at KCoreCommonDebug.scala:115
+-(2) MappedRDD[8] at distinct at KCoreCommonDebug.scala:115
| FilteredRDD[7] at filter at KCoreCommonDebug.scala:112
| MappedRDD[6] at map at KCoreCommonDebug.scala:102
| MappedRDD[5] at repartition at KCoreCommonDebug.scala:101
| CoalescedRDD[4] at repartition at KCoreCommonDebug.scala:101
| ShuffledRDD[3] at repartition at KCoreCommonDebug.scala:101
+-(2) MapPartitionsRDD[2] at repartition at
KCoreCommonDebug.scala:101
| D:\graphData\verylarge.txt MappedRDD[1] at textFile at
KCoreCommonDebug.scala:100
| D:\graphData\verylarge.txt HadoopRDD[0] at textFile at
KCoreCommonDebug.scala:100
| VertexRDD ZippedPartitionsRDD2[26] at zipPartitions at
VertexRDD.scala:200
| VertexRDD, VertexRDD MapPartitionsRDD[18] at mapPartitions at
VertexRDD.scala:441
| MapPartitionsRDD[17] at mapPartitions at VertexRDD.scala:457
| ShuffledRDD[16] at ShuffledRDD at RoutingTablePartition.scala:36
+-(2) VertexRDD.createRoutingTables - vid2pid (aggregation)
MapPartitionsRDD[15] at mapPartitions at VertexRDD.scala:452
| EdgeRDD MapPartitionsRDD[12] at mapPartitionsWithIndex at
EdgeRDD.scala:169
| MappedRDD[11] at map at Graph.scala:392
| MappedRDD[10] at distinct at KCoreCommonDebug.scala:115
| ShuffledRDD[9] at distinct at KCoreCommonDebug.scala:115
+-(2) MappedRDD[8] at distinct at KCoreCommonDebug.scala:115
| FilteredRDD[7] at filter at KCoreCommonDebug.scala:112
| MappedRDD[6] at map at KCoreCommonDebug.scala:102
| MappedRDD[5] at repartition at KCoreCommonDebug.scala:101
| CoalescedRDD[4] at repartition at KCoreCommonDebug.scala:101
| ShuffledRDD[3] at repartition at KCoreCommonDebug.scala:101
+-(2) MapPartitionsRDD[2] at repartition at
KCoreCommonDebug.scala:101
| D:\graphData\verylarge.txt MappedRDD[1] at textFile at
KCoreCommonDebug.scala:100
| D:\graphData\verylarge.txt HadoopRDD[0] at textFile at
KCoreCommonDebug.scala:100
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[24] at
zipPartitions at VertexRDD.scala:301
| VertexRDD, VertexRDD MapPartitionsRDD[18] at mapPartitions at
VertexRDD.scala:441
| MapPartitionsRDD[17] at mapPartitions at VertexRDD.scala:457
| ShuffledRDD[16] at ShuffledRDD at RoutingTablePartition.scala:36
+-(2) VertexRDD.createRoutingTables - vid2pid (aggregation)
MapPartitionsRDD[15] at mapPartitions at VertexRDD.scala:452
| EdgeRDD MapPartitionsRDD[12] at mapPartitionsWithIndex at
EdgeRDD.scala:169
| MappedRDD[11] at map at Graph.scala:392
| MappedRDD[10] at distinct at KCoreCommonDebug.scala:115
| ShuffledRDD[9] at distinct at KCoreCommonDebug.scala:115
+-(2) MappedRDD[8] at distinct at KCoreCommonDebug.scala:115
| FilteredRDD[7] at filter at KCoreCommonDebug.scala:112
| MappedRDD[6] at map at KCoreCommonDebug.scala:102
| MappedRDD[5] at repartition at KCoreCommonDebug.scala:101
| CoalescedRDD[4] at repartition at KCoreCommonDebug.scala:101
| ShuffledRDD[3] at repartition at KCoreCommonDebug.scala:101
+-(2) MapPartitionsRDD[2] at repartition at
KCoreCommonDebug.scala:101
| D:\graphData\verylarge.txt MappedRDD[1] at textFile at
KCoreCommonDebug.scala:100
| D:\graphData\verylarge.txt HadoopRDD[0] at textFile at
KCoreCommonDebug.scala:100
| ShuffledRDD[23] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[22] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[12] at mapPartitionsWithIndex at
EdgeRDD.scala:169
| MappedRDD[11] at map at Graph.scala:392
| MappedRDD[10] at distinct at KCoreCommonDebug.scala:115
| ShuffledRDD[9] at distinct at KCoreCommonDebug.scala:115
+-(2) MappedRDD[8] at distinct at KCoreCommonDebug.scala:115
| FilteredRDD[7] at filter at KCoreCommonDebug.scala:112
| MappedRDD[6] at map at KCoreCommonDebug.scala:102
| MappedRDD[5] at repartition at KCoreCommonDebug.scala:101
| CoalescedRDD[4] at repartition at KCoreCommonDebug.scala:101
| ShuffledRDD[3] at repartition at KCoreCommonDebug.scala:101
+-(2) MapPartitionsRDD[2] at repartition at
KCoreCommonDebug.scala:101
| D:\graphData\verylarge.txt MappedRDD[1] at textFile at
KCoreCommonDebug.scala:100
| D:\graphData\verylarge.txt HadoopRDD[0] at textFile at
KCoreCommonDebug.scala:100
Checkpoint can shorten the lineage. However, the lineage is still very long
even with checkpoint. For example,
[Iter 3][DEBUG] (2) VertexRDD[71] at RDD at VertexRDD.scala:58
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
[Iter 4][DEBUG] (2) VertexRDD[92] at RDD at VertexRDD.scala:58
| VertexRDD ZippedPartitionsRDD2[91] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[82] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[89] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[82] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[88] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[87] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[84] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[76] at zipPartitions at
ReplicatedVertexView.scala:114
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[75] at partitionBy at ReplicatedVertexView.scala:112
+-(2) ReplicatedVertexView.updateVertices - shippedVerts true true
(broadcast) MapPartitionsRDD[74] at mapPartitions at VertexRDD.scala:347
| VertexRDD ZippedPartitionsRDD2[72] at zipPartitions at
VertexRDD.scala:174
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
[Iter 5][DEBUG] (2) VertexRDD[113] at RDD at VertexRDD.scala:58
| VertexRDD ZippedPartitionsRDD2[112] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[103] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[91] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[82] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[89] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[82] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[88] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[87] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[84] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[76] at zipPartitions at
ReplicatedVertexView.scala:114
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[75] at partitionBy at ReplicatedVertexView.scala:112
+-(2) ReplicatedVertexView.updateVertices - shippedVerts true true
(broadcast) MapPartitionsRDD[74] at mapPartitions at VertexRDD.scala:347
| VertexRDD ZippedPartitionsRDD2[72] at zipPartitions at
VertexRDD.scala:174
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[110] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[103] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[91] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[82] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[89] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[82] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at zipPartitions
at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[88] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[87] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[84] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[76] at zipPartitions at
ReplicatedVertexView.scala:114
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[75] at partitionBy at ReplicatedVertexView.scala:112
+-(2) ReplicatedVertexView.updateVertices - shippedVerts true true
(broadcast) MapPartitionsRDD[74] at mapPartitions at VertexRDD.scala:347
| VertexRDD ZippedPartitionsRDD2[72] at zipPartitions at
VertexRDD.scala:174
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[109] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[108] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[105] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[97] at zipPartitions at
ReplicatedVertexView.scala:114
| EdgeRDD MapPartitionsRDD[84] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[76] at zipPartitions at
ReplicatedVertexView.scala:114
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[75] at partitionBy at ReplicatedVertexView.scala:112
+-(2) ReplicatedVertexView.updateVertices - shippedVerts true true
(broadcast) MapPartitionsRDD[74] at mapPartitions at VertexRDD.scala:347
| VertexRDD ZippedPartitionsRDD2[72] at zipPartitions at
VertexRDD.scala:174
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[96] at partitionBy at ReplicatedVertexView.scala:112
+-(2) ReplicatedVertexView.updateVertices - shippedVerts true true
(broadcast) MapPartitionsRDD[95] at mapPartitions at VertexRDD.scala:347
| VertexRDD ZippedPartitionsRDD2[93] at zipPartitions at
VertexRDD.scala:174
| VertexRDD MapPartitionsRDD[82] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| VertexRDD ZippedPartitionsRDD2[91] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[82] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[89] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[82] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[66] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[88] at ShuffledRDD at MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg MapPartitionsRDD[87] at
mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[84] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[76] at zipPartitions at
ReplicatedVertexView.scala:114
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
| ShuffledRDD[75] at partitionBy at
ReplicatedVertexView.scala:112
+-(2) ReplicatedVertexView.updateVertices - shippedVerts true
true (broadcast) MapPartitionsRDD[74] at mapPartitions at
VertexRDD.scala:347
| VertexRDD ZippedPartitionsRDD2[72] at zipPartitions at
VertexRDD.scala:174
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD ZippedPartitionsRDD2[70] at zipPartitions at
VertexRDD.scala:200
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| VertexRDD, GraphOps.degrees ZippedPartitionsRDD2[68] at
zipPartitions at VertexRDD.scala:301
| VertexRDD MapPartitionsRDD[61] at mapPartitions at
VertexRDD.scala:127
| VertexRDD ZippedPartitionsRDD2[47] at zipPartitions at
VertexRDD.scala:200
| CheckpointRDD[56] at apply at List.scala:318
| ShuffledRDD[67] at ShuffledRDD at
MessageToPartition.scala:31
+-(2) GraphImpl.mapReduceTriplets - preAgg
MapPartitionsRDD[66] at mapPartitions at GraphImpl.scala:192
| EdgeRDD MapPartitionsRDD[63] at mapPartitions at
EdgeRDD.scala:85
| EdgeRDD ZippedPartitionsRDD2[53] at zipPartitions at
ReplicatedVertexView.scala:114
| CheckpointRDD[57] at apply at List.scala:318
