At 2014-08-04 20:52:26 +0800, Bin <wubin_phi...@126.com> wrote: > I wonder how spark parameters, e.g., number of paralellism, affect Pregel > performance? Specifically, sendmessage, mergemessage, and vertexprogram? > > I have tried label propagation on a 300,000 edges graph, and I found that no > paralellism is much faster than 5 or 500 paralellism.
Increasing the level of parallelism will increase storage overhead (because each vertex will need to be replicated to more edge partitions to form the triplets) and will also increase communication. Unless there's something to be gained from higher parallelism, this will worsen performance. Additionally, going from no parallelism to some parallelism will incur the extra cost of task communication via shuffles. Parallelism has two benefits: it allows edge scans and aggregations to proceed in parallel, and it enables the graph to be stored across many machines. For small graphs, the slight performance gain due to parallelism is vastly outweighed by the cost of inter-process communication and shuffling to disk, and distributed storage is not necessary since the graph fits on a single machine. There are single-machine graph processing systems such as X-Stream [1] and GraphChi [2] that optimize performance for these kinds of graphs. However, parallelism becomes necessary for larger graphs with hundreds of millions of edges or large amounts of associated vertex and edge data. GraphX is designed for this scale of data. Ankur [1] http://infoscience.epfl.ch/record/188535/files/paper.pdf [2] http://graphlab.org/projects/graphchi.html --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
