Does this assume that Spark is running on the same hosts as HDFS? Hence does increasing the latency affects the network latency on Hadoop nodes as well in your tests?
The best network results are achieved when Spark nodes share the same hosts as Hadoop or they happen to be on the same subnet. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 22 September 2016 at 14:54, gusiri <dreame...@gmail.com> wrote: > Hi, > > When I increase the network latency among spark nodes, > > I see compute time (=executor computing time in Spark Web UI) also > increases. > > In the graph attached, left = latency 1ms vs right = latency 500ms. > > Is there any communication between worker and driver/master even 'during' > executor computing? or any idea on this result? > > > <http://apache-spark-user-list.1001560.n3.nabble.com/ > file/n27779/Screen_Shot_2016-09-21_at_5.png> > > > > > > Thank you very much in advance. > > //gusiri > > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Is-executor-computing-time- > affected-by-network-latency-tp27779.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
