I mostly use Amazon S3 for reading input data and writing output data for my spark jobs. I want to know the numbers of bytes read & written by my job from S3.
In hadoop, there are FileSystemCounters for this, is there something similar in spark ? If there is, can you please guide me on how to use it ? I saw there are some read/write metrics in TaskMetrics.scala. Is there a way to get this by specifying a DataReadMethod in TaskMetrics.scala ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Read-write-metrics-for-jobs-which-use-S3-tp23356.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
