The REST interface does also provide metrics about the number of records and the size of the input and output of all tasks. See: - /jobs/<jobid>/vertices/<vertexid> - /jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt> in https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html#details-of-a-running-or-completed-job
However, not all of this data is going over the network because some tasks can be locally connected. Best, Fabian 2016-01-29 8:50 GMT+01:00 Philip Lee <philjj...@gmail.com>: > Thanks, > > Is there any way to measure shuffle data (read and write) on Flink or > Dashboard? > > I did not find the network usage metric in it. > > Best, > Phil > > On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske <fhue...@gmail.com> wrote: > >> You can start a job and then periodically request and store information >> about the running job and vertices from using corresponding REST calls [1]. >> The data will be in JSON format. >> After the job finished, you can stop requesting data. >> >> Next you parse the JSON, extract the information you need and give it to >> some plotting library. >> As I said, it is not possible to pass this data back into Flink's >> dashboard, but you have to process and plot it yourself. >> >> Best, Fabian >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs >> >> >> >> 2016-01-25 16:15 GMT+01:00 Philip Lee <philjj...@gmail.com>: >> >>> Hello, >>> >>> According to >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html, >>> I cannot retrieve the job history from Dashboard after turnning off JM. >>> >>> But as Fabian mentioned here, >>> "However, you can query all stats that are displayed by the dashboard >>> via a REST API [1] while the JM is running and save them yourself. This way >>> you can analyze the data also after the JM was stopped" could you explain >>> about this sentence in detail. >>> >>> I want to evaluate timeline view of each function after a job is done. >>> >>> Thanks, >>> Phil >>> >> >> >
