This does sound like a nice feature, both per-job and per-taskmanager bytes written to and read from disk.
On Fri, Dec 9, 2016 at 8:51 AM, Chesnay Schepler <ches...@apache.org> wrote: > We do not measure how much data we are spilling to disk. > > > On 09.12.2016 14:43, Fabian Hueske wrote: > > Hi, > > the heap mem usage should be available via Flink's metrics system. > Not sure if that also captures spilled data. Chesnay (in CC) should know > that. > > If the spilled data is not available as a metric, you can try to write a > small script that monitors the directories to which Flink spills (Config > parameter: taskmanager.tmp.dirs [1]). > The script would repeatedly list all files and keep for each file the max > size (files are deleted once the are not used anymore). This is not super > precise but might be good enough. > > Hope this helps, > Fabian > > [1] https://ci.apache.org/projects/flink/flink-docs- > release-1.1/setup/config.html#jobmanager-amp-taskmanager > > 2016-12-09 14:12 GMT+01:00 otherwise777 <wou...@onzichtbaar.net>: > >> Currently i'm doing some analysis for some algorithms that i use in Flink, >> I'm interested in the Space and time it takes to execute them. For the >> Time >> i used getNetRuntime() in the executionenvironment, but I have no idea how >> to analyse the amount of space an algorithm uses. >> Space can mean different things here, like Heap space, disk space, overal >> memory or allocated memory. I would like to analyze some of these. >> >> >> >> -- >> View this message in context: http://apache-flink-user-maili >> ng-list-archive.2336050.n4.nabble.com/How-to-analyze-spac >> e-usage-of-Flink-algorithms-tp10555.html >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> > > >
