Hi all, I'm doing some tuning work on our rrdtool setup: we'd like to get a bit more disk i/o overhead. I have two questions/assumptions I'd love to get a sanity check on.
First, some background on our setup: * Hardware ** 16 2.27 GHZ CPUs ** 16 GB of RAM ** Dedicated to RRDTool - no other services are installed on this server. ** SSD Drive * RRDTool Config ** RRDtool 1.4.5 ** 20k RRD Files, each 539K in size ** 20 data sources per-file ** 18 RRAs per file (Min/Max/Average of each: 5 minutes for 6 hours, 10 minutes for 12 hours, 1 hour for 3 days, 6 hours for one week, 12 hours for 2 weeks, 1 day for 2 years) * Performance ** Typically, almost no reads (showing buffer cache is working) ** 315 writes/sec | 4 MB/sec Second, what we're planning: I've load tested several different rrdtool file configurations and what I saw aligned with the behavior in this paper by David Plonka: http://www.usenix.org/event/lisa07/tech/full_papers/plonka/plonka_html/. If I'm summarizing the paper correctly, the 2 RRDTool file configurations that impact I/O activity are (1) # of data sources and (2) # of RRAs. The number of rows in an RRA does not impact performance. * Reducing data sources per-file to 5: Most of our fields are reserved for later use (since you can't add a new data source to an existing file via an rrdtool command). Instead, we're going to start with 5 and manually add new data sources via rrdtool dump / restore. It's rare to add new data sources to a file. * Reducing the number of RRAs to 7 (Adding LAST of one-minute data for 60 minutes, extending 5-minutes for 1 week, and dropping all other RRAs but daily). When needing to graph data over a several day period, I'm planning on using the --step option with a 60 minute or greater value to limit the number of data points returned. I haven't tested this yet, but I'm assuming reading the many more rows of 5-minute storage may be the primary performance penalty of the new RRD setup. And finally, the RRD performance pieces I'm having trouble understanding: Memory Sizing --- The "Tuning RRD" wiki page states: "To keep the header (4k) and the active block of at least one RRA (4k) in memory, you need 8k per RRD file." For our current configuration, we have 20 data sources and 18 RRAs. >From the illustration on the page, it looks like our DS Header storage space would be: 4k per data source * 20 data sources = 80k Is my maths legit? For the RRA Header, is it (1) one per-RRA independent of the number of data sources or (2) one per-RRA * the number of data sources? If one-per RRA independent of data sources: 18 RRAs * 4k = 72k If one-per RRA for each data source: 18 RRA * 20 data sources = 360k Assuming my data source math checks out, this means either 152k or 440k must be stored in the buffer cache for the entire header. free -m shows that we have 5,119 MB cached: with nearly 20k rrd files, it looks like the header contains one RRA for each data source. Does this check out? High I/O Wait during peak aggregation periods --- At 00:00 UTC, we see our peak I/O Wait. At this time, reads increase to 11 reads/sec (typically we have almost no read activity) and writes increase to 700 writes/sec (typically around 315 writes/sec). One observation: even though our read throughput is far less than our write throughput, this increase in reads seems to have a dramatic impact on I/O Wait. I can see why there is an increase in writes: currently, we have 18 RRAs across 20 data sources that need to append data to the file at this time. I'm a bit confused why the read activity increases dramatically: * We have 16 GB of RAM * It looks like < 5 GB of that is used by the buffer cache My low-level Linux fu is poor: can anyone give some insight into the read activity at these peak aggregation times? RRDTool is am amazing piece of work: it's a testament to how well it works that we haven't needed to investigate performance significantly in the 3+ years we've been using it. Cheers, Derek -- Derek Haynes Scout Web Monitoring and Reporting ~ http://scoutapp.com Blog ~ http://blog.scoutapp.com 415.871.7979 _______________________________________________ rrd-users mailing list [email protected] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
