On Mon, Oct 30, 2017 at 11:33:36PM -0700, Dan Hitt wrote:
I plan to produce some time series data and i would like it to be in a form that makes it easy or effortless for pre-existing programs to plot or even process further.The data would consist of a few hundred to a few thousand time points, and for each time point there would be a half-dozen to a dozen values. So conceptually it would be something like a csv file with a few hundred to a few thousand rows, and a half-dozen to a dozen columns. However, a csv file doesn't seem exactly right, as i would like the files to include some embedded metadata (ideally in some form that at least in principle could be digested by another program). I suppose i could have some comment lines somehow, but that feels pretty ad hoc. But maybe i'm being overly fastidious. Something like a RIFF file format (like WAVE) might be better, but not sure what it would be. For the plotting or processing software i'd want to use something like octave or python/matplotlib or a derivative. I googled around for time series file formats, and i didn't see exactly what i was looking for. So i'd appreciate any advice or suggestions, and thanks in advance!
The lowest-common-denominator here is usually RRD [1], the Round-Robin Database. This will give you a fixed-size file containing various "buckets" for data at specific intervals (for example every 5 minutes for a day, every hour for a month and every day for a year). Lots of programs can read and write RRD because it's been stable for ages.
If you want something a bit more new/capable/interesting, there's whisper [2], the TSDB for carbon/graphite. Or OpenTSDB [3], or InfluxDB [4] or Prometheus [5].
[1] https://oss.oetiker.ch/rrdtool/index.en.html [2] http://graphite.wikidot.com/whisper [3] http://opentsdb.net [4] https://www.influxdata.com [5] https://prometheus.io
dan
-- For more information, please reread.
signature.asc
Description: PGP signature
