On 03/05/2023 16:51, Marcus Müller wrote:
Do agree, but really don't like CSV, too underspecified a format, too
many ways that comes back to bite you (aside from a thousand SDR users
writing emails that their PC can't keep up with writing a few MS/s of
CSV…)
I like CSV because you can hand your data files to someone who doesn't
have a complete suite of astrophysics tools, and they
can slurp it into Excel and play with it.
How important is plain-textness in your applications?
I (and many others in my community) tend to throw ad-hoc tools at data
from ad-hoc experiments. In the past, I used a lot
of AWK to post-process data, and these days, I use a lot of
Python. Text-based formats lend themselves well to this kind
of processing. Rates are quite low, typically. Like logging an
integrated power spectrum a few times a minute, for example.
There are other observing modes where text-based formats aren't quite so
obvious--like pulsar observations, where filterbank
outputs might be recorded at 10s of kHz, and then post-processed with
any of a number of pulsar tools.
In all of this, part of the "science" is extracted in "real-time" and
part in post-processing.
Best,
Marcus
