On Tuesday, 15 November 2016 07:51:21 GMT Maarten Wullink wrote: > Did you consider columnar storage formats like Apache Parquet + Snappy > compression? It would be an interesting to compare this to cbor-dns. > > https://parquet.apache.org/ > https://github.com/google/snappy
I did look briefly at Snappy to compare it to gzip and xz, but our focus at the time was on the smallest possible output and so Snappy wasn't really what we wanted. We didn't look directly at Parquet, but Tomáš Gavenčiak from CZ.nic did some experiments with us on a simple columnar storage format. After compression, the output sizes did get close to the C-DNS sizes, but C-DNS retained a small advantage in post-compression file size so we stuck with the current design. My records indicate an input 370Mb PCAP produced a 12.34Mb output with the columnar output after xz compression, versus C-DNS at 10.34Mb. But you needed good compression to get that close - with gzip or lz4, the columnar final size was more like 1.6-18x C-DNS. I did not look at compression resources. I believe that Tomáš is using Parquet in his project, which has similarities but slightly different requirements (and so trade-offs) to our brief. -- Jim Hague - j...@sinodun.com Never trust a computer you can't lift. _______________________________________________ DNSOP mailing list DNSOP@ietf.org https://www.ietf.org/mailman/listinfo/dnsop
