Hi hackers,
I'd like to propose a new data type for PostgreSQL: SLIM (Structured
Lightweight Interchange Markup), a compact alternative to JSONB that
achieves 17-62% storage reduction while remaining human-readable.
== The Problem ==
JSONB is great, but has inherent inefficiencies:
- Redundant field names in arrays of objects
- Verbose syntax (quotes, colons, brackets)
- Boolean representation uses 4-5 bytes each
- No format-level compression
For databases with millions of JSON documents, this adds up.
== SLIM Format ==
JSON:
{"users":[{"id":1,"name":"Alice","active":true},{"id":2,"name":"Bob","active":false}]}
SLIM: {users:|2|id#,name,active?|1,Alice,T|2,Bob,F}
Savings: 45%
Key optimizations:
- Table format for arrays: schema once, data in rows
- Type markers: # (number), ? (boolean), ! (null)
- Compact booleans: T/F instead of true/false
- No quotes for simple strings
== Real-World Benchmarks ==
Tested on a production TimescaleDB database (74.9 million rows, 74 GB):
Format | Sample | JSON | SLIM | Savings
--------------------------|-----------|---------|---------|--------
Object (token_metrics) | 100K rows | 16 MB | 13 MB | 18.7%
Object (wallet_activity) | 100K rows | 20 MB | 16 MB | 17.0%
Table format (arrays) | 10K rows | 580 KB | 222 KB | 61.7%
Extrapolated on 74 GB: 12.5-45.9 GB savings depending on data structure.
== Implementation ==
I've built a working PostgreSQL extension (pg_slim) that provides:
- Native SLIM data type with TOAST support
- Implicit casts to/from JSONB and TEXT
- Full operator support: ->, ->>, @>, <@, ?, ?|, ?&
- B-tree and hash indexing
- Comparison operators for sorting
Example usage:
CREATE TABLE documents (id SERIAL PRIMARY KEY, data SLIM);
INSERT INTO documents (data) VALUES ('{name:Alice,age:#30}'::slim);
INSERT INTO documents (data) VALUES
(slim_encode('{"name":"Bob"}'::jsonb));
SELECT data->>'name' FROM documents; -- Works like JSONB
SELECT * FROM documents WHERE data @> '{name:Alice}'::slim;
The extension compiles and passes tests on PostgreSQL 15.
== Questions for Discussion ==
1. Is there community interest in a SLIM type?
2. Should this remain an extension or be considered for contrib/core?
3. Which index types should be prioritized (GIN support is planned)?
4. Any concerns about the format design?
== Links ==
- Extension: https://github.com/matteuccimarco/pg-slim
- Full RFC:
https://github.com/matteuccimarco/pg-slim/blob/main/RFC-SLIM-TYPE.md
- SLIM spec: https://github.com/matteuccimarco/slim-protocol-core
- Benchmarks:
https://github.com/matteuccimarco/pg-slim/blob/main/BENCHMARK-RESULTS.md
Looking forward to your feedback.
Best regards,
Marco Matteucci
--
Marco Matteucci
+39 340 7063047
Vuoi un appuntamento con me? Clicca qui
<https://calendar.app.google/SaCBAJfiJywVfQJb7>
