Andreas Gal wrote:
Preferences are as the name implies intended for preferences. There is no sane
use case for storing data in preferences. I would give any patch I come across
doing that an automatic sr- for poor taste and general insanity.
SQLite is definitely not cheap, and we should look at more suitable backends
for our storage needs, but done right off the main thread, its definitely the
saner way to go than (1).
While (2) is a foot-gun, (3) is a guaranteed foot-nuke. While its easy to use
sqlite wrong, its almost guaranteed that you get your own atomic storage file
use wrong, across our N platforms.
3) is not a footnuke. Atomic file IO is how we do prefs, other things.
We provide a writeAtomic API in OS.File. So long as your 'io
transactions' are within a single file writeAtomic is your friend. If
one needs io transactions across files, then one is in trouble indeed,
but that is not the case for 99% of our code.
Big advantage of 3 is that one does not pay the abstraction penalty of
heavier sqlite or indexeddb solutions. cost of a write+fsync + followup
read is easy to reason about.
One can also layer compression/checksums on top of atomic file IO
easily. This hard with a more complex storage layer(eg we can't compress
our sqlite place database even though that'd be a nice win).
Chrome is working on replacing sqlite with leveldb for indexeddb and most their
storage needs. Last time we looked it wasn't ready for prime time. Maybe it is
now. This might be the best option.
leveldb sounds nice, but the level of complexity there is overkill for
most of our usecases. Filesystems work well, complex abstractions on top
of them tend to be flakey.
Taras
ps. sorry for chiming in late. I was away doing outdoorsy vacation stuff
Thurs-Sun.
Andreas
On Apr 26, 2013, at 11:17 AM, Gregory Szorc<g...@mozilla.com> wrote:
I'd like to start a discussion about the state of storage in Gecko.
Currently when you are writing a feature that needs to store data, you
have roughly 3 choices:
1) Preferences
2) SQLite
3) Manual file I/O
Preferences are arguably the easiest. However, they have a number of
setbacks:
a) Poor durability guarantees. See bugs 864537 and 849947 for real-life
issues. tl;dr writes get dropped!
b) Integers limited to 32 bit (JS dates overflow b/c milliseconds since
Unix epoch).
c) I/O is synchronous.
d) The whole method for saving them to disk is kind of weird.
e) The API is awkward. See Preferences.jsm for what I'd consider a
better API.
f) Doesn't scale for non-trivial data sets.
g) Clutters about:config (all preferences aren't config options).
We have SQLite. You want durability: it's your answer. However, it too
has setbacks:
a) It eats I/O operations for breakfast. Multiple threads. Lots of
overhead compared to prefs. (But hard to lose data.)
b) By default it's not configured for optimal performance (you need to
enable the WAL, muck around with other PRAGMA).
c) Poor schemas can lead to poor performance.
d) It's often overkill.
e) Storage API has many footguns (use Sqlite.jsm to protect yourself).
f) Lots of effort to do right. Auditing code for 3rd party extensions
using SQLite, many of them aren't doing it right.
And if one of those pre-built solutions doesn't offer what you need, you
can roll your own with file I/O. But that also has setbacks:
a) You need to roll your own. (How often do I flush? Do I use many small
files or fewer large files? Different considerations for mobile (slow
I/O) vs desktop?)
b) You need to roll your own. (Listing it twice because it's *really*
annoying, especially for casual developers that just want to implement
features - think add-on developers.)
c) Easy to do wrong (excessive flushing/fsyncing, too many I/O
operations, inefficient appends, poor choices for mobile, etc).
d) Wheel reinvention. Atomic operations/transactions. Data marshaling. etc.
I believe there is a massive gap between the
easy-but-not-ready-for-prime-time preferences and
the-massive-hammer-solving-the-problem-you-don't-have-and-introducing-many-new-ones
SQLite. Because this gap is full of unknowns, I'm arguing that
developers tend to avoid it and use one of the extremes instead. And,
the result is features that have poor durability and/or poor
performance. Not good. What's worse is many developers (including
myself) are ignorant of many of these pitfalls. Yes, we have code review
for core features. But code review isn't perfect and add-ons likely
aren't subjected to the same level of scrutiny. The end result is the
same: Firefox isn't as awesome as it could be.
I think there is an opportunity for Gecko to step in and provide a
storage subsystem that is easy to use, somewhere between preferences and
SQLite in terms of durability and performance, and "just works." I don't
think it matters how it is implemented under the hood. If this were to
be built on top of SQLite, I think that would be fine. But, please don't
make consumers worry about things like SQL, schema design, and PRAGMA
statements. So, maybe I'm advocating a generic key-value store. Maybe
something like DOM Storage? Maybe SQLite 4 (which is emphasizing
key-value storage and speed)? Just... something. Please.
Anyway, I just wanted to see if others have thought about this. Do
others feel it is a concern? If so, can we formulate a plan to address
it? Who would own this?
Gregory
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
