I'd like to start a discussion about the state of storage in Gecko. Currently when you are writing a feature that needs to store data, you have roughly 3 choices:
1) Preferences 2) SQLite 3) Manual file I/O Preferences are arguably the easiest. However, they have a number of setbacks: a) Poor durability guarantees. See bugs 864537 and 849947 for real-life issues. tl;dr writes get dropped! b) Integers limited to 32 bit (JS dates overflow b/c milliseconds since Unix epoch). c) I/O is synchronous. d) The whole method for saving them to disk is kind of weird. e) The API is awkward. See Preferences.jsm for what I'd consider a better API. f) Doesn't scale for non-trivial data sets. g) Clutters about:config (all preferences aren't config options). We have SQLite. You want durability: it's your answer. However, it too has setbacks: a) It eats I/O operations for breakfast. Multiple threads. Lots of overhead compared to prefs. (But hard to lose data.) b) By default it's not configured for optimal performance (you need to enable the WAL, muck around with other PRAGMA). c) Poor schemas can lead to poor performance. d) It's often overkill. e) Storage API has many footguns (use Sqlite.jsm to protect yourself). f) Lots of effort to do right. Auditing code for 3rd party extensions using SQLite, many of them aren't doing it right. And if one of those pre-built solutions doesn't offer what you need, you can roll your own with file I/O. But that also has setbacks: a) You need to roll your own. (How often do I flush? Do I use many small files or fewer large files? Different considerations for mobile (slow I/O) vs desktop?) b) You need to roll your own. (Listing it twice because it's *really* annoying, especially for casual developers that just want to implement features - think add-on developers.) c) Easy to do wrong (excessive flushing/fsyncing, too many I/O operations, inefficient appends, poor choices for mobile, etc). d) Wheel reinvention. Atomic operations/transactions. Data marshaling. etc. I believe there is a massive gap between the easy-but-not-ready-for-prime-time preferences and the-massive-hammer-solving-the-problem-you-don't-have-and-introducing-many-new-ones SQLite. Because this gap is full of unknowns, I'm arguing that developers tend to avoid it and use one of the extremes instead. And, the result is features that have poor durability and/or poor performance. Not good. What's worse is many developers (including myself) are ignorant of many of these pitfalls. Yes, we have code review for core features. But code review isn't perfect and add-ons likely aren't subjected to the same level of scrutiny. The end result is the same: Firefox isn't as awesome as it could be. I think there is an opportunity for Gecko to step in and provide a storage subsystem that is easy to use, somewhere between preferences and SQLite in terms of durability and performance, and "just works." I don't think it matters how it is implemented under the hood. If this were to be built on top of SQLite, I think that would be fine. But, please don't make consumers worry about things like SQL, schema design, and PRAGMA statements. So, maybe I'm advocating a generic key-value store. Maybe something like DOM Storage? Maybe SQLite 4 (which is emphasizing key-value storage and speed)? Just... something. Please. Anyway, I just wanted to see if others have thought about this. Do others feel it is a concern? If so, can we formulate a plan to address it? Who would own this? Gregory _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
