----- Original Message ----- > Great post, Taras! > > Per IRC conversations, we'd like to move subsequent discussion of > actions into a meeting so we can more quickly arrive at a resolution. > > Please meet in Gregory Szorc's Vidyo Room at 1400 PDT Tuesday, April > 30. > That's 2200 UTC. Apologies to the European and east coast crowds. If > you'll miss it because it's too late, let me know and I'll consider > moving it. > > https://v.mozilla.com/flex.html?roomdirect.html&key=yJWrGKmbSi6S
Did someone post a summary of this meeting? Is there a link to share? Lawrence > > On 4/29/13 10:51 AM, Taras Glek wrote: > > * How to robustly write/update small datasets? > > > > #3 above is it for small datasets. The correct way to do this is to > > write blobs of JSON to disk. End of discussion. > > > > Writes of data <= ~64K should just be implemented as atomic > > whole-file > > read/write operations. Those are almost always single blocks on > > disk. > > > > Writing a whole file at once eliminates risk of data corruption. > > Incremental updates are what makes sqlite do the WAL/fsync/etc > > dance > > that causes much of the slowness. > > > > We invested a year worth of engineering effort into a pure-js IO > > library > > to facilitate efficient application-level IO. See OS.File docs, eg > > https://developer.mozilla.org/en-US/docs/JavaScript_OS.File/OS.File_for_the_main_thread > > > > > > As you can see from above examples, manual IO is not scary > > > > If one is into convenience APIs, one can create arbitrary > > json-storage > > abstractions in ~10lines of code. > > > > * What about writes > 64K? > > Compression gives you 5-10x reduction of json. > > https://bugzilla.mozilla.org/show_bug.cgi?id=846410 > > Compression also means that your read-throughput is up to 5x better > > too. > > > > > > * What about fsync-less writes? > > Many log-type performance-sensitive data-storage operations are ok > > with > > lossy appends. By lossy I mean "data will be lost if there is a > > power > > outage within a few seconds/minutes of write", consistency is still > > important. For this one should create a directory and write out log > > entries as checksummed individual files...but one should really use > > compression(and get checksums for free). > > https://bugzilla.mozilla.org/show_bug.cgi?id=846410 is about > > facilitating such an API. > > > > Use-cases here: telemetry saved-sessions, FHR session-statistics. > > > > * What about large datasets? > > These should be decided on a case-by-case basis. Universal > > solutions > > will always perform poorly in some dimension. > > > > * What about indexeddb? > > IDB is overkill for simple storage needs. It is a restrictive > > wrapper > > over an SQLite schema. Perhaps some large dataset (eg an > > addressbook) is > > a good fit for it. IDB supports filehandles to do raw IO, but that > > still > > requires sqlite to bootstrap, doesn't support compression, etc. > > IDB also makes sense as a transitional API for web due to the need > > to > > move away from DOM Local Storage... > > > > * Why isn't there a convenience API for all of the above > > recommendations? > > Because speculatively landing APIs that anticipate future consumers > > is > > risky, results in over-engineering and unpleasant surprises...So > > give us > > use-cases and we(ie Yoric) will make them efficient. > > > > Taras > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
