Re: Storage in Gecko

On 5/16/13 2:26 AM, Robert Kaiser wrote: > David Rajchenbach-Teller schrieb: >> I'd even go as far as limiting it to 16kb. >> (possibly with a transition phase during which going above 16kb only >> prints warnings) > > I think most of us agree, but the problem is that apparently a number of > add-

Re: Storage in Gecko

David Rajchenbach-Teller schrieb: I'd even go as far as limiting it to 16kb. (possibly with a transition phase during which going above 16kb only prints warnings) I think most of us agree, but the problem is that apparently a number of add-ons rely on large prefs atm, so right now we did set t

Re: Storage in Gecko

I'd even go as far as limiting it to 16kb. (possibly with a transition phase during which going above 16kb only prints warnings) On 5/10/13 12:47 AM, Robert Kaiser wrote: > FWIW, we had a pretty high-ranking topcrash in > https://bugzilla.mozilla.org/show_bug.cgi?id=836263 where an add-on > stored

Re: Storage in Gecko

Gregory Szorc schrieb: Perhaps this should be advertised more, especially to the add-on community. Looking at about:config of my main profile, about 2/3 of my preferences are user set. There are hundreds of preferences apparently being used for key-value storage by add-ons (not to pick on one, bu

Re: Storage in Gecko

On 06/05/2013 18:41, David Dahl wrote:> KyotoCabinet might make a good backend for a new storage API: http://fallabs.com/kyotocabinet/ There is also a companion indexing engine: http://fallabs.com/kyototycoon/ Regards, David SQLite4 implements something very similar (log-structured merge d

Re: Storage in Gecko

On 06/05/13 20:12, David Dahl wrote: > That is unfortunate. The Kyoto-* tools are FAST and easy to use. I > wonder if the author would be willing to issue Mozilla a license that > is compatible with MPL? That would be the functional equivalent of relicensing under the MPL, which is a weaker copyle

Re: Storage in Gecko

quot; , "Taras Glek" , "David Rajchenbach-Teller" , "dev-platform" , "Lawrence Mandel" , "Kyle Huey" Sent: Monday, May 6, 2013 1:34:54 PM Subject: Re: Storage in Gecko On Mon, May 06, 2013 at 09:41:08AM -0700, David Dahl wrote: > KyotoCabinet mi

Re: Storage in Gecko

On Mon, May 06, 2013 at 09:41:08AM -0700, David Dahl wrote: > KyotoCabinet might make a good backend for a new storage API: > > http://fallabs.com/kyotocabinet/ It's released under the GPL, so it's MPL-incompatible, if I understand correctly. As for the "Kyoto Products Specific FOSS Library Link

Re: Storage in Gecko

lek" , "David Rajchenbach-Teller" , "dev-platform" , "Lawrence Mandel" Sent: Thursday, May 2, 2013 6:43:14 PM Subject: Re: Storage in Gecko On 5/2/2013 4:40 PM, Kyle Huey wrote: > > > On Thu, May 2, 2013 at 4:36 PM, Gregory Szorc <mailto:g...@mozil

Re: Storage in Gecko

If you guys don't need transactions and only a simple key+value storage, the new localstorage code be used. Only thing needed to make it work in a completely non-blocking way is to expose API telling the consumer that localstorage has loaded from disk and is fully cached in memory and thus acc

Re: Storage in Gecko

/flex.html?roomdirect.html&key=yJWrGKmbSi6S Did someone post a summary of this meeting? Is there a link to share? Notes at https://etherpad.mozilla.org/storage-in-gecko We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with specific consistency, flush

Re: Storage in Gecko

On 5/2/2013 7:43 PM, Gregory Szorc wrote: On 5/2/2013 4:40 PM, Kyle Huey wrote: On Thu, May 2, 2013 at 4:36 PM, Gregory Szorc > wrote: We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with specific consistency,

Re: Storage in Gecko

Did someone post a summary of this meeting? Is there a link to share? Notes at https://etherpad.mozilla.org/storage-in-gecko We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with specific consistency, flushing, etc guarantees. Clients would obtain a

Re: Storage in Gecko

i6S Did someone post a summary of this meeting? Is there a link to share? Notes at https://etherpad.mozilla.org/storage-in-gecko We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with specific consistency, flushing, etc guarantees. Clients would

Re: Storage in Gecko

gory Szorc" To: "Lawrence Mandel" Cc: "David Rajchenbach-Teller" , "Taras Glek" , "dev-platform" Sent: Friday, May 3, 2013 1:36:15 AM Subject: Re: Storage in Gecko On 5/2/2013 4:13 PM, Lawrence Mandel wrote: > > - Original Message - &g

Re: Storage in Gecko

On 5/2/2013 4:40 PM, Kyle Huey wrote: On Thu, May 2, 2013 at 4:36 PM, Gregory Szorc > wrote: We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with specific consistency, flushing, etc guarantees. Clients woul

Re: Storage in Gecko

On Thu, May 2, 2013 at 4:36 PM, Gregory Szorc wrote: > We seemed to converge on a (presumably C++-based) storage service that has > named branches/buckets with specific consistency, flushing, etc guarantees. > Clients would obtain a handle on a "branch," and perform basic I/O > operations, includ

Re: Storage in Gecko

meeting? Is there a link to share? Notes at https://etherpad.mozilla.org/storage-in-gecko We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with specific consistency, flushing, etc guarantees. Clients would obtain a handle on a "branch,"

Re: Storage in Gecko

- Original Message - > Great post, Taras! > > Per IRC conversations, we'd like to move subsequent discussion of > actions into a meeting so we can more quickly arrive at a resolution. > > Please meet in Gregory Szorc's Vidyo Room at 1400 PDT Tuesday, April > 30. > That's 2200 UTC. Apolo

Re: Storage in Gecko

On 26/04/2013 22:18, Andrew Sutherland wrote: Specifically, it does not look like IndexedDB's model; it still does a lot of fsync's in order to maintain the requisite SQL ACID semantics. Right, we can't expect miracles just by moving from SQLite3 to SQLite4, though it still uses an enhanced WA

Re: Storage in Gecko

On 4/30/2013 8:37 AM, Joshua Cranmer 🐧 wrote: On 4/30/2013 12:33 AM, Ehsan Akhgari wrote: On 2013-04-29 1:51 PM, Taras Glek wrote: Writes of data <= ~64K should just be implemented as atomic whole-file read/write operations. Those are almost always single blocks on disk. Writing a whole file a

Re: Storage in Gecko

Ehsan Akhgari Monday, April 29, 2013 22:33 On 2013-04-29 1:51 PM, Taras Glek wrote: * How to robustly write/update small datasets? #3 above is it for small datasets. The correct way to do this is to write blobs of JSON to disk. End of discussion. For an API

Re: Storage in Gecko

On 4/30/2013 12:33 AM, Ehsan Akhgari wrote: On 2013-04-29 1:51 PM, Taras Glek wrote: * How to robustly write/update small datasets? #3 above is it for small datasets. The correct way to do this is to write blobs of JSON to disk. End of discussion. For an API that is meant to be used by add-on

Re: Storage in Gecko

On 2013-04-29 1:51 PM, Taras Glek wrote: * How to robustly write/update small datasets? #3 above is it for small datasets. The correct way to do this is to write blobs of JSON to disk. End of discussion. For an API that is meant to be used by add-on authors, I'm afraid the situation is not as

Re: Storage in Gecko

Great post, Taras! Per IRC conversations, we'd like to move subsequent discussion of actions into a meeting so we can more quickly arrive at a resolution. Please meet in Gregory Szorc's Vidyo Room at 1400 PDT Tuesday, April 30. That's 2200 UTC. Apologies to the European and east coast crowds.

Re: Storage in Gecko

On 4/26/2013 1:17 PM, Gregory Szorc wrote: I'd like to start a discussion about the state of storage in Gecko. Currently when you are writing a feature that needs to store data, you have roughly 3 choices: 1) Preferences 2) SQLite 3) Manual file I/O One of the ongoing tasks I dabbled i

Re: Storage in Gecko

rsy vacation stuff Thurs-Sun. Andreas On Apr 26, 2013, at 11:17 AM, Gregory Szorc wrote: I'd like to start a discussion about the state of storage in Gecko. Currently when you are writing a feature that needs to store data, you have roughly 3 choices: 1) Preferences 2) SQLite 3) Ma

Re: Storage in Gecko

On 4/29/13 1:57 PM, Andrew McCreight wrote: A WebIDL callback interface thing could probably be set up to make calling from C++ into JS less awful, if somebody has a need. Sort of. IndexedDB keys are done on raw JS values ("any" in the IDL) because they want to tell apart Arrays from other o

Re: Storage in Gecko

- Original Message - > On Apr 27, 9:37 am, Mounir Lamouri wrote: > > Why? Wouldn't be the idea of such component to make sure it is > > usable > > from C++? > > Perhaps some day, but IndexedDB was always designed with JS in mind. > To use it you pass special JS dictionaries for options, c

Re: Storage in Gecko

So there is no general 'good for performance' way of doing IO. However I think most people who need this need to write small bits of data and there is a good way to do that. Gregory Szorc wrote: I'd like to start a discussion about the state of storage in Gecko. Current

Re: Storage in Gecko

On Apr 27, 9:37 am, Mounir Lamouri wrote: > Why? Wouldn't be the idea of such component to make sure it is usable > from C++? Perhaps some day, but IndexedDB was always designed with JS in mind. To use it you pass special JS dictionaries for options, clone things to/from JS objects, etc. Using it

Re: Storage in Gecko

On 26/04/13 20:42, bent wrote: > IndexedDB is our answer for this for JS... C++ folks are still pretty > much on their own! Why? Wouldn't be the idea of such component to make sure it is usable from C++? -- Mounir ___ dev-platform mailing list dev-platf

Re: Storage in Gecko

Hi, I once met a similar need: a simple key/storage API for my addon. I ended up writing a "SimpleStorage" module that uses an underlying SQLite database. I'm pretty sure I fell into most of the pitfalls of using SQLite without being a guru, but here's the link for posterity: - sample usage

Re: Storage in Gecko

On 27/04/2013 02:17, Gregory Szorc wrote: > I'd like to start a discussion about the state of storage in Gecko. > > Currently when you are writing a feature that needs to store data, you > have roughly 3 choices: > > 1) Preferences > 2) SQLite > 3) Manual file I

Re: Storage in Gecko

IndexedDB is our answer for this for JS... C++ folks are still pretty much on their own! IndexedDB handles indexing (hence the rather awkward name), transactions with abort/rollback, object-graph serialization (not just JSON), usage from multiple tabs/windows/components/processes simultaneously, d

"Experimental Technology" in Gecko (Re: Storage in Gecko)

On 4/26/2013 11:43 AM, Gregory Szorc wrote: Have you explored using IndexedDB? Not seriously. The "this is an experimental technology" warning on MDN is off-putting. The largest audience for MDN is web developers, so we put that warning on anything that's not ready for widespread use on the

Re: Storage in Gecko

Gregory Szorc wrote: c) I/O is synchronous. To be fair, the pref API is mostly reading and writing a big hashtable; few functions actually do any I/O. e) The API is awkward. Well, XPCOM was all the rage at the time. (Then again, so was RDF, and its dynamic bulk read API is still better

Re: Storage in Gecko

We filed a bug for this and I am working on the patch. Andreas Sent from Mobile. On Apr 26, 2013, at 16:06, Mounir Lamouri wrote: > On 26/04/13 11:17, Gregory Szorc wrote: >> Anyway, I just wanted to see if others have thought about this. Do >> others feel it is a concern? If so, can we formul

Re: Storage in Gecko

On 26/04/13 11:17, Gregory Szorc wrote: > Anyway, I just wanted to see if others have thought about this. Do > others feel it is a concern? If so, can we formulate a plan to address > it? Who would own this? As others, I believe that we should use IndexedDB for Gecko internal storage. I opened a b

Re: Storage in Gecko

I have to admit; I've been thinking about this as well… and considering the complexities involved with developing algorithms to deal with caches, fsyncs and concurrency I tend to lean toward NOT rolling your own, but instead look at what's out there. Event though it's oriented towards server us

Re: Storage in Gecko

On 4/26/13 11:17 AM, Gregory Szorc wrote: But, please don't make consumers worry about things like SQL, schema design, and PRAGMA statements. Ideally, yes. But I suspect there will never be a one-size-fits all solution, and so we should probably be clear about what it's appropriate/intended

Re: Storage in Gecko

On 04/26/2013 03:30 PM, Gregory Szorc wrote: However, before that happens, I'd like some consensus that IndexedDB is the best solution here. I'd especially like to hear what Performance thinks: I don't want to start creating a "preferred" storage solution without their blessing. If they have sugg

Re: Storage in Gecko

On 04/26/2013 03:21 PM, Dirkjan Ochtman wrote: Also, I wonder if SQLite 4 (which is more like a key-value store) SQLite 4 is not actually more like a key-value store. The underlying storage model used by the SQL-interface-that-is-the-interface changed from being a page-centric btree structur

Re: Storage in Gecko

We use IndexedDB extensively in a lot of the WebAPIs, see Contacts, Settings, SMS, MMS, Push, NetworkStats… Right now there's a lot of boilerplate[1] involved in setting up IndexedDB, and people end up duplicating a lot of the boilerplate code. It'd be great to see a more polished wrapper aroun

Re: Storage in Gecko

On 4/26/2013 12:10 PM, Benjamin Smedberg wrote: > On 4/26/2013 2:50 PM, Gavin Sharp wrote: >> On Fri, Apr 26, 2013 at 11:36 AM, Andreas Gal wrote: >>> Preferences are as the name implies intended for preferences. There >>> is no sane use case for storing data in preferences. I would give >>> any p

Re: Storage in Gecko

On Fri, Apr 26, 2013 at 12:10 PM, Benjamin Smedberg wrote: > I really hope the outcome of this discussion is that we end up storing > everything that isn't a true preference in some other datastore, and that is > an async-by-default datastore ;-) > With a pretty simple JSM wrapper, indexeddb coul

Re: Storage in Gecko

On Fri, Apr 26, 2013 at 8:17 PM, Gregory Szorc wrote: > Anyway, I just wanted to see if others have thought about this. Do > others feel it is a concern? If so, can we formulate a plan to address > it? Who would own this? AIUI the Performance team is experimenting with moving things into JSON fil

Re: Storage in Gecko

On 4/26/2013 3:07 PM, Justin Lebar wrote: The current level of flakiness in the IndexedDB test suite (especially on OSX) makes me concerned about what to expect if it starts getting heavier use across the various platforms. Is that just in the OOP tests, or everywhere? Mostly IPC. __

Re: Storage in Gecko

On Fri, Apr 26, 2013 at 11:57 AM, Ryan VanderMeulen wrote: > The current level of flakiness in the IndexedDB test suite (especially on > OSX) makes me concerned about what to expect if it starts getting heavier > use across the various platforms. Of the 24 open intermittent failure bugs in the I

Re: Storage in Gecko

Resending to list. On Fri, Apr 26, 2013 at 12:02 PM, Gregory Szorc wrote: > On 4/26/2013 11:52 AM, Kyle Huey wrote: > > Could you please point me at a "good" implementation of a Gecko consumer > of IndexedDB? If you don't know which are good, an MXR search URL will > suffice :) > I haven't look

Re: Storage in Gecko

On 4/26/2013 2:50 PM, Gavin Sharp wrote: On Fri, Apr 26, 2013 at 11:36 AM, Andreas Gal wrote: Preferences are as the name implies intended for preferences. There is no sane use case for storing data in preferences. I would give any patch I come across doing that an automatic sr- for poor tast

Re: Storage in Gecko

> The current level of flakiness in the IndexedDB test suite (especially on > OSX) makes me concerned about what to expect if it starts getting heavier > use across the various platforms. Is that just in the OOP tests, or everywhere? ___ dev-platform mai

Re: Storage in Gecko

On 4/26/2013 2:52 PM, Kyle Huey wrote: Yeah that's not accurate. It's pretty solid now. It's the storage backend for everything in b2g for instance ... and it's not going to see any changes that aren't backwards compatible. Is IndexedDB ready for use by internal Gecko consumers, including a

Re: Storage in Gecko

On Fri, Apr 26, 2013 at 11:43 AM, Gregory Szorc wrote: > On 4/26/2013 11:26 AM, Kyle Huey wrote: > > On Fri, Apr 26, 2013 at 11:17 AM, Gregory Szorc wrote: > >> I'd like to start a discussion about the state of storage in Gecko. >> >> Currently when you

Re: Storage in Gecko

On Fri, Apr 26, 2013 at 11:36 AM, Andreas Gal wrote: > Preferences are as the name implies intended for preferences. There is no > sane use case for storing data in preferences. I would give any patch I come > across doing that an automatic sr- for poor taste and general insanity. As Greg sugge

Re: Storage in Gecko

On 4/26/2013 11:26 AM, Kyle Huey wrote: > On Fri, Apr 26, 2013 at 11:17 AM, Gregory Szorc <mailto:g...@mozilla.com>> wrote: > > I'd like to start a discussion about the state of storage in Gecko. > > Currently when you are writing a feature that needs to store

Re: Storage in Gecko

a discussion about the state of storage in Gecko. > > Currently when you are writing a feature that needs to store data, you > have roughly 3 choices: > > 1) Preferences > 2) SQLite > 3) Manual file I/O > > Preferences are arguably the easiest. However, they h

Re: Storage in Gecko

On Fri, Apr 26, 2013 at 11:17 AM, Gregory Szorc wrote: > I'd like to start a discussion about the state of storage in Gecko. > > Currently when you are writing a feature that needs to store data, you > have roughly 3 choices: > > 1) Preferences > 2) SQLite > 3) Ma

Storage in Gecko

I'd like to start a discussion about the state of storage in Gecko. Currently when you are writing a feature that needs to store data, you have roughly 3 choices: 1) Preferences 2) SQLite 3) Manual file I/O Preferences are arguably the easiest. However, they have a number of setbacks: a)