Thanks guys. (I'm sure Perrin is tired of answering these same old question in all of it's forms.)
The lists are functionally similar to Unix security lists (group id=name, etc). With thousands of users, these key lists are getting larger and the time to re-build them will continue to grow but they are invaluable in cutting down the time to do all sorts of good stuff. For example, to keep mysql indexing and joins simple (and relational updates simple), we simply pull de-normalized datasets from mysql and the lists are used to provide 'real' values on screen where ID's are present. So, the ideal situation would be to load once and just edit the lists in memory directly (after updating mysql) and never rebuild them except on restart. Loading and reloading the lists seems exhausting and counter-productive and running a query for the denormalized values at each session seems like another big waste. Can apache processes meaningfully access any external (i.e. shell, other) structures? On 5/7/07, Perrin Harkins <[EMAIL PROTECTED]> wrote:
On 5/7/07, Jonathan Vanasco <[EMAIL PROTECTED]> wrote: > Ah, I reread the post. I saw "large lists" and thought "complex data > structure", not simple text. I think we were talking about different things, actually. For reading and writing a large and complex data structure in its entirety, a Storable file is as good as it gets, unless you can rig something with an mmap'ed file. I was talking about reading/writing pieces of it in BDB or MySQL, which should not need Storable. > I didn't know that BDB does shared memory caching. And no socket overhead too. All the calls are in-process. > > Primary key lookups in MySQL over local sockets are very fast -- > > faster than memcached. > > Really ? I had read that they were about the same, but that mysql > selects are blocking & FIFO , while memcached is threaded and > supports concurrent access. Memcached is single-threaded and uses non-blocking I/O. It's a very different approach from a multi-threaded daemon like MySQL, and should scale better ultimately. However, for simple lookups, the network overhead from the TCP socket that memcached requires seems to outweigh any advantages. MySQL can use a pipe instead of a TCP socket, which it does automatically when you connect to a server on localhost. - Perrin
