Hey Viktor, I've been doing a bunch of research on this and it turns out my original post was incorrect. Unlike the version of Memcached that is out in the wild, the version of Memcached being used for App Engine *does* provide namespaced partitioning and nested namespace partitioning, with LRU expiration implemented both globally and per namespace (not 100% sure about namespaces deeper than root + 1). What this allows is constant time namespace flushing, among other security features. This is the mechanism that allows App Engine developers to all use Memcache without trampling over each others' data and is more sophisticated than prefixing a key - allowing us to track usage per user without have to iterate over a key range.
I'm going to file a bug internally against this mirroring your bug report. It's possible we just haven't exposed the interface for nested LRU grabTail correctly. I'll update the external bug if I hear anything. On Thu, Feb 4, 2010 at 1:17 AM, phraktle <[email protected]> wrote: > Hi Ikai, > > Any updates? (I see there's a mention of adding grabTail in the 1.3.1 > release notes - though it has been there in 1.3.0 already) > > In the meantime, I have implemented the queue with a head/tail > pointer, using the increment call... This works, but if grabTail > worked as advertised, that would be more elegant. > > Regards, > Viktor > > > On Feb 2, 7:28 pm, "Ikai L (Google)" <[email protected]> wrote: > > That's interesting. Let me follow up with the team to try to understand > how > > this is supposed to work. > > > > > > > > > > > > On Tue, Feb 2, 2010 at 12:05 AM, phraktle <[email protected]> wrote: > > > > > Plus, it's working okay in the development server :) > > > > > V. > > > > > On Feb 2, 9:03 am, phraktle <[email protected]> wrote: > > > > Hi, > > > > > > In this case grabTail would be quite useless, as one cannot really > > > > construct queues with it (ie. it would just force removing items from > > > > all namespaces that wouldn't even expire otherwise). The > documentation > > > > also implies that there's a separate LRU list per namespace, which > > > > does make sense, but is not what's happening in production: > > > > > >http://code.google.com/appengine/docs/java/javadoc/com/google/appengi > .. > > > .) > > > > > > "Grabs (atomically get and delete) items off the tail of LRU list. > > > > This can be used to implement queue system with high throughput and > > > > low latency, but low reliability. Current namespace should be set and > > > > not empty for the service. For each namespace memcache maintains a > > > > separate LRU list." > > > > > > Regards, > > > > Viktor > > > > > > On Feb 1, 9:22 pm, "Ikai L (Google)" <[email protected]> wrote: > > > > > > > I'll raise the issue with some other members of the team, but it > seems > > > like > > > > > this is working as expected. A memcache namespace is nothing more > than > > > a > > > > > prefix applied to a memcache key. There's no true partitioning > > > mechanism > > > > > within memcache. grabTail simply returns the item that would be > expired > > > by > > > > > the LRU mechanism if memcache needed more space. There isn't a > > > different > > > > > "queue" per namespace, only a "queue" for global expirations. > > > > > > > On Mon, Feb 1, 2010 at 4:29 AM, phraktle <[email protected]> > wrote: > > > > > > Hi, > > > > > > > > On production, grabTail returns objects from other namespaces. > This > > > is > > > > > > a significant problem that > > > > > > makes grabTail (thus queue-like usage) unusable. > > > > > > > > I filed this as a bug, with a very simple example here: > > > > > >http://code.google.com/p/googleappengine/issues/detail?id=2706 > > > > > > > > Can you please look into this? I don't even see a workaround that > I > > > > > > can implement in the meantime... > > > > > > > > Thanks, > > > > > > Viktor > > > > > > > > -- > > > > > > You received this message because you are subscribed to the > Google > > > Groups > > > > > > "Google App Engine for Java" group. > > > > > > To post to this group, send email to > > > > > > [email protected]. > > > > > > To unsubscribe from this group, send email to > > > > > > [email protected]<google-appengine-java%[email protected]><google-appengine-java%2B > [email protected]><google-appengine-java%2B > > > [email protected]> > > > > > > . > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > > > -- > > > > > Ikai Lan > > > > > Developer Programs Engineer, Google App Enginehttp:// > > > googleappengine.blogspot.com|http://twitter.com/app_engine > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to > > > [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine-java%[email protected]><google-appengine-java%2B > [email protected]> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Enginehttp:// > googleappengine.blogspot.com|http://twitter.com/app_engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine-java%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
