Queue peek?
Hi all I'm looking for a queue that I can use with multiprocessing, which has a peek method. I've seen some discussion about queue.peek but don't see anything in the docs about it. Does python have a queue class with peek semantics? Michael -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue peek?
On Mar 2, 1:18 pm, Raymond Hettinger wrote: > On Mar 2, 8:29 am, Veloz wrote: > > > Hi all > > I'm looking for a queue that I can use with multiprocessing, which has > > a peek method. > > > I've seen some discussion about queue.peek but don't see anything in > > the docs about it. > > > Does python have a queue class with peek semantics? > > Am curious about your use case? Why peek at something > that could be gone by the time you want to use it. > > val = q.peek() > if something_i_want(val): > v2 = q.get() # this could be different than val > > Wouldn't it be better to just get() the value and return if you don't > need it? > > val = q.peek() > if not something_i_want(val): > q.put(val) > > Raymond Yeah, I hear you. Perhaps queue is not the best solution. My highest level use case is this: The user visits a web page (my app is a Pylons app) and requests a "report" be created. The report takes too long to create and display on the spot, so the user expects to visit some url "later" and see if the specific report has completed, and if so, have it returned to them. At a lower level, I'm thinking of using some process workers to create these reports in the background; there'd be a request queue (into which requests for reports would go, each with an ID) and a completion queue, into which the workers would write an entry when a report was created, along with an ID matching the original request. The "peek" parts comes in when the user comes back later to see if their report has done. That is, in my page controller logic, I'd like to look through the complete queue and see if the specific report has been finished (I could tell by matching up the ID of the original request to the ID in the completed queue). If there was an item in the queue matching the ID, it would be removed. It's since occurred to me that perhaps a queue is not the best way to handle the completions. (We're ignoring the file system as a solution for the time being, and focusing on in-memory structures). I'm wondering now if a simple array of completed items wouldn't be better. Of course, all the access to the array would have to be thread/process- proof. As you pointed out, for example, multi-part operations such as "is such-and-such an ID in the list? If so, remove it and return in" would have to be treated atomically to avoid concurrency issues. Any thoughts on this design approach are welcomed :-) Michael -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue peek?
On Mar 3, 1:14 am, Gregory Ewing wrote: > MRAB wrote: > > I suppose it depends on the complexity of the data structure. A dict's > > methods are threadsafe, for example, but if you have a data structure > > where access leads to multiple method calls then collectively they need > > a lock. > > It also depends on the nature of the objects being used > as dict keys. Dict operations invoke the hashing and > comparison methods of the keys, which in general can > execute arbitrary Python code. > > If the keys are elementary built-in types such as > strings or ints, then dict operations will *probably* > be atomic. But I think I'd use a lock anyway, to be > on the safe side. > > -- > Greg Unless I missed where you guys were going, I think we got off the main point. The main question at hand was this: what's the best way (heck, any way) to implement a sort of "peek" whereby a number of processes can write results to some common "object" and some other process can "peek" into this object, looking for specific items they're interested in? I've tried doing this with a queue, as follows: children all write results to queue, each result has an identifier. Another interested party, which wants to know if identifier XXX has been placed in the queue, removes all the items, one by one from the queue, "keeps" the one matching the identifier (if found) and puts the rest of the items back on the queue, so other interested parties can also look through it. This is not a good solution, but it illustrates what I'm trying to achieve.. I'm looking at multiprocessing.Manager, ctypes, etc, but nothing's really jumped out. I also tried creating my own list class which uses locks to provide a "peek and remove" method, but I don't have a good way to share an instance of this object across processes. Any thoughts would be welcomed! Michael -- http://mail.python.org/mailman/listinfo/python-list
When to lock data items returned by multiprocessing.Manager?
So I'm using a multiprocessing.Manager instance in my main app and asking it to create a dictionary, which I am providing to instances of the application that I'm forking off with Process. The Processes and main app will be reading/writing to the dictionary. It's not clear to me what I have to "lock" and what I don't have to lock. Any ideas? I'm thinking that what Manager brings to the table is the ability to share the dictionary but it may not provide anything to control concurrent access to it, and thus I should lock all operations I perform on it.. But I don't know this for a fact. Any input is welcomed on this!! Michael -- http://mail.python.org/mailman/listinfo/python-list
