jams...@googlemail.com wrote:
[...]
> The program is multithreaded to speed up the processing...there are
> input and output Queues.
It's not the major point here, but are you aware of Python's GIL?
> Now, each domain entry is an class object containing various bits of
> info. Each domain class also has its own lock, so that only one thread
> can modify each domain at a time.
>
> I'm load-testing the program with a sample of 1 million email
> addresses and when i hit about the 500,000 mark i get a locking
> error...
Does that correspond to creating 500,000 locks? For sure?
> sem_init: No space left on device
[...]
> error: can't allocate lock
What happens if you try to allocate a million locks, as in:
from threading import Lock
locks = []
for i in range(10**6):
try:
locks.append(Lock())
locks[-1].acquire()
except:
print "Failed after %d locks." % i
raise
I tried the above in WindowsXP and Ubuntu 8.04 (Python 2.6 and 2.52
respectively), and it ran on both without the exception.
> Googling for this sort of error doesn't yield any results, and i can't
> find any information about limits to the number of locks you can have
> in Python. The 'No space left on device' message indicates a memory
> issue, however i doubt this since its running on a linux server with 4
> cores and 16GB ram. It seems more like an internal Python limit has
> been hit (sem_init - semaphore initialisation?). Does anyone know more
> about threading internals and any internal limits?
Python treading is implemented on top of some thread facility the target
O.S. provides. Creating a Python lock may or may not use up some limited
kernel resource. Given my experiments, limited as they were I do not
think this is a Python limit. If could be an issue with your platform,
or a defect in the code you inherited.
> For the timebeing, I've implemented locking at a higher level which
> will reduce performance but keep the number of lock objects to a
> minumum.
That's probably a reasonable strategy whether or not you can create a
million locks.
--
--Bryan
--
http://mail.python.org/mailman/listinfo/python-list
