Re: a huge shared read-only data in parallel accesses -- How? multithreading? multiprocessing?

2009-12-10 Thread Klauss
On Dec 9, 11:58 am, Valery wrote: > Hi all, > > Q: how to organize parallel accesses to a huge common read-only Python > data structure? > > Details: > > I have a huge data structure that takes >50% of RAM. > My goal is to have many computational threads (or processes) that can > have an efficient

Re: a huge shared read-only data in parallel accesses -- How? multithreading? multiprocessing?

2009-12-14 Thread Klauss
On Dec 11, 11:00 am, Antoine Pitrou wrote: > I was going to suggest memcached but it probably serializes non-atomic > types. Atomic as well. memcached communicates through sockets[3] (albeit possibly unix sockets, which are faster than TCP ones). multiprocessing has shared memory schemes, but doe

Re: a huge shared read-only data in parallel accesses -- How? multithreading? multiprocessing?

2010-01-07 Thread Klauss
On Dec 31 2009, 6:36 pm, garyrob wrote: > One thing I'm not clear on regarding Klauss' patch. He says it's > applicable where the data is primarily non-numeric. In trying to > understand why that would be the case, I'm thinking that the increased > per-obje