Re: MemoryError and Pickle

2016-11-21 Thread Steve D'Aprano
ked beautifully until today when the large data (1GB) > file caused a MemoryError :( At what point do you run out of memory? When building the list? If so, then you need more memory, or smaller lists, or avoid creating a giant list in the first place. If you can successfully build the list, but the

Re: MemoryError and Pickle

2016-11-21 Thread Steve D'Aprano
On Tue, 22 Nov 2016 11:40 am, Peter Otten wrote: > Fillmore wrote: > >> Hi there, Python newbie here. >> >> I am working with large files. For this reason I figured that I would >> capture the large input into a list and serialize it with pickle for >> later (faster) usage. > > But is it really

Re: MemoryError and Pickle

2016-11-21 Thread Chris Kaynor
On Mon, Nov 21, 2016 at 3:43 PM, John Gordon wrote: > In Fillmore > writes: > > >> Question for experts: is there a way to refactor this so that data may >> be filled/written/released as the scripts go and avoid the problem? >> code below. > > That depends on how the data will be read. Here is

Re: MemoryError and Pickle

2016-11-21 Thread Peter Otten
Fillmore wrote: > Hi there, Python newbie here. > > I am working with large files. For this reason I figured that I would > capture the large input into a list and serialize it with pickle for > later (faster) usage. But is it really faster? If the pickle is, let's say, twice as large as the or

Re: MemoryError and Pickle

2016-11-21 Thread John Gordon
In Fillmore writes: > Question for experts: is there a way to refactor this so that data may > be filled/written/released as the scripts go and avoid the problem? > code below. That depends on how the data will be read. Here is one way to do it: fileObject = open(filename, "w") for

MemoryError and Pickle

2016-11-21 Thread Fillmore
Hi there, Python newbie here. I am working with large files. For this reason I figured that I would capture the large input into a list and serialize it with pickle for later (faster) usage. Everything has worked beautifully until today when the large data (1GB) file caused a MemoryError

Re: MemoryError in data conversion

2014-04-15 Thread Gregory Ewing
Mok-Kong Shen wrote: (It means that I have to pickle out the list to a file and read in the content of the file in order to have it as a bytearray etc. etc.) No, you don't -- pickle.dumps() returns the pickled data as a bytes object instead of writing it to a file. -- Greg -- https://mail.pyth

Re: MemoryError in data conversion

2014-04-15 Thread Peter Otten
Gregory Ewing wrote: > Mok-Kong Shen wrote: >> I have yet a question out of curiosity: Why is my 2nd list structure, >> that apparently is too complex for handling by eval and json, seemingly >> not a problem for pickle? > > Pickle is intended for arbitrary data structures, so it > is designed to

Re: MemoryError in data conversion

2014-04-15 Thread Mok-Kong Shen
Am 15.04.2014 01:51, schrieb Gregory Ewing: Mok-Kong Shen wrote: I have yet a question out of curiosity: Why is my 2nd list structure, that apparently is too complex for handling by eval and json, seemingly not a problem for pickle? Pickle is intended for arbitrary data structures, so it is de

Re: MemoryError in data conversion

2014-04-14 Thread Gregory Ewing
Mok-Kong Shen wrote: I have yet a question out of curiosity: Why is my 2nd list structure, that apparently is too complex for handling by eval and json, seemingly not a problem for pickle? Pickle is intended for arbitrary data structures, so it is designed to be able to handle deeply-nested and

Re: MemoryError in data conversion

2014-04-14 Thread Mok-Kong Shen
Am 14.04.2014 15:59, schrieb Peter Otten: You could use json, but you may run into the same problem with that, too (only later): import json items = [] for i in range(1000): ... s = json.dumps(items) ... items = [items] ... Traceback (most recent call last): File "", line 2, in

Re: MemoryError in data conversion

2014-04-14 Thread Peter Otten
Mok-Kong Shen wrote: > Am 14.04.2014 09:46, schrieb Peter Otten: > >> You ran into a limitation of the compiler. For us to suggest a workaround >> you'd have to explain why you want to convert the list returned from >> buildhuffmantree() into python source code and back. > > That list gives the

Re: MemoryError in data conversion

2014-04-14 Thread Mok-Kong Shen
Am 14.04.2014 09:46, schrieb Peter Otten: You ran into a limitation of the compiler. For us to suggest a workaround you'd have to explain why you want to convert the list returned from buildhuffmantree() into python source code and back. That list gives the Huffman encoding tree for compressin

Re: MemoryError in data conversion

2014-04-14 Thread Peter Otten
Mok-Kong Shen wrote: > The code attached below produces in one of the two IMHO similar cases > (excepting the sizes of the lists involved) MemoryError. Could experts > kindly tell why that's so and whether there is any work-around feasible. Here's a simpler way to

Re: MemoryError in data conversion

2014-04-13 Thread dieter
Mok-Kong Shen writes: > The code attached below produces in one of the two IMHO similar cases > (excepting the sizes of the lists involved) MemoryError. Could experts > kindly tell why that's so and whether there is any work-around feasible. "MemoryError" means: the P

MemoryError in data conversion

2014-04-13 Thread Mok-Kong Shen
The code attached below produces in one of the two IMHO similar cases (excepting the sizes of the lists involved) MemoryError. Could experts kindly tell why that's so and whether there is any work-around feasible. Thanks in advances. M. K.

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-04-02 Thread Dan Sommers
On Mon, 01 Apr 2013 21:45:30 -0700, Tim Roberts wrote: > morphex wrote: >> >>While we're on the subject, wouldn't it be nice to have some cap there >>so that it isn't possible to more or less block the system with large >>exponentiation? > &g

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-04-02 Thread Nobody
On Mon, 01 Apr 2013 00:39:56 +, Alex wrote: > Given that > > 3 > 5 > 4 > > (i.e.: 4**5**3) is transitive, I think you meant "associative", and exponentiation isn't associative, i.e. (x**y)**z is not, in general, equal to x**(y**z). In fact, (x**y)**z is equal to x**(y*z). Conventional m

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-04-01 Thread Chris Angelico
On Tue, Apr 2, 2013 at 3:45 PM, Tim Roberts wrote: > morphex wrote: >> >>While we're on the subject, wouldn't it be nice to have some cap there so >>that it isn't possible to more or less block the system with large >>exponentiation? > > There IS

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-04-01 Thread Tim Roberts
morphex wrote: > >While we're on the subject, wouldn't it be nice to have some cap there so >that it isn't possible to more or less block the system with large >exponentiation? There IS a cap. It's called the "MemoryError" exception. But, seriously,

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-04-01 Thread Roy Smith
In article <51590a2b$0$3$c3e8da3$54964...@news.astraweb.com>, Steven D'Aprano wrote: > Concrete examples of transitive relations: greater than, equal to, less > than and equal to. Will Python 4 implement "less than and equal to"? :-) [Warning: topic creep] Well, they are transitive over

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Steven D'Aprano
On Mon, 01 Apr 2013 00:39:56 +, Alex wrote: > Chris Angelico wrote: > > >> Opening paragraph, "... exponentiation, which groups from right to >> left". It follows the obvious expectation from mathematics. (The OP is >> using Python 2, but the same applies.) > > Thanks. I did miss that paren

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Chris Angelico
On Mon, Apr 1, 2013 at 11:39 AM, Alex wrote: > Given that > > 3 > 5 > 4 > > (i.e.: 4**5**3) is transitive, I would have expected Python to exhibit > more consistency with the other operators. I guess that is one of the > foolish consistencies that comprise the hobgoblins of my little mind, > th

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Alex
Chris Angelico wrote: > > Opening paragraph, "... exponentiation, which groups from right to > left". It follows the obvious expectation from mathematics. (The OP is > using Python 2, but the same applies.) Thanks. I did miss that parenthetical comment in para 6.15, and that would have been the

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Dave Angel
On 03/31/2013 06:06 PM, Alex wrote: Dave Angel wrote: On 03/31/2013 02:56 AM, morphex wrote: 1**2 1 1**2**3 1 1**2**3**4 1L 1**2**3**4**5 Traceback (most recent call last): File "", line 1, in MemoryError Does anyone know why this raises a MemoryError? Doesn't m

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Chris Angelico
On Mon, Apr 1, 2013 at 9:28 AM, Chris Angelico wrote: > On Mon, Apr 1, 2013 at 9:06 AM, Alex wrote: >> Really? >> >> The Python 3 documentation >> (http://docs.python.org/3/reference/expressions.html) says in section >> 6.14 (Evaluation order) that "Python evaluates expressions from left to >> ri

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Chris Angelico
On Mon, Apr 1, 2013 at 9:06 AM, Alex wrote: > Dave Angel wrote: > >> On 03/31/2013 02:56 AM, morphex wrote: >> > > > > 1**2 >> > 1 >> > > > > 1**2**3 >> > 1 >> > > > > 1**2**3**4 >> > 1L >> > &g

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Alex
Dave Angel wrote: > On 03/31/2013 02:56 AM, morphex wrote: > > > > > 1**2 > > 1 > > > > > 1**2**3 > > 1 > > > > > 1**2**3**4 > > 1L > > > > > 1**2**3**4**5 > > Traceback (most recent call last): > >

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Jason Swails
On Sun, Mar 31, 2013 at 9:15 AM, Roy Smith wrote: > > > $ prtstat 29937 > > Process: mongodState: S (sleeping) > > [...] > > Memory > > Vsize: 1998285 MB > > RSS: 5428 MB > > RSS Limit: 18446744073709 MB > > If I counted the digits right, that 1.9 TB. I love the R

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Roy Smith
In article , Dave Angel wrote: > I'm typing this while a terminal is open doing the particular operation, > and the system doesn't seem in the least sluggish. > > Currently the memory used is at 10gig, and while there are some pauses > in my typing, the system has not died. This is on Linux

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Roy Smith
In article <8276eff6-9e5c-4060-b9e8-94fab6062...@googlegroups.com>, morphex wrote: > Aha, OK. Thought I found a bug but yeah that makes sense ;) > > While we're on the subject, wouldn't it be nice to have some cap there so > that it isn't possible to more or less block the system with large

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Roy Smith
In article <5157e6cc$0$29974$c3e8da3$54964...@news.astraweb.com>, Steven D'Aprano wrote: > For what it's worth, that last intermediate result (two to the power of > the 489-digit number) has approximately a billion trillion trillion > trillion trillion trillion trillion trillion trillion trill

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Dave Angel
On 03/31/2013 08:07 AM, morphex wrote: Aha, OK. Thought I found a bug but yeah that makes sense ;) While we're on the subject, wouldn't it be nice to have some cap there so that it isn't possible to more or less block the system with large exponentiation? There's an assumption there. The O

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread morphex
$ python > > > Python 2.7.3 (default, Sep 26 2012, 21:53:58) [GCC 4.7.2] on linux2 > > > Type "help", "copyright", "credits" or "license" for more information. > > >>>> 1**2 > > > 1 >

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Dave Angel
2] on linux2 Type "help", "copyright", "credits" or "license" for more information. 1**2 1 1**2**3 1 1**2**3**4 1L 1**2**3**4**5 Traceback (most recent call last): File "", line 1, in MemoryError Does anyone know why this rai

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Steven D'Aprano
Type "help", "copyright", "credits" or "license" for more information. >>>> 1**2 > 1 >>>> 1**2**3 > 1 >>>> 1**2**3**4 > 1L >>>> 1**2**3**4**5 > Traceback (most recent call last): > File &q

Re: Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread Dave Angel
credits" or "license" for more information. 1**2 1 1**2**3 1 1**2**3**4 1L 1**2**3**4**5 Traceback (most recent call last): File "", line 1, in MemoryError Does anyone know why this raises a MemoryError? Doesn't make sense to me. Perhaps you didn&#x

Why does 1**2**3**4**5 raise a MemoryError?

2013-03-31 Thread morphex
re information. >>> 1**2 1 >>> 1**2**3 1 >>> 1**2**3**4 1L >>> 1**2**3**4**5 Traceback (most recent call last): File "", line 1, in MemoryError >>> Does anyone know why this raises a MemoryError? Doesn't make sense to me. Happy Easter! -Morten -- http://mail.python.org/mailman/listinfo/python-list

Re: MemoryError vs malloc error

2011-07-16 Thread Dan Stromberg
python process (running on freebsd). Sometimes when it > uses too much memory it core dumps. I would've expected it to raise > MemoryError. Normally, when would a python process raise MemoryError and > when would it fail with malloc error and cores? This is happening in pure > pyt

MemoryError vs malloc error

2011-07-15 Thread Amit Dev
Hi, I've a long running python process (running on freebsd). Sometimes when it uses too much memory it core dumps. I would've expected it to raise MemoryError. Normally, when would a python process raise MemoryError and when would it fail with malloc error and cores? This is happeni

Re: Pickle MemoryError - any ideas?

2010-07-20 Thread John Nagle
On 7/20/2010 3:01 PM, Peter wrote: I have created a class that contains a list of files (contents, binary) - so it uses a LOT of memory. When I first pickle.dump the list it creates a 1.9GByte file on the disk. I can load the contents back again, but when I attempt to dump it again (with or with

Re: Pickle MemoryError - any ideas?

2010-07-20 Thread Carl Banks
>   File "c:\Python26\Lib\pickle.py", line 615, in _batch_appends >     save(x) >   File "c:\Python26\Lib\pickle.py", line 286, in save >     f(self, obj) # Call unbound method with explicit self >   File "c:\Python26\Lib\pickle.py", line 488, in save

Re: Pickle MemoryError - any ideas?

2010-07-20 Thread Emile van Sebille
On 7/20/2010 3:01 PM Peter said... I have created a class that contains a list of files (contents, binary) - so it uses a LOT of memory. Any ideas? Switch to 64 bit Windows & Python? Emile -- http://mail.python.org/mailman/listinfo/python-list

Pickle MemoryError - any ideas?

2010-07-20 Thread Peter
kle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "c:\Python26\Lib\pickle.py", line 488, in save_string self.write(STRING + repr(obj) + '\n') MemoryError I get this error either attempting to dump the entire list or dumping

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-10 Thread Emin.shopper Martinian.shopper
em and > gotten a bunch of stuff right. There is no good reason why > constructing a 50 kilobyte dict should fail with a MemoryError while > constructing 50 megabyte lists succeeds. > > > -- > --Bryan Olson > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-04 Thread Dan Stromberg
On Thu, Jun 3, 2010 at 3:43 PM, Emin.shopper Martinian.shopper < emin.shop...@gmail.com> wrote: > Dear Experts, > > I am getting a MemoryError when creating a dict in a long running > process and suspect this is due to memory fragmentation. Any > suggestions would be welcome

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-04 Thread Bryan
e on that level, but Emin seems to have worked his problem and gotten a bunch of stuff right. There is no good reason why constructing a 50 kilobyte dict should fail with a MemoryError while constructing 50 megabyte lists succeeds. -- --Bryan Olson -- http://mail.python.org/mailman/listinfo/python-list

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-04 Thread Philip Semanchuk
On Jun 4, 2010, at 12:06 PM, Bryan wrote: Emin.shopper wrote: dmtr wrote: I'm still unconvinced that it is a memory fragmentation problem. It's very rare. You could be right. I'm not an expert on python memory management. But if it isn't memory fragmentation, then why is it that I can

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-04 Thread Bryan
Emin.shopper wrote: > dmtr wrote: > > I'm still unconvinced that it is a memory fragmentation problem. It's > > very rare. > > You could be right. I'm not an expert on python memory management. But > if it isn't memory fragmentation, then why is it that I can create > lists which use up 600 more MB

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-04 Thread Emin.shopper Martinian.shopper
On Thu, Jun 3, 2010 at 10:00 PM, dmtr wrote: > I'm still unconvinced that it is a memory fragmentation problem. It's > very rare. You could be right. I'm not an expert on python memory management. But if it isn't memory fragmentation, then why is it that I can create lists which use up 600 more M

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-03 Thread dmtr
I'm still unconvinced that it is a memory fragmentation problem. It's very rare. Can you give more concrete example that one can actually try to execute? Like: python -c "list([list([0]*xxx)+list([1]*xxx)+list([2]*xxx) +list([3]*xxx) for xxx in range(10)])" & -- Dmitry -- http://mail.python.

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-03 Thread Emin.shopper Martinian.shopper
On Thu, Jun 3, 2010 at 7:41 PM, dmtr wrote: > On Jun 3, 3:43 pm, "Emin.shopper Martinian.shopper" > wrote: >> Dear Experts, >> > > Are you sure you have enough memory available? > Dict memory usage can jump x2 during re-balancing. > I'm pretty sure. When I did  p setattr(self,'q',dict([(xxx,xxx

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-03 Thread dmtr
> I have a long running processing which eventually dies to a > MemoryError exception. When it dies, it is using roughly 900 MB on a 4 > GB Windows XP machine running Python 2.5.4. If I do "import pdb; BTW have you tried the same code with the Python 2.6.5? -- Dmitry -- http://m

Re: getting MemoryError with dicts; suspect memory fragmentation

2010-06-03 Thread dmtr
On Jun 3, 3:43 pm, "Emin.shopper Martinian.shopper" wrote: > Dear Experts, > > I am getting a MemoryError when creating a dict in a long running > process and suspect this is due to memory fragmentation. Any > suggestions would be welcome. Full details of the problem are

getting MemoryError with dicts; suspect memory fragmentation

2010-06-03 Thread Emin.shopper Martinian.shopper
Dear Experts, I am getting a MemoryError when creating a dict in a long running process and suspect this is due to memory fragmentation. Any suggestions would be welcome. Full details of the problem are below. I have a long running processing which eventually dies to a MemoryError exception

Re: MemoryError, can I use more?

2010-02-14 Thread Tommy Grav
On Feb 14, 2010, at 10:16 PM, Dave Angel wrote: > There are three different limits at play here. Since you're still not saying > how you're "measuring" usage, we've all been guessing just which one you're > hitting. There's physical RAM, virtual address space, and swappable space > (swapfile

Re: MemoryError, can I use more?

2010-02-14 Thread Dave Angel
egory, M.Sc., E.I. Ph.D Candidate University of Miami Phone 305 284-3611 From: Chris Kaynor [ckay...@zindagigames.com] Sent: Friday, February 12, 2010 7:44 PM To: Echavarria Gregory, Maria Angelica Cc: python-list@python.org Subject: Re: MemoryError, can I use more?

Re: MemoryError, can I use more?

2010-02-14 Thread Tommy Grav
On Feb 14, 2010, at 7:20 PM, Echavarria Gregory, Maria Angelica wrote: > > Dear Chris, > > One of the machines I tested my app in is 64 bit and happened the same. The > RAM consumed by the OS and other processes is already included in the 2.2 I'm > telling... my app enters to work when the RA

FW: MemoryError, can I use more?

2010-02-14 Thread Echavarria Gregory, Maria Angelica
ty of Miami Phone 305 284-3611 From: Chris Kaynor [ckay...@zindagigames.com] Sent: Friday, February 12, 2010 7:44 PM To: Echavarria Gregory, Maria Angelica Cc: python-list@python.org Subject: Re: MemoryError, can I use more? A 32 bit app can only use 4 GB of memory i

FW: MemoryError, can I use more?

2010-02-14 Thread Echavarria Gregory, Maria Angelica
, M.Sc., E.I. Ph.D Candidate University of Miami Phone 305 284-3611 From: sstein...@gmail.com [sstein...@gmail.com] Sent: Friday, February 12, 2010 7:58 PM To: Echavarria Gregory, Maria Angelica Cc: python-list@python.org Subject: Re: MemoryError, can I use more

Re: MemoryError, can I use more?

2010-02-14 Thread Diez B. Roggisch
Am 14.02.10 12:28, schrieb Laszlo Nagy: 2010.02.13. 17:40 keltezéssel, Diez B. Roggisch írta: Am 13.02.10 17:18, schrieb Anssi Saari: Nobody writes: A single process can't use much more than 2GiB of RAM without a 64-bit CPU and OS. That's not really true. Even Windows XP has the /3GB boot o

Re: MemoryError, can I use more?

2010-02-14 Thread Ross Ridge
"Diez B. Roggisch" writes: > No, PAE can be used to access much more memory than 4GB - albeit > through paging. AFAIK up to 2^36 Bytes. Anssi Saari wrote: >That too. I admit, after checking, that you can't go above 3 GiB per >process even in server Windows. But for Linux there exists (or >exist

Re: Please help with MemoryError

2010-02-14 Thread Steve Howell
On Feb 11, 5:50 pm, Steven D'Aprano wrote: > On Thu, 11 Feb 2010 15:39:09 -0800, Jeremy wrote: > > My Python program now consumes over 2 GB of memory and then I get a > > MemoryError.  I know I am reading lots of files into memory, but not 2GB > > worth. > > 2.  

Re: Please help with MemoryError

2010-02-14 Thread Steve Howell
On Feb 14, 10:32 am, Steve Holden wrote: > rantingrick wrote: > > On Feb 12, 4:10 pm, Steve Holden wrote: > >> Antoine Pitrou wrote: > >>> Le Fri, 12 Feb 2010 17:14:57 +, Steven D'Aprano a écrit : > > > On Feb 12, 4:10 pm, Steve Holden wrote: > >> Antoine Pitrou wrote: > >>> Le Fri, 12 Feb 2

Re: Please help with MemoryError

2010-02-14 Thread Steve Holden
rantingrick wrote: > On Feb 12, 4:10 pm, Steve Holden wrote: >> Antoine Pitrou wrote: >>> Le Fri, 12 Feb 2010 17:14:57 +, Steven D'Aprano a écrit : > > On Feb 12, 4:10 pm, Steve Holden wrote: >> Antoine Pitrou wrote: >>> Le Fri, 12 Feb 2010 17:14:57 +, Steven D'Aprano a écrit : > > Stev

Re: MemoryError, can I use more?

2010-02-14 Thread Laszlo Nagy
2010.02.13. 17:40 keltezéssel, Diez B. Roggisch írta: Am 13.02.10 17:18, schrieb Anssi Saari: Nobody writes: A single process can't use much more than 2GiB of RAM without a 64-bit CPU and OS. That's not really true. Even Windows XP has the /3GB boot option to allow 3 GiB per process. On PC

Re: Please help with MemoryError

2010-02-13 Thread rantingrick
On Feb 12, 4:10 pm, Steve Holden wrote: > Antoine Pitrou wrote: > > Le Fri, 12 Feb 2010 17:14:57 +, Steven D'Aprano a écrit : On Feb 12, 4:10 pm, Steve Holden wrote: > Antoine Pitrou wrote: > > Le Fri, 12 Feb 2010 17:14:57 +, Steven D'Aprano a écrit : Steve, Why do so many of your posts

Re: MemoryError, can I use more?

2010-02-13 Thread Diez B. Roggisch
Am 13.02.10 17:18, schrieb Anssi Saari: Nobody writes: A single process can't use much more than 2GiB of RAM without a 64-bit CPU and OS. That's not really true. Even Windows XP has the /3GB boot option to allow 3 GiB per process. On PCs, free operating systems and server Windows can use PAE

Re: MemoryError, can I use more?

2010-02-13 Thread Anssi Saari
Nobody writes: > A single process can't use much more than 2GiB of RAM without a 64-bit CPU > and OS. That's not really true. Even Windows XP has the /3GB boot option to allow 3 GiB per process. On PCs, free operating systems and server Windows can use PAE to give access to full 4 GB per process

MemoryError in imaplib?

2010-02-13 Thread Márcio Faustino
Hi, When using "imaplib" to fetch e-mail messages, the "IMAP4_SSL.read" and "IMAP4_SSL.readline" functions sometimes throw a "MemoryError" exception in "chunks.append(data)" and "line.append(char)", respectively. But if I change thos

Re: MemoryError, can I use more?

2010-02-13 Thread Anssi Saari
"Diez B. Roggisch" writes: > Am 13.02.10 17:18, schrieb Anssi Saari: >> Nobody writes: >> >>> A single process can't use much more than 2GiB of RAM without a 64-bit CPU >>> and OS. >> >> That's not really true. Even Windows XP has the /3GB boot option to >> allow 3 GiB per process. On PCs, free

Re: MemoryError, can I use more?

2010-02-12 Thread Nobody
AM of the machine, > python always gives the MemoryError message when it has occupied exactly > only 2.2 GB. I have tested this in 4 different machines, all with memory > of 3 to 4 GB... I'm amazed. > > Could any of you please help me to figure out how to change that limit? I > t

Re: MemoryError, can I use more?

2010-02-12 Thread Christian Heimes
machine, python > always gives the MemoryError message when it has occupied exactly only 2.2 > GB. I have tested this in 4 different machines, all with memory of 3 to 4 > GB... I'm amazed. The amount of RAM in your boxes is unrelated to your issue. A 32bit process is able to addre

Re: MemoryError, can I use more?

2010-02-12 Thread Benjamin Kaplan
of the physical RAM of the machine, python > always gives the MemoryError message when it has occupied exactly only 2.2 > GB. I have tested this in 4 different machines, all with memory of 3 to 4 > GB... I'm amazed. > > Could any of you please help me to figure out how to

Re: MemoryError, can I use more?

2010-02-12 Thread Mark Lawrence
the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed. Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help

Re: MemoryError, can I use more?

2010-02-12 Thread sstein...@gmail.com
of the physical RAM of the machine, python > always gives the MemoryError message when it has occupied exactly only 2.2 > GB. I have tested this in 4 different machines, all with memory of 3 to 4 > GB... I'm amazed. > > Could any of you please help me to figure out how to

Re: Please help with MemoryError

2010-02-12 Thread Gregory Ewing
Antoine Pitrou wrote: It's just that assignment ("=") means a different thing in Python than in non-object languages (or fake-object languages such as C++ or PHP): it rebinds instead of mutating in-place. If it mutated, you wouldn't have the AssertionError. It doesn't really have anything t

Re: MemoryError, can I use more?

2010-02-12 Thread Chris Kaynor
; but have found that, independent of the physical RAM of the machine, python > always gives the MemoryError message when it has occupied exactly only 2.2 > GB. I have tested this in 4 different machines, all with memory of 3 to 4 > GB... I'm amazed. > > Could any of you please help

MemoryError, can I use more?

2010-02-12 Thread Echavarria Gregory, Maria Angelica
Dear group: I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has

Re: Please help with MemoryError

2010-02-12 Thread Robert Kern
On 2010-02-12 17:30 PM, Antoine Pitrou wrote: Le Fri, 12 Feb 2010 23:49:38 +0100, Alf P. Steinbach a écrit : The main reason for not using that term for Python is that "pass by reference" has the extremely strong connotation of being able to implement 'swap'. But 'swap' is so easy to write as

Re: Please help with MemoryError

2010-02-12 Thread Gregory Ewing
Steven D'Aprano wrote: Python's calling convention already has an well-established name, established over thirty years ago by the distinguished computer scientist Barbara Liskov, namely call-by-sharing. And she was mistaken in thinking it needed a new name. -- Greg -- http://mail.python.org/

Re: Please help with MemoryError

2010-02-12 Thread Antoine Pitrou
Le Fri, 12 Feb 2010 23:49:38 +0100, Alf P. Steinbach a écrit : > > The main reason for not using that term for Python is that "pass by > reference" has the extremely strong connotation of being able to > implement 'swap'. But 'swap' is so easy to write as a one-line statement that it's foolish t

Re: Please help with MemoryError

2010-02-12 Thread Steve Holden
Gib Bogle wrote: > Steven D'Aprano wrote: > >> def swap(a, b): >> a, b = b, a >> >> x = 1 >> y = 2 >> swap(x, y) >> assert (x == 2) and (y==1) > > Can't the same point be more simply made with this example: > > def setval(a): > a = 12345 > > x = 1 > setval(x) > print x > Yes, and it wi

Re: Please help with MemoryError

2010-02-12 Thread Gib Bogle
Steven D'Aprano wrote: def swap(a, b): a, b = b, a x = 1 y = 2 swap(x, y) assert (x == 2) and (y==1) Can't the same point be more simply made with this example: def setval(a): a = 12345 x = 1 setval(x) print x ? -- http://mail.python.org/mailman/listinfo/python-list

Re: Please help with MemoryError

2010-02-12 Thread Alf P. Steinbach
* Antoine Pitrou: Le Fri, 12 Feb 2010 23:12:06 +0100, Alf P. Steinbach a écrit : Steven talks about the standard meaning of "pass by reference". See my answer to Steve's message. You can't postulate a "standard meaning" of "pass by reference" independently of the specificities of each langua

Re: Please help with MemoryError

2010-02-12 Thread Antoine Pitrou
Le Fri, 12 Feb 2010 23:12:06 +0100, Alf P. Steinbach a écrit : > > Steven talks about the standard meaning of "pass by reference". See my answer to Steve's message. You can't postulate a "standard meaning" of "pass by reference" independently of the specificities of each language. For example a

Re: Please help with MemoryError

2010-02-12 Thread Antoine Pitrou
Le Fri, 12 Feb 2010 17:10:01 -0500, Steve Holden a écrit : > > As has already been pointed out, if Python used call by reference then > the following code would run without raising an AssertionError: > > def exchange(a, b): > a, b = b, a > > x = 1 > y = 2 > exchange(x, y) > assert (x == 2 an

Re: Please help with MemoryError

2010-02-12 Thread Alf P. Steinbach
* Antoine Pitrou: Le Fri, 12 Feb 2010 17:14:57 +, Steven D'Aprano a écrit : What Python does is called "pass by sharing", or sometimes "pass by object reference". It is exactly the same as what (e.g.) Ruby and Java do, except that confusingly the Ruby people call it "pass by reference" and t

Re: Please help with MemoryError

2010-02-12 Thread Steve Holden
Antoine Pitrou wrote: > Le Fri, 12 Feb 2010 17:14:57 +, Steven D'Aprano a écrit : >> What Python does is called "pass by sharing", or sometimes "pass by >> object reference". It is exactly the same as what (e.g.) Ruby and Java >> do, except that confusingly the Ruby people call it "pass by refe

Re: Please help with MemoryError

2010-02-12 Thread Antoine Pitrou
Le Fri, 12 Feb 2010 17:14:57 +, Steven D'Aprano a écrit : > > What Python does is called "pass by sharing", or sometimes "pass by > object reference". It is exactly the same as what (e.g.) Ruby and Java > do, except that confusingly the Ruby people call it "pass by reference" > and the Java pe

Re: Please help with MemoryError

2010-02-12 Thread Steven D'Aprano
On Fri, 12 Feb 2010 21:07:08 +0100, mk wrote: > John Posner wrote: > >>> http://effbot.org/zone/call-by-object.htm >>> http://en.wikipedia.org/wiki/Evaluation_strategy > >> [1] http://mail.python.org/pipermail/edu-sig/2008-May/008583.html > > Hmm how about "call by label-value"? Pyth

Re: Please help with MemoryError

2010-02-12 Thread Ethan Furman
mk wrote: John Posner wrote: http://effbot.org/zone/call-by-object.htm http://en.wikipedia.org/wiki/Evaluation_strategy [1] http://mail.python.org/pipermail/edu-sig/2008-May/008583.html Hmm how about "call by label-value"? That is, you change labels by assignment, but pass the v

Re: Please help with MemoryError

2010-02-12 Thread Alf P. Steinbach
* Christian Heimes: mk wrote: Hmm how about "call by label-value"? Or "call by guido"? How do you like "call like a dutch"? :] Just a note: it might be more clear to talk about "pass by XXX" than "call by XXX". Unless you're talking about something else than argument passing. The standard

Re: Please help with MemoryError

2010-02-12 Thread Christian Heimes
mk wrote: > Hmm how about "call by label-value"? Or "call by guido"? How do you like "call like a dutch"? :] -- http://mail.python.org/mailman/listinfo/python-list

Re: Please help with MemoryError

2010-02-12 Thread Mel
mk wrote: > John Posner wrote: > >>> http://effbot.org/zone/call-by-object.htm >>> http://en.wikipedia.org/wiki/Evaluation_strategy > >> [1] http://mail.python.org/pipermail/edu-sig/2008-May/008583.html > > Hmm how about "call by label-value"? Nothing egregiously wrong with it.. mayb

Re: Please help with MemoryError

2010-02-12 Thread mk
John Posner wrote: http://effbot.org/zone/call-by-object.htm http://en.wikipedia.org/wiki/Evaluation_strategy [1] http://mail.python.org/pipermail/edu-sig/2008-May/008583.html Hmm how about "call by label-value"? That is, you change labels by assignment, but pass the value of the

Re: Please help with MemoryError

2010-02-12 Thread John Posner
On 2/12/2010 12:14 PM, Steven D'Aprano wrote: On Fri, 12 Feb 2010 06:45:31 -0800, Jeremy wrote: You also confirmed what I thought was true that all variables are passed "by reference" so I don't need to worry about the data being copied (unless I do that explicitly). No, but yes. No, variabl

Re: Please help with MemoryError

2010-02-12 Thread Steven D'Aprano
On Fri, 12 Feb 2010 06:45:31 -0800, Jeremy wrote: > You also confirmed what I thought was true that all variables are passed > "by reference" so I don't need to worry about the data being copied > (unless I do that explicitly). No, but yes. No, variables are not passed by reference, but yes, you

Re: Please help with MemoryError

2010-02-12 Thread Tim Chase
Aahz wrote: Not quite. One critical difference between dbm and dicts is the need to remember to "save" changes by setting the key's valud again. Could you give an example of this? I'm not sure I understand what you're saying. Well, you're more likely to hit this by wrapping dbm with shelve

Re: Please help with MemoryError

2010-02-12 Thread Jeremy
On Feb 11, 6:50 pm, Steven D'Aprano wrote: > On Thu, 11 Feb 2010 15:39:09 -0800, Jeremy wrote: > > My Python program now consumes over 2 GB of memory and then I get a > > MemoryError.  I know I am reading lots of files into memory, but not 2GB > > worth. > > Are

  1   2   >