On 03/07/2019 18.37, Israel Brewster wrote:
> I have a script that benefits greatly from multiprocessing (it’s generating a
> bunch of images from data). Of course, as expected each process uses a chunk
> of memory, and the more processes there are, the more memory used. The amount
> used per pr
On 2019-07-03 08:37:50 -0800, Israel Brewster wrote:
> 1) Determine the total amount of RAM in the machine (how?), assume an
> average of 10GB per process, and only launch as many processes as
> calculated to fit. Easy, but would run the risk of under-utilizing the
> processing capabilities and tak
On 7/3/19 9:37 AM, ijbrews...@alaska.edu wrote:
I have a script that benefits greatly from multiprocessing (it’s generating a
bunch of images from data). Of course, as expected each process uses a chunk of
memory, and the more processes there are, the more memory used. The amount used
per pro
I have a script that benefits greatly from multiprocessing (it’s generating a
bunch of images from data). Of course, as expected each process uses a chunk of
memory, and the more processes there are, the more memory used. The amount used
per process can vary from around 3 GB (yes, gigabytes) to
On 9/9/2014 11:34 AM, Michael Torrie wrote:
On 09/08/2014 08:45 PM, kjs wrote:
You're right, a dictionary can do everything I need and more.
Actually I am wrong in suggesting a dictionary. A list or an array
would probably be more appropriate.
Thinking about it this morning, one additional r
On September 9, 2014 8:57:02 AM PDT, Michael Torrie wrote:
>On 09/09/2014 09:37 AM, Chris Angelico wrote:
>> On Wed, Sep 10, 2014 at 1:32 AM, Michael Torrie
>wrote:
>>> Yes you're correct. It is the equivalent. But it always involves
>>> lookup in the object's dictionary, which is big O order
On 09/09/2014 09:37 AM, Chris Angelico wrote:
> On Wed, Sep 10, 2014 at 1:32 AM, Michael Torrie wrote:
>> Yes you're correct. It is the equivalent. But it always involves
>> lookup in the object's dictionary, which is big O order O(n log n)
>> complexity for each and every access.
>
> Where do
On Wed, Sep 10, 2014 at 1:32 AM, Michael Torrie wrote:
> Yes you're correct. It is the equivalent. But it always involves
> lookup in the object's dictionary, which is big O order O(n log n)
> complexity for each and every access.
Where do you get that figure from? A CPython dictionary is imple
On 09/08/2014 08:45 PM, kjs wrote:
> You're right, a dictionary can do everything I need and more.
Actually I am wrong in suggesting a dictionary. A list or an array
would probably be more appropriate.
Thinking about it this morning, one additional reason why getattr and
setattr aren't appropria
Reposting to list, instead of directly to kjs
On 09/08/2014 08:45 PM, kjs wrote:
> Thanks for the consideration Michael. If you do get the data, and are
> able to run the code, let me know if you notice anything interesting.
Yeah I don't think I'll be able to have the time to download a 3 GB file
On 09/09/2014 03:45, kjs wrote:
You're right, a dictionary can do everything I need and more. This
happened to be the first thing I thought of, and I didn't imagine it
would be very expensive. I figured it was simply a different way of
defining and retrieving a class variable. IE setattr(self, f
Thanks for the consideration Michael. If you do get the data, and are
able to run the code, let me know if you notice anything interesting.
Michael Torrie:
> On 09/07/2014 02:39 PM, kjs wrote:
>> The code is minimal[0]. The only other widgets are a start button that
>> fires off the plotting and
On 09/07/2014 02:39 PM, kjs wrote:
> The code is minimal[0]. The only other widgets are a start button that
> fires off the plotting and a stop button that calls sys.exit().
Unfortunately there are no data files in your git repository so I can't
run it.
>
> Lines 112-114 appear to be causing the
Michael Torrie:
> On 09/07/2014 01:11 PM, kjs wrote:
>> Thanks for the advice. I commented out the graph generation and PyQt call
>>
> self.app.processEvents()
>>
>> where in the class __init__
>>
> self.app = QtGui.QApplication(sys.argv)
>>
>> This stopped the weakref proliferation. All
On 09/07/2014 01:11 PM, kjs wrote:
> Thanks for the advice. I commented out the graph generation and PyQt call
>
self.app.processEvents()
>
> where in the class __init__
>
self.app = QtGui.QApplication(sys.argv)
>
> This stopped the weakref proliferation. All other objects grow and
>
Antoine Pitrou:
> kjs riseup.net> writes:
>>
>> I have come to believe that the growing number of weakrefs is slowing
>> down execution. Is my analysis misguided? How can I introspect further?
>> If the slowdown can be attributed to weakref escalation, what are some
>> next steps?
>
> The way t
kjs riseup.net> writes:
>
> I have come to believe that the growing number of weakrefs is slowing
> down execution. Is my analysis misguided? How can I introspect further?
> If the slowdown can be attributed to weakref escalation, what are some
> next steps?
The way to analyze this is to build s
I built a small application using PyQt4 and pyqtgraph to visualize some
data. The app has 32 graphs that plot deques of size 512. The plots are
updated when 200 ints are cycled through each deque.
The plotting slows down in a linear manner with respect to time. In
other words after cycling through
> Python version and OS please. And is the Python 32bit or 64bit? How
>
> much RAM does the computer have, and how big are the swapfiles ?
>
Python 2.7.3
ubuntu 12.04 64 bit
4GB RAM
>
> "Fairly big" is fairly vague. To some people, a list with 100k members
>
> is huge, but not to a modern
On 02/18/2013 10:29 AM, Sudheer Joseph wrote:
HI,
I have been trying to compute cross correlation between a time series
at a location f(1) and the timeseries of spatial data f(XYT) and saving the
resulting correlation coefficients and lags in a 3 dimensional array which is
of fairly b
HI,
I have been trying to compute cross correlation between a time series
at a location f(1) and the timeseries of spatial data f(XYT) and saving the
resulting correlation coefficients and lags in a 3 dimensional array which is
of fairly big size. Though the code I made for this purpose
ption?
-juan
-Original Message-
From: Dave Angel [mailto:d...@davea.name]
Sent: Monday, November 07, 2011 1:50 PM
To: Juan Declet-Barreto
Cc: python-list@python.org
Subject: Re: memory management
On 11/07/2011 03:33 PM, Juan Declet-Barreto wrote:
> Well, I am using Python 2.5 (and the ID
On 11/7/2011 3:47 PM, Stefan Krah wrote:
Juan Declet-Barreto wrote:
Well, I am using Python 2.5 (and the IDLE shell) in Windows XP, which
ships with ESRI's ArcGIS. In addition, I am using some functions in the
arcgisscripting Python geoprocessing module for geographic information
systems (GIS)
On Tue, Nov 8, 2011 at 6:43 AM, Juan Declet-Barreto
wrote:
>
> I have a script that traverses a file structure using os.walk and adds
> directory names to a list. It works for a small number of directories, but
> when I set it loose on a directory with thousands of dirs/subdirs, it crashes
> th
On 11/07/2011 03:33 PM, Juan Declet-Barreto wrote:
Well, I am using Python 2.5 (and the IDLE shell) in Windows XP, which ships
with ESRI's ArcGIS. In addition, I am using some functions in the
arcgisscripting Python geoprocessing module for geographic information systems
(GIS) applications, wh
Juan Declet-Barreto wrote:
> Well, I am using Python 2.5 (and the IDLE shell) in Windows XP, which
> ships with ESRI's ArcGIS. In addition, I am using some functions in the
> arcgisscripting Python geoprocessing module for geographic information
> systems (GIS) applications, which can complicate t
ython-list@python.org
Subject: Re: memory management
On 11/07/2011 02:43 PM, Juan Declet-Barreto wrote:
> Hi,
>
> Can anyone provide links or basic info on memory management, variable
> dereferencing, or the like? I have a script that traverses a file structure
> using os.walk and
On 11/07/2011 02:43 PM, Juan Declet-Barreto wrote:
Hi,
Can anyone provide links or basic info on memory management, variable
dereferencing, or the like? I have a script that traverses a file structure
using os.walk and adds directory names to a list. It works for a small number
of
Hi,
Can anyone provide links or basic info on memory management, variable
dereferencing, or the like? I have a script that traverses a file structure
using os.walk and adds directory names to a list. It works for a small number
of directories, but when I set it loose on a directory with
imple memory pool/heap that students will have to use to
allocate/deallocate objects of a particular type definitely seems the
way to go. It will get the ideas across, and allow them to contrast the
garbage collection and explicit memory management paradigms.
-Daniel
--
http://mail.python.org/ma
On May 4, 12:51 pm, Daniel Neilson wrote:
> Hello,
> I'm hoping that there will be someone here with sufficient expertise
> to answer a question on Python 3 for me.
>
> I work in the Computer Science department at a large Canadian
> University. We are currently doing a feasibility analysis for
means C. If teaching C in the first year in parallel with
Python is considered too steep, then leave explicit memory
management until later in the curriculum.
(It's really a shame that Pascal is not taught any more. It
provided a fairly clean environment for teaching about things
like this, w
On 5/4/2011 12:51 PM, Daniel Neilson wrote:
Hello,
I'm hoping that there will be someone here with sufficient expertise to
answer a question on Python 3 for me.
I work in the Computer Science department at a large Canadian
University. We are currently doing a feasibility analysis for switching
C. It might be
nice as a way of introducing explicit memory management.
4) You could also build a heap (not the tree kind, but the malloc kind) in
pure Python, and give it alloc and destroy operations. Underneath it all,
things would still be reference counted/garbage collected, but that wouldn&#x
On May 4, 6:51 pm, Daniel Neilson wrote:
> In either case, if such a module is possible, any pointers you could
> provide regarding how to implement such a module would be appreciated.
The gc module will hook into the garbage collector.
The del statement will remove an object from the curren
Hello,
I'm hoping that there will be someone here with sufficient expertise
to answer a question on Python 3 for me.
I work in the Computer Science department at a large Canadian
University. We are currently doing a feasibility analysis for switching
to using Python in our first year major
On 21 Oct, 16:45, Nobody wrote:
> On Thu, 21 Oct 2010 02:34:15 -0700, Jon Clements wrote:
> > I'm after something that says: "I want 512mb of physical RAM, I don't
> > want you to page/swap it, if you can't do that, don't bother at all".
> > Now I'm guessing, that an OS might be able to grant that
On Thu, 21 Oct 2010 02:34:15 -0700, Jon Clements wrote:
> I'm after something that says: "I want 512mb of physical RAM, I don't
> want you to page/swap it, if you can't do that, don't bother at all".
> Now I'm guessing, that an OS might be able to grant that, but later on
> have to kill the proces
Jon Clements writes:
> Is there a cross-platform way using Python to guarantee that an object
> will never be swapped/paged to disk? I'll be honest and say I'm really
> not sure if this is a particular language question or rather specific
> to an OS.
>
> Under linux it appears I could create a r
Hi all,
Is there a cross-platform way using Python to guarantee that an object
will never be swapped/paged to disk? I'll be honest and say I'm really
not sure if this is a particular language question or rather specific
to an OS.
Under linux it appears I could create a ramfs and mmap a file under
le bug - why doesn't closing the plot window release all
> memory it uses? Especially when this approaches machine memory size.
>
> 3/ Are there python/matplotlib memory management tuning parameters I
> can tweak?
First off, let's clear up a couple misconceptions.
1. CPython g
monitor
[...]
I do not know what closing the window does, but in my programs, running
on debian and opensuse, I found that explicitly calling close() solved
my memory leaks with matplotlib.
3/ Are there python/matplotlib memory management tuning parameters I
can tweak?
You could try to import gc
ize to ubuntu/python - how to check this?
2/ possible bug - why doesn't closing the plot window release all
memory it uses? Especially when this approaches machine memory size.
3/ Are there python/matplotlib memory management tuning parameters I
can tweak?
--
http://mail.python.org/mailman/listinfo/python-list
gravityzoo-dmo wrote:
On 24 aug, 20:35, "Martin P. Hellwig"
wrote:
gravityzoo-dmo wrote:
Hello everyone,
I was wondering if anyone here has had any experience in running
Python in a virtualized server environment?
The reason I'm asking is the recent thing I noticed when running my
server appli
On 24 aug, 20:35, "Martin P. Hellwig"
wrote:
> gravityzoo-dmo wrote:
> > Hello everyone,
>
> > I was wondering if anyone here has had any experience in running
> > Python in a virtualized server environment?
> > The reason I'm asking is the recent thing I noticed when running my
> > server applica
gravityzoo-dmo wrote:
Hello everyone,
I was wondering if anyone here has had any experience in running
Python in a virtualized server environment?
The reason I'm asking is the recent thing I noticed when running my
server application (written in Python + Twisted);
The memory of the server applic
Hello everyone,
I was wondering if anyone here has had any experience in running
Python in a virtualized server environment?
The reason I'm asking is the recent thing I noticed when running my
server application (written in Python + Twisted);
The memory of the server application seems to only grow
On Apr 20, 9:40 am, "Hank @ITGroup" <[EMAIL PROTECTED]> wrote:
> Hi, people!
>
> Greetings~
> These days I have been running a text processing program, written by
> python, of cause.
> In order to evaluate the memory operation, I used the codes below:
>
> """
> > string1 = ['abcde']*99 # th
On Sun, 20 Apr 2008 18:40:26 +1000, Hank @ITGroup wrote:
> In order to evaluate the memory operation, I used the codes below:
>
> """
> > string1 = ['abcde']*99# this took up an obvious memory space...
> > del string1 # this freed the memory
> successfully
On 4/20/08, Hank @ITGroup <[EMAIL PROTECTED]> wrote:
> Hi, people!
>
> Greetings~
> These days I have been running a text processing program, written by
> python, of cause.
> In order to evaluate the memory operation, I used the codes below:
>
> """
> > string1 = ['abcde']*99# this took up
Hi, people!
Greetings~
These days I have been running a text processing program, written by
python, of cause.
In order to evaluate the memory operation, I used the codes below:
"""
> string1 = ['abcde']*99# this took up an obvious memory space...
> del string1
Huayang Xia schrieb:
> I have a piece of code like this:
>
> void funct(PyObject* pyobj)
> {
> char str[128];
> strncpy(str, "just a test string", sizeof(str));
> PyObject* pydata = PyObject_CallMethod(pyobj, "method_x",
> "s", str);
> Py_DECREF(py
Hi there,
I have a piece of code like this:
void funct(PyObject* pyobj)
{
char str[128];
strncpy(str, "just a test string", sizeof(str));
PyObject* pydata = PyObject_CallMethod(pyobj, "method_x",
"s", str);
Py_DECREF(pydata);
}
After the fu
[EMAIL PROTECTED] wrote:
>I just checked the comsuptiom with the 'top' unix util. I am procesing
> html docs and the amount of memory rises continiously.
what library are you using for this ?
> I am using a lot of lists and docs. Some of them with objects. Do i
> have to make any special thing i
I just checked the comsuptiom with the 'top' unix util. I am procesing
html docs and the amount of memory rises continiously.
I am using a lot of lists and docs. Some of them with objects. Do i
have to make any special thing in order to get them released back to
the Memory Manager? For instantec..
[EMAIL PROTECTED] wrote:
> Hi, I am starting to have a look to a python program that does not free
> memory (I am using python 2.4.3). As I have read about a new memory
> management in python 2.5 (http://evanjones.ca/python-memory.html) I
> decided to try the program with the new ver
[EMAIL PROTECTED] skrev:
> Hi, I am starting to have a look to a python program that does not free
> memory (I am using python 2.4.3). As I have read about a new memory
> management in python 2.5 (http://evanjones.ca/python-memory.html) I
> decided to try the program with the new ver
Hi, I am starting to have a look to a python program that does not free
memory (I am using python 2.4.3). As I have read about a new memory
management in python 2.5 (http://evanjones.ca/python-memory.html) I
decided to try the program with the new version.
With the new version of python the memory
Hi all,
I do have a problem with python and it
is that it raise an outofmemory (i comment lines in Py.java to avoid
system.exit, to debug),
i try to debug this issue with jprobe and realize
that i get the exception even although the java heap is not in the limit, i can
notice that
pytho
59 matches
Mail list logo
