Overhead of individual python apps
I'm setting up a system that consists of several small python applications that all communicate amongst each other on the same pc. When running in Windows, launching each application generates a process, and each of those processes ends up taking up > 4MB of system memory. This memory usage is as reported by the Windows Task manager for the python.exe image name. My Question: Is there any way to reduce this per-process overhead? eg: can you set it somehow so that one python.exe instance handles multiple processes? One possibility considered is to run them as threads of a single process rather than multiple processes, but this has other drawbacks for my application and I'd rather not, Another possibility I considered is to strip out all but the most essential imports in each app, but I tested this out and it has marginal benefits. I demonstrated to myself that a simple one liner app consisting of 'x = raw_input()' still eats up > 2.7MB . I also tried -O but it, not surprisingly, did nothing for the one-liner. I'm simply running the .py files and I am still on v2.3 All help appreciated! Thanks, Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: __call__ in module?
Nope - you can't even force it by binding a __call__ method to the
module.
For future reference, you can check to see what things *are* callable
with the built-in function 'callable'.
eg (with sys instead of MyApp):
>>> import sys
>>> callable(sys)
False
Also - a thing you can do to sort of do what you want (?) is wrap the
code to be executed by the module in a main() function. eg:
#-- Start of MyApp.py --
def main(foo):
print "My cat loves", foo
if __name__ == "__main__":
import sys
main(" ".join(sys.argv[1:]))
#-- EOF --
The main function then lets you either run your module from the command
line (MyApp.py Meow Mix) or have another module use it with:
import MyApp
MyApp.main("Meow Mix")
--
http://mail.python.org/mailman/listinfo/python-list
Re: Overhead of individual python apps
Thanks - this is all very interesting... > Ah, but is that physical memory consumed, or virtual memory MAPPED > to the processes. and > for python, the "private" memory use is usually ~1.5 megabytes for a "empty" > 2.4 > process, and some of that will only occupy space in the paging file... for > firefox with > a simple page loaded into a single tab, the private space is ~10 megabytes) and most useful: > http://www.itwriting.com/dotnetmem.php I had no idea that the memory usage reported by Windows Task Manager (WTM) was so different than what I expected it would be. It doesn't seem terribly useful to me right now. After looking into that link (and discovering the perfmon app... can't believe I never knew about such an amazingly useful tool!) below are the results of some memory reporting checks that I ran on the simple one-liner app (x = raw_input()). The "On Launch" is what was reported immediately after launching the app, and the "Window Min'd" is following a simple minimization for the window. Memory report: On Launch Window Min'd -- - WTM Mem Usage: 2,756K 88K PerfMon Private bytes: 1,540,096 1,540,096 PerfMon Working set:2,822,14490,112 ** Basically it looks like the privately allocated memory for a 2.3 app that cannot be shared by other processes (close to what I want, neglecting paging possibilities) is the 1.5 MB that Fredrik reported. Presumably using perfmon as well? Maybe I'm a stickler, but this still seems pretty high for each and every application to need to privately hold. Am I still misreading this somehow? If not, I'd still love for this to be smaller... is there any way to reduce this further? Russ ** There was an interesting transient in the Working set during minimization all the way up to 9 MB... presumably this is because during minimization the "set of memory pages touched recently" jumps up because of the transactions needed for the minimization. -- http://mail.python.org/mailman/listinfo/python-list
"Compile time" checking?
Hi there, I'm pretty new to Python and am trying to figure out how to get "will this code compile?"-like code checking. To me this is a pretty basic language/environment requirement, especially when working with large projects. It is *much* better to catch errors at "compile-time" rather than at run-time. One thing I've "found" is the PyChecker module (conveniently embedded in SPE), but it doesn't seem to do that great of a job. For example, the following simple program checks out perfectly as far as PyChecker is concerned: # def tester(a,b,c): print "bogus test function",a,b,c tester(1,2,3) #this runs fine tester(1,2)#this obviously causes a run-time TypeError exception # It seems to me that this should be an obvious catch for PyChecker. I suppose you could argue that you don't want PyChecker to bark at you any time an exception would be raised since you may intentionally be causing exceptions, but this one seems a pretty simple and obvious one to catch. My questions are: - Am I missing something with my tester example? - Are there other code-checking options other than PyChecker? Any other comments appreciated (aside from things like "just right good code that doesn't have bugs like that" :) ). Thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: "Compile time" checking?
> Why not just find out, by trying to compile it? :-) This will likely certify me as a python newbie, but... how do you mean? How do you compile a .py file? If you mean to .pyc by doing an import on it, that may work fine for the simple example I typed up earlier, but that is easy to bypass by slapping the offending line in a function. The sample below also passes PyChecker with not even a warning: # def tester(a,b,c): print "bogus test function",a,b,c def try1(): tester(1,2,3) def try2(): tester(1,2)#still no error here # Do you mean something different? Also - thanks for the pylint comment... haven't tried it yet. It would be nice to have the capability in an IDE like SPE, though. -- http://mail.python.org/mailman/listinfo/python-list
Re: "Compile time" checking?
How embarassing... thanks, jk. I grabbed a copy of pychecker v0.8.14
directly (not the one in SPE) and it catches it exactly as you showed.
Now I wonder why the SPE one doesn't catch it (and why it is sooo
comparatively slow)!
Now I'm running into another snag when checking some other code I have.
Pychecker gets hung up on raw_input... it actually executes code
rather than just checking it, it seems. For example, the snippet below
hangs pychecker::
#---
while 1:
x = raw_input("meh:")
#---
Curious.
I'm going to look into some of the code checking capabilities (if
present) of Komodo and Wing. Anyone familiar enough with their ability
to comment?
--
http://mail.python.org/mailman/listinfo/python-list
Re: "Compile time" checking?
> def tester(a, b, c): > global tester > print "bogus test function", a, b, c > def tester(a, b): > print "other test function", a, b > > tester(1, 2, 3) # This runs fine. > tester(1, 2)# This too. Interesting example. In that case, pychecker does spit out a warning since it has trouble deciphering the redefinition. I have no problem whatsoever with a compiler/code-checker getting confused in such an oddball situation. As you say, it is difficult for an automated process to follow such flows. A warning is fine here (as I got with the "proper" pychecker on my initial example - it did easily catch what I thought should have been, and was, obvious). With your example, I was curious how pychecker would deal with it if you altered the flow a bit so that all calls would/should make sense in what seems to me to be logical locals order, and tried this: #--- def tester(a, b, c): global tester print "bogus test function", a, b, c def tester(a, b): print "other test function", a, b tester(1, 2) #no pychecker complaint since local tester(1, 2, 3) # pychecker complains here (?) #--- I'm a bit confused why pychecker complained where it did - I don't get how it got the 2 arg version at that point, but I actually don't really care that much due to the weirdness level of this code. A compiler (or code-checker) warning on this code is perfectly acceptable to me. I'm a big fan of Python's ability to easily rebind everything in sight, but this particular usage seems like a strange abuse I wouldn't expect a code-checker to be able to figure out. I'll just avoid writing confusing code like that... it's not only confusing to a program, but to a human as well! Dynamically massacring a function definition (as opposed to just rebinding a new implementation) like that seems odd to me. > Compile it by running it and write unit tests. ... sure, that works, I'm just used to the integrated tools I've had available to me for the last 15 years to help me write more robust code wy faster than having to unit test a zillion blocks of code when you change a single definition somewhere. PyChecker seems like it may fit the bill right now... just need to try it some more and figure out how to get around that weird raw_input thing. The basis for my first post was a jerk what-the-heck reaction to the fact that it seemed that pychecker didn't get the simple arg count mismatch error, but jk showed that that was wrong and I just have to sort out something with SPE. Cheers, Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: "Compile time" checking?
> if __name__ == '__main__':
Yep - that does it... should have thought of that. Thanks.
This works fine for pychecker with no hangage:
#---
if __name__ == "__main__":
while 1:
x = raw_input("meh:")
#---
--
http://mail.python.org/mailman/listinfo/python-list
Re: "Compile time" checking?
> if debug: print "v=%s" % (v,) Not that important, but I assume the first one was supposed to be: if debug: print "v=", s right? -- http://mail.python.org/mailman/listinfo/python-list
Suppressing checking of modules with pychecker
Does anyone know how to stop the command line pychecker from analyzing particular modules? It really gets slowed down on some big ones. In particular having 'import wx' takes a long while (30 - 60s). If you try pycheck'ing the program below it takes a while and prints a zillion warnings. #--- import wx print "Go make a sandwich while this finishes..." #--- I tried the blacklisting -b option, but can't seem to get it to work right. Plus I think it just suppresses the warnings but does not stop it from digging through the wx module. Anyone using pychecker with wxPython apps know what to do? -- http://mail.python.org/mailman/listinfo/python-list
