Re: Well written open source Python apps
Steve M wrote: > Here is an article discussing the coding style of BitTorrent. > > http://www.onlamp.com/pub/a/python/2003/7/17/pythonnews.html > > Maybe that code is worth looking at. [didn't read this thread or that article until I saw the summary in Dr. Dobb's Python-URL] FWIW, the BitTorrent code seemed like an incredible hack to me. The above article tries to put a positive spin on it, but seriously, the code was a mess. Hats off to Mr. Cohen for creating BitTorrent in the first place, but I wouldn't go looking at that code for any "best practices". It took several days of head scratching to figure out what was really going on because the code is almost *completely* devoid of comments - even high level stuff like "this module is for X" or "this class does Y", not to mention comments to clarify code that was obscure or trying to be too cute. A day or so into it I discovered that there were two different public classes with the exact same name, so anytime you saw it used elsewhere you had to dig around to figure out which class was being used. There were also lots of more subjective things that were pretty annoying - state was passed around in various dictionaries (they were begging to be refactored into classes), a lot of the variable names seemed like misnomers, values were passed out from functions via 1-item lists, etc. - drove me nuts. :) To be clear, I'm not trying to rag on BitTorrent, just pointing out that it is probably not at all what the OP is looking for (well-written Python, stuff that is generally considered "very Pythonic"). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Extention Woes
Tuvas wrote: > I am in the process of writing an extention module, and am coming up > with lots of problems. This isn't a direct answer to your question, but what sort of extension is it? Do you need to actually write a Python extension? I ask because I've all but stopped writing Python extension modules and just use ctypes instead (i.e. write your module as a normal .dll or .so that exports some functions, and then use ctypes to call them). There are probably some cases where this approach isn't a good fit, but it has worked really well for me for a lot of different applications. Best of luck, -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: destroy your self????
KraftDiner wrote: > if I create an object like... > > obj = None > ... > obj = anObject() > > can obj set itself to none in some method of the class? No - Python doesn't work that way. What are you trying to accomplish? There's probably a way to do what you need to do, but this isn't it. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: destroy your self????
KraftDiner wrote: > Well I guess what I'm trying to achive is the invalidate the instance > of the object. > I have been using None to denote an invalide or uninitialized instance > of an object. > > There is a degenerate case in my code where a polygon has less than 3 > points and > I want to class to flag this instance of the object as invalid. > > so.. like super.self = None :) > In Python, variables don't "hold" values - variables are just named references to objects: x = 5 # x refers to the 5 integer object x = 'hi' # now x refers to the string object By reassigning a name to something else, that's all you're doing - ever other reference to that object will still be intact. If you were able to flag the degenerate case, code that uses the object would have to check for it, right? That being the case, the same code could just as easily test a member function. Better yet, if you can catch this case at the point when the object is created, just throw an exception or at least perform the test at that point only. -- http://mail.python.org/mailman/listinfo/python-list
Re: System tray Icon
Mike Pippin wrote: > How would I have an app run with just a system tray Icon??? any help > would be greatly appreciated. I have no clue where to start. Choose a GUI toolkit, e.g. wxPython. -- http://mail.python.org/mailman/listinfo/python-list
Re: Threading
Tuvas wrote: > I am trying to write a thread that will execute a function every 2 > seconds until the program is close, in which case it will stop. I can > write the program easy enough to execute the command every 2 seconds, > the problem comes when I try to close the program. It won't close the > thread. Any ideas as to what I can do? Thanks! > The simplest thing to do is call setDaemon on the thread object: t = threading.Thread(target=MyFunc, args=(1,2,3)) t.setDaemon(1) t.start() This doesn't really give your other thread a chance to shutdown cleanly though, so if you need to "guarantee" clean shutdown, then have your thread check some shared flag or have some other way to know when it needs to quit, and then have the main thread wait for it to terminate by calling Thread.join. For example, if you have a work queue, you could decide on the convention that None means all the work is done: import threading, Queue, time def Work(q): while 1: work = q.get() if work is None: break print 'Working on', work time.sleep(1) print 'Worker is quitting now' q = Queue.Queue() # this queue is shared with the worker thread # Start up a worker t = threading.Thread(target=Work, args=(q,)) t.start() # Give it stuff to do for i in range(5): q.put(i) # signal end of work q.put(None) print 'Main thread waiting for worker to be done' t.join() # returns once all the work is done print 'All done' -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: python speed
Steven Bethard wrote: > David Rasmussen wrote: > >>Harald Armin Massa wrote: >> >> >>>Dr. Armin Rigo has some mathematical proof, that High Level Languages >>>like esp. Python are able to be faster than low level code like >>>Fortran, C or assembly. >> >>Faster than assembly? LOL... :) > > > I think the claim goes something along the lines of "assembly is so hard > to get right that if you can automatically generate it from a HLL, not > only will it be more likely to be correct, it will be more likely to be > fast because the code generator can provide the appropriate optimizations". > > OTOH, you can almost certainly take automatically generated assembly > code and make optimizations the code generator wasn't able to, thanks to > knowing more about the real semantics of the program. Yeah, but the other important part of the claim has to do with just that: the real [runtime] semantics of the program. Hand optimization happens before runtime, obviously, whereas the HLL->assembly conversion can happen once the program is running and more info is known about the actual sets of data being operated on, the frequence of function calls, i.e. where the need for optimization actually exists. The optimization could even happen multiple times to adapt over time, or to allow multiple optimizations for different classes of data so that the code that actually executes is highly specialized. So, yeah, knowledge of the runtime behavior can allow a hand-optimized version to run faster than a static, automatically-generated version, but knowledge of the runtime behavior could also allow a dynamic code generator to even beat the hand-optimized version. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Python evolution: Unease
Istvan Albert wrote: Doug Holton wrote: application is so important that I expect Python 3000 will have optional type declarations integrated into the argument list." I think that *optional* part of the "optional type declaration" is a myth. It may be optional in the sense that the language will accept missing declarations but as soon as the feature is available it will become "mandatory" to use it (peer pressure, workplace practices). That's my fear - type declarations could become one of the most abused language features because they'd get used too often. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Python evolution: Unease
Roman Suzi wrote: On Tue, 4 Jan 2005, Dave Brueck wrote: It may be optional in the sense that the language will accept missing declarations but as soon as the feature is available it will become "mandatory" to use it (peer pressure, workplace practices). What about generic programming coming into fashion anytime soon? Roman, I think I've read every single thread in the past year or three wherein you've brought up generic programming, and I think you'd do well to choose a new term for the idea you're trying to convey. The term "generic programming" is too... er... generic. :) As you know, Python already includes a _lot_ of support for generic programming (a function that iterates over a sequence can easily process a list, or a string, or a tuple as input; a function that takes a file-like object can often work just as will with a true file object or a cStringIO object; etc.). So when you bring up "generic programming", it's too easy to dismiss the comment because (1) it's too vague and (2) Python already does a lot of it. So, what is your term for the type of generic programming that Python doesn't yet support? Interfaces? Protocols? Adapters? Metatype hierarchies? -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Python evolution: Unease
Roman Suzi wrote: The term "generic programming" is too... er... generic. :) Nope. It is not generic. It has it's definition made by the co-author of STL - A.Stepanov. And the Boost C++ library (many of us know it as Boost Python) standardise on the approach, AFAIK. Ok, "too broad" then; Python already supports at least some aspects of generic programming (at least, in the sense that I think you mean it), so it'd be good to spell out what specific features you're referring to. Python could have honest support of concepts. Everything else will be available with them. "Concepts" is a pretty generic term too! ;-) Do you mean concepts as defined here: http://www.boost.org/more/generic_programming.html ? And BTW, are we really disputing? No, not at all - I'm just trying to better understand what you mean. Words like "generic" and "concepts" don't yet have a widely recognized, strict definition in the context of programming. If somebody has assigned some specific definition to them, that's great, it's just not universal yet so references and additional explanations are helpful. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Determining if a client PC has an Internet connection
torment wrote: [snip] Have you tried just parsing the output from the command "ipconfig"? It's pretty obvious from the output that might give you if a connection is availible. It's tempting to use ipconfig's output, but the info it gives you is unreliable - you can incorrectly infer the presence of an net connection when you don't really have one (if you also run VMWare, if you have installed a loopback adapter, if you have a static public IP, etc.). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: how to call python code from C#
Peter Hansen wrote: paritosh mahana wrote: How can I call python code from my C# code. [snip] You could use ctypes or the pywin32 package to provide your Python code with an ActiveX interface. Then you could just use it via COM, like any other COM object. Lots of references available via Google if you want to learn more about this approach... Lemme add my two cents and say that this approach works well. We have a component that uses ctypes and runs as a COM local server (its own .exe) and we currently use it both from Internet Explorer and from a C# application. COM can be hairy initially, but if you have any experience with COM then this approach is pretty straightforward. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Best python postgres module?
Roland Heiber wrote: i recently migrated from mysql to postgresql and did use severel python postgres-modules. All do what they are designed for, so which one would you use? psycopg, pygresql, pypgsql? psycopg seems to be the best solution for heavy traffic/multiple connections i have no real testing environment, so any advice which one to use for different usecases would be nice. Hi Roland, We've been using psycopg (via Zope and straight Python programs) for about a year and haven't had any problems at all. We don't store much binary data so I don't know how well psycopg handles blobs and what-not, but performance and reliability has been fine. I have no idea how psycopg stacks up against the others; in the end we chose it because the project appeared to be active, it was easy enough to set up, and a few test scripts we made all worked (how's that for rigorous evaluation? ;-) ). HTH, Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Where is WSAEnumNetworkEvents???
[EMAIL PROTECTED] wrote:
I was trying to write an asyncronous TCP server for win32 using
WSAEventSelect (which lives if win32file). Such events require
WaitForMultipleObjects (which lives if win32event) and
WSAEnumNetworkEvents WHICH IS NOT EXPOSED. This makes WSAEventSelect
useless. Does somebody know how to add the WSAEnumNetworkEvents and
WSANetwotkEvents structure win32file?
Maybe I'm missing something?
You could both build the structures and call the functions in ctypes
(http://starship.python.net/crew/theller/ctypes/)
[untested code follows]
from ctypes import *
class WSANETWORKEVENTS(Structure):
_fields_ = [('lNetworkEvents', c_long),
('iErrorCode', c_int * 10) # 10 = FD_MAX_EVENTS
]
You'd access the API via windll.ws2_32.WSAEnumNetworkEvents, e.g.
networkEvents = WSANETWORKEVENTS()
i = windll.ws2_32.WSAEnumNetworkEvents(s, hEventObject, byref(networkEvents))
if i != 0:
# Handle an error
HTH,
Dave
--
http://mail.python.org/mailman/listinfo/python-list
Re: OT: why are LAMP sites slow?
Paul Rubin wrote: How would you go about building such a site? Is LAMP really the right approach? Two major problems I've noticed, don't know if they are universal, but they sure hurt the performance: 1) Some sites have not put any thought into caching - i.e. the application server is serving up images or every single page is dynamically generated even though all (or most) of it is static such that most of the requests just aren't cacheable. 2) Because a database is there, it gets used, even when it shouldn't, and it often gets used poorly - bad or no connection pooling, many trips to the database for each page generated, no table indices, bizarro database schemas. Overall I'd say my first guess is that too much is being generated on the fly, often because it's just easier not to worry about cacheability, but a good web cache can provide orders of magnitude improvement in performance, so it's worth some extra thought. One project we had involved the users navigating through a big set of data, narrowing down the set by making choices about different variables. At any point it would display the choices that had been made, the remaining choices, and the top few hits in the data set. We initially thought all the pages would have to be dynamically generated, but then we realized that each page really represented a distinct and finite state, so we went ahead and built the site with Zope + Postgres, but made it so that the URLs were input to Zope and told what state to generate. The upshot of all this is that we then just ran a web spider against Zope any time the data changed (once a week or so), and so the site ended up "feeling" pretty dynamic to a user but pretty much everything came straight out of a cache. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: OT: why are LAMP sites slow?
Steve Holden wrote: Paul Rubin wrote: Skip Montanaro <[EMAIL PROTECTED]> writes: It's more than a bit unfair to compare Wikipedia with Ebay or Google. Even though Wikipedia may be running on high-performance hardware, it's unlikely that they have anything like the underlying network structure (replication, connection speed, etc), total number of cpus or monetary resources to throw at the problem that both Ebay and Google have. I suspect money trumps LAMP every time. I certainly agree about the money and hardware resource comparison, which is why I thought the comparison with 1960's mainframes was possibly more interesting. You could not get anywhere near the performance of today's servers back then, no matter how much money you spent. Re connectivity, I wonder what kind of network speed is available to sites like Ebay that's not available to Jane Webmaster with a colo rack at some random big ISP. Also, you and Tim Danieliuk both mentioned caching in the network (e.g. Akamai). I'd be interested to know exactly how that works and how much difference it makes. It works by distributing content across end-nodes distributed throughout the infrastructure. I don't think Akamai make any secret of their architecture, so Google (:-) can help you there. They definitely didn't make it a secret - they patented it. The gist of their approach was to put web caches all over the place and then have their DNS servers resolve based on where the request was coming from - when your browser asks their DNS server where whatever.akamai.com is, they try to respond with a web cache that is topologically close. Of course it makes a huge difference, otherwise Google wouldn't have registered their domain name as a CNAME for an Akamai node set. Yes and no - web caching can be very beneficial. Web caching with Akamai may or may not be worth the price; their business was originally centered around the idea that quality bandwidth is expensive - while still true to a degree, prices have fallen a ton in the last few years and continue to fall. And who knows what sort of concessions they made to win the Google contract (not saying that's bad, just realize that Akamai would probably even take a loss on the Google contract because having Google as a customer makes people conclude that their service must make a huge difference ;-) ). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: OT: why are LAMP sites slow?
Fredrik Lundh wrote: Tim Daneliuk wrote: THis is why, IMHO, things like SOAP a laughable - RPC is a poor foundation for reliable, durable, and high-performance TP. It might be fine for sending an order or invoice now and then, but sustained through- put of the sort I think of as "high" performance is likely never going to be accomplished with session-oriented architectures. does 50 gigabytes per day, sustained, count as high performance in your book? In and of itself, no. But really, the number is meaningless in transaction processing without some idea of the size and content of the messages (i.e. if it's binary data that has to be base64 encoded just to make it work with SOAP) because what's most interesting is the number of transactions that can be handled. If 50 GB/day is just a measurement of the throughput of the data transport layer, then that's fairly low - less than 5Mbps! -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Possibly OT: Controlling winamp with Python
Brent W. Hughes wrote: I'm running Windows XP and I'm using winamp to listen to internet radio stations. Occasionally, an annoying commercial will come on. I would like to write a Python program that, when run, will, in essence, push on winamp's mute button. Then, after say 20 seconds, it will push the mute button again to restore the sound. Is such a thing possible? Yes - IIRC WinAmp supports some "global" Windows messages that any app can send to make WinAmp do different things. Don't know if there's a "mute" command, but at the very leasy you could control the volume. You'd have to read their SDK and then use something like ctypes to fire the Windows message - very doable. If that doesn't do what you want, then you could go the more hacky route and grab the WinAmp window and fire a button click in the precise location. Something like AutoIt (which is controllable via Python) could work as well. Another route would be to just mute all audio in the system for 20 seconds - that might be the easiest approach of all. Again, ctypes is your friend - figure out what APIs you'd need to call and people here can help you come up with the Python code to call those APIs. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there something similar to ?: operator (C/C++) in Python?
D H wrote: > Peter Hansen wrote: >>D H wrote: >>>Peter Hansen wrote: Bo Peng wrote: >>Actually that's just one possible answer. Whether it's _the_ answer is >>obviously again a matter of opinion, and as usual we differ. > > Quit being such an idiot and refuting everything I say. The answer is > simply no, just use an if statement instead. Please keep the discussion civil; please help keep c.l.py a nice place to visit. Thanks, Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Lisp development with macros faster than Python development?..
[EMAIL PROTECTED] wrote: > I've been reading the beloved Paul Graham's "Hackers and Painters". > He claims he developed a web app at light speed using Lisp and lots > of macros. > > It got me curious if Lisp > is inherently faster to develop complex apps in. It would seem if you > could create your own language in Lisp using macros that that would be > quite an advantage > > I realize that Python has operator overloading and OOP so I'm not sure. > > Any ideas? Any *evidence* one way or another? Well, his Viaweb company was founded in about '95, right? So he probably just used Lisp because Python wasn't as well known yet. ;-) IMO one of Python's strengths over Lisp is that it plays well with many other technologies. It should be remembered that Graham's use of Lisp in Viaweb was in building a web application, so that Lisp's main links to the "outside world" were the filesystem and Apache. They didn't use a database, and Apache had to be modified in order to work with Lisp. -- http://mail.python.org/mailman/listinfo/python-list
Re: Yet Another Python Web Programming Question
Daniel Bickett wrote: > He would read the documentation of Nevow, Zope, and Quixote, and would > find none of them to his liking because: > > * They had a learning curve, and he was not at all interested, being > eager to fulfill his new idea for the web app. It was his opinion that > web programming should feel no different from desktop programming. > > * They required installation (as opposed to, simply, the placement of > modules), whereas the only pythonic freedom he had on his hosting was > a folder in his /home/ dir that was in the python system path. I've been playing with CherryPy lately, and it's been a lot of fun (for me, it meets both the above requirements - obviously anything will have a learning curve, but CherryPy's is very small). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Decline and fall of scripting languages ?
Paul Rubin wrote: > Ruby just doesn't interest me that much though (maybe I'm missing > something). I don't think you are. My impression is that if you've never used Python or Ruby, you'll generally end up liking whichever of the two you really discover first (since the common case is that you're coming from Java/C++/PHP/etc - the more mainstream languages). IIRC, the creator of Ruby got really hung up on the fact that Python was not a pure OO language, so he decided to make a new language that was (this was in the pre-metaclass, old-style class days of Python). > I was hoping for a concise explanation of what Rails does I'd say it's similar to Zope in that (1) For both, the introductory tutorials make it seem deceptively easy to use, but they hide a sometimes steep learning curve (2) For both, it's hard to find clear, concise documentation midway between introductory tutorials and reading the source code (3) For both, being frameworks, you have to commit yourself to them almost entirely and be prepared for some serious lock-in (4) Oh yeah, they're both web application frameworks :) (ROR has libraries to aid in building database-centric web applications: it includes an database ORM, a web templating language, libraries for user sessions, etc.) > and whether it's feasible to do something like it in (say) Python. I > did look at the rubyonrails.com site but there were too many marketing > buzzwords and downloadable videos for me to deal with. Yes, it's incredibly feasible. I think the Subway project is sort of heading down a similar path but using Python instead. I've tried a couple of times to use Ruby on Rails and, I have to admit, I had a tough time cutting through the hype (also, it seemed like the preferred method of learning about features was through downloading large videos). The ActiveRecord library (for handling mapping objects to the database) seems sort of powerful, but the tutorials and demo videos make a huge deal about how ROR can generate a web form by inspecting the database table metadata. (Useful? Probably. Mind-blowingly cool? Hardly.) Beyond ActiveRecord, there is some additional stuff to help you build Model-View-Controller web UIs, and then lots of the standard web app components (user sessions, security, logging, etc.). I think ROR's big selling point isn't technology-related at all: it's hype machine has helped build an active community, and it's a single framework as opposed to Python's bazillions. :) -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Pervasive Database Connection
Alex Le Dain wrote: > What is the best way to access a Pervasive database on another machine? Hi Alex, You can use ODBC / ADO to access Pervasive DBs. I found this page helpful: http://www.mayukhbose.com/python/ado/index.php -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Why do Pythoneers reinvent the wheel?
Stefano Masini wrote: > I wonder how many people (including myself) have implemented their own > versions of such modules, at least once in their pythonic life. I > indeed have my own odict (even same name! :). My own pathutils > (different name, but same stuff). My own validate... and so forth. > > This is just too bad. > There are a few ares where everybody seems to be implementing their > own stuff over and over: logging, file handling, ordered dictionaries, > data serialization, and maybe a few more. > I don't know what's the ultimate problem, but I think there are 3 main > reasons: > 1) poor communication inside the community (mhm... arguable) > 2) lack of a rich standard library (I heard this more than once) > 3) python is such an easy language that the "I'll do it myself" evil > side lying hidden inside each one of us comes up a little too often, > and prevents from spending more time on research of what's available. IMO the reason is something similar to #3 (above and beyond #1 and #2 by a long shot). The cost of developing _exactly_ what you need often is (or at least *appears* to be) the same as or lower than bending to use what somebody else has already built. (my wheel reinvention has typically covered config files, logging, and simple HTTP request/response/header processing) > It seems to me that this tendency is hurting python I think it helps on the side of innovation - the cost of exploring new ideas is cheaper than in many other languages, so in theory the community should be able to stumble upon truly great ways of doing things faster than would otherwise be possible. The problem lies in knowing when we've found that really good way of doing something, and then nudging more and more people to use it and refine it without turning it into a bloated one-size-fits-all solution. I think we have half of what we need - people like Fuzzyman coming up with handy modules and then making them available for others to use. But right now it's hard for a developer to wade through all the available choices out there and know which one to pick. Maybe instead of being included in the standard library, some modules could at least attain some "recommended" status by the community. You can't exactly tell people to stop working on their pet project because it's not special or different enough from some other solution, so maybe the solution is to go the other direction and single out some of the really best ones, and hope that the really good projects can begin to gain more momentum. For example, there are several choices available to you if you need to create a standalone Windows executable; if it were up to me I'd label py2exe "blessed by the BDFL!", ask the other tool builders to justify the existence of their alternatives, and then ask them to consider joining forces and working on py2exe instead. But of course I'm _not_ in charge, I don't even know if the BDFL likes py2exe, and it can be really tough knowing which 1 or 2 solutions should receive recommended status. FWIW, RubyOnRails vs all the Python web frameworks is exactly what you're talking about. What makes ROR great has little to do with technology as far as I can tell, it's all about lots of people pooling their efforts - some of them probably not seeing things develop precisely as they'd prefer, but remaining willing to contribute anyway. Many projects (Python-related or not) often seem to lack precisely what has helped Python itself evolve so well - a single person with decision power who is also trusted enough to make good decisions, such that when disagreements arise they don't typically end in the project being forked (the number of times people disagreed but continued to contribute to Python is far higher than the number of times they left to form Prothon, Ruby, and so on). In the end, domain-specific BDFLs and their projects just might have to buble to the top on their own, so maybe the best thing to do is find the project you think is the best and then begin contributing and promoting it. > and I wonder if > there is something that could be done about it. I once followed a > discussion about placing one of the available third party modules for > file handling inside the standard library. I can't remember its name > right now, but the discussion quickly became hot with considerations > about the module not being "right" enough to fit the standard library. I think an extremely rich standard library is both a blessing and a curse. It's so handy to have what you need already there, but as you point out it becomes quite a debate to know what should be added. For one, a module to be added needs to be sufficiently broad in scope and power to be widely useful, but this often breeds complexity (e.g. the logging package added in Py2.3 sure looks powerful, but other than playing around with it for a few minutes I've never used it in a real app because it's a little overwhelming and it see
Re: Kill GIL
Mike Meyer wrote: [EMAIL PROTECTED] (Aahz) writes: In article <[EMAIL PROTECTED]>, Mike Meyer <[EMAIL PROTECTED]> wrote: Here here. I find that threading typically introduces worse problems than it purports to solve. Threads are also good for handling blocking I/O. Actually, this is one of the cases I was talking about. I find it saner to convert to non-blocking I/O and use select() for synchronization. That solves the problem, without introducing any of the headaches related to shared access and locking that come with threads. This whole tangent to the original thread intrigues me - I've found that if you're going to use threads in any language, Python is the one to use because the GIL reduces so many of the problems common to multithreaded programming (I'm not a huge fan of the GIL, but its presence effectively prevents a pure Python multithreaded app from corrupting the interpreter, which is especially handy for those just learning Python or programming). I've done a lot of networking applications using select/poll (usually for performance reasons) and found that going that route *can* in some cases simplify things but it requires looking at the problem differently, often from perspectives that seem unnatural to me - it's not just an implementation detail but one you have to consider during design. One nice thing about using threads is that components of your application that are logically separate can remain separate in the code as well - the implementations don't have to be tied together at some common dispatch loop, and a failure to be completely non-blocking in one component doesn't necessarily spell disaster for the entire app (I've had apps in production where one thread would die or get hung but I was relieved to find out that the main service remained available). Another related benefit is that a lot of application state is implicitly and automatically managed by your local variables when the task is running in a separate thread, whereas other approaches often end up forcing you to think in terms of a state machine when you don't really care* and as a by-product you have to [semi-]manually track the state and state transitions - for some problems this is fine, for others it's downright tedious. Anyway, if someone doesn't know about alternatives to threads, then that's a shame as other approaches have their advantages (often including a certain elegance that is just darn *cool*), but I wouldn't shy away from threads too much either - especially in Python. -Dave * Simple case in point: a non-blocking logging facility. In Python you can just start up a thread that pops strings off a Queue object and writes them to an open file. A non-threaded version is more complicated to implement, debug, and maintain. -- http://mail.python.org/mailman/listinfo/python-list
Re: For American numbers
Peter Hansen wrote: Martin v. Löwis wrote: Peter Hansen wrote: For the rest of the computer world, unless I've missed a changing of the guard or something, "kilo" is 1024 and "mega" is 1024*1024 and so forth... In case this isn't clear yet: you have missed a changing of the guard or something. "kibi" is 1024, "mebi" is 1024*1024 and so forth. "kilo" is 1000. Yeah, saw the link. The IEC doesn't speak for me, I'm afraid. And as the wikipedia notes, "As of 2005 this naming convention has not gained widespread use." I suspect I and many others will go to our graves not being comfortable mebbling and gibbling over our bytes, though we'll probably spend a lot of time kibbling over the issue... Indeed - it's a little early to say that we've missed a changing of the guard! :) Multiple definitions aside, "kilo" and "mega" are far too entrenched - even if I could manage to say "kibibyte" with a straight face, I'd get nothing but blank stares in return. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Kill GIL
Donn Cave wrote: Quoth Dave Brueck <[EMAIL PROTECTED]>: ... | Another related benefit is that a lot of application state is implicitly and | automatically managed by your local variables when the task is running in a | separate thread, whereas other approaches often end up forcing you to think in | terms of a state machine when you don't really care* and as a by-product you | have to [semi-]manually track the state and state transitions - for some | problems this is fine, for others it's downright tedious. I don't know if the current Stackless implementation has regained any of this ground, but at least of historical interest here, the old one's ability to interrupt, store and resume a computation could be used to As you may know, it used to be, in Stackless Python, that you could have both. Your function would suspend itself, the select loop would resume it, for something like serialized threads. (The newer version of Stackless lost this continuation feature, but for all I know there may be new features that regain some of that ground.) Yep, I follow Stackless development for this very reason. Last I heard, a more automatic scheduler was in the works, without which in can be a little confusing about when non-I/O tasks should get resumed (and by who), but it's not insurmountable. Ideally with Stackless you'd avoid OS threads altogether since the interpreter takes a performance hit with them, but this can be tough if you're e.g. also talking to a database via a blocking API. I put that together with real OS threads once, where the I/O loop was a message queue instead of select. A message queueing multi-threaded architecture can end up just as much a state transition game. Definitely, but for many cases it does not - having each thread represent a distinct "worker" that pops some item of work off one queue, processes it, and puts it on another queue can really simplify things. Often this maps to real-world objects quite well, additional steps can be inserted or removed easily (and dynamically), and each worker can be developed, tested, and debugged independently. I like threads when they're used in this way, as application components that manage some device-like thing like a socket or a graphic user interface window, interacting through messages. Even then, though, there tend to be a lot of undefined behaviors in events like termination of the main thread, receipt of signals, etc. That's how I tend to like using threads too. In practice I haven't found the undefined behaviors to be too much trouble though, e.g. deciding on common shutdown semantics for all child threads and making them daemon threads pretty much takes care of both expected and unexpected shutdown of the main thread. Usings threads and signals can be confusing and troublesome, but often in cases where I would use them I end up wanting a richer interface anyway so something besides signals is a better fit. -Dave -- http://mail.python.org/mailman/listinfo/python-list
ZODB performance (was Re: low-end persistence strategies?)
Chris Cioffi wrote: I'd like to second this one...ZODB is *extremely* easy to use. I use it in projects with anything from a couple dozen simple objects all the way up to a moderately complex system with several hundred thousand stored custom objects. (I would use it for very complex systems as well, but I'm not working on any right now...) Chris (or anyone else), could you comment on ZODB's performance? I've Googled around a bit and haven't been able to find anything concrete, so I'm really curious to know how ZODB does with a few hundred thousand objects. Specifically, what level of complexity do your ZODB queries/searches have? Any idea on how purely ad hoc searches perform? Obviously it will be affected by the nature of the objects, but any insight into ZODB's performance on large data sets would be helpful. What's the general ratio of reads to writes in your application? I'm starting on a project in which we'll do completely dynamic (generated on the fly) queries into the database (mostly of the form of "from the set of all objects, give me all that have property A AND have property B AND property B's value is between 10 and 100, ..."). The objects themselves are fairly dynamic as well, so building it on top of an RDBMS will require many joins across property and value tables, so in the end there might not be any performance advantage in an RDBMS (and it would certainly be a lot work to use an object database - a huge portion of the work is in the object-relational layer). Anyway, thanks for any info you can give me, -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: kill a process in XP
Tor Erik Sønvisen wrote: From my Python-program I spawn a new process. When using P_NOWAIT spawnl returns the pid but in windows it returns a process handle. Later I want to kill this process. How can I do this when I only have the process handle? Try ctypes - if it's really a Windows handle, then this should work: from ctypes import * windll.kernel32.TerminateProcess(h, 0) # or whatever return code you want -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Python and "Ajax technology collaboration"
John Willems wrote: Interesting GUI developments, it seems. Anyone developed a "Ajax" application using Python? Very curious thx (Ajax stands for: XHTML and CSS; dynamic display and interaction using the Document Object Model; data interchange and manipulation using XML and XSLT; asynchronous data retrieval using XMLHttpRequest; and JavaScript binding everything together We're using it in a couple of projects with Zope as the backend and it works really well - the web applications are way more responsive now (I'm hoping someone with lots of Zope knowledge will redo the ZMI itself using this approach). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Dealing with config files what's the options
Jorgen Grahn wrote: On Tue, 22 Feb 2005 20:38:28 -0500, Tom Willis <[EMAIL PROTECTED]> wrote: How are the expert pythoneers dealing with config files? ... Any ideas? How about writing them in Python? Depending on who will be editing the config files, this can be a great approach. At the simplest level, a config.py file like this is so easy to use: # Net settings timeoutSec = 5.3 maxConnections = 3 # Other stuff foo = 'bar' This type of a format is easy to use for just about anybody who has ever had to use config files before. What's nice is that the code to use it is straightforward too: import config conn = config.maxConnections ... A few times I've tried to use accessor functions to ensure that the values are present or valid or whatever, but I stopped doing that because in practice it's just not needed (again, for users who are familiar with the concept of config files). A slightly more elaborate approach gives you full structure: class Net: maxConnections = 12 class System: class Logging: root = '/var/logs' This prevents individual setting names from getting unwieldy, and the code that uses it can be pretty readable too: logRoot = config.System.Logging.root or, if there are lots of retrievals to do: Logging = config.System.Logging logRoot = Logging.root ... etc ... Using classes asks a little bit more of the users (they can break it a little more easily), but again, in practice it really hasn't been a problem at all. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Dealing with config files what's the options
Tom Willis wrote:
On Fri, 25 Feb 2005 15:02:04 -0700, Dave Brueck
How about writing them in Python?
Depending on who will be editing the config files, this can be a great approach.
[snip]
I actually thought of this, and I was kind of on the fence due to the
intended audience.
I don't think it's too much to ask that they are comfy with the
concept of variables. I mean, if it was a shell script they'd be at
the top of the file anyway.
Then again I'm some what hesitant to help them make the connection
that I'm giving them the ability to indirectly edit the code. Kind of
like opening pandoras box. Once the figure out they can open any file
(*.py) with notepad, there will be utter anarchy and I'll get the call
at 4am that somethings wrong with the production data.
If you're giving them an executable (i.e. py2exe'd), then you can just exclude
config.py from the exe.
Either way, if you're not so worried about malicious breakage but ignorant
breakage, then you could always name your config file something like
'options.cfg' and then:
import new, sys
# Do this at app startup
sys.modules['config'] = new.module('config')
exec file('options.cfg').read() in config.__dict__
# All other modules do this
import config
conn = config.maxConnections
... etc ...
-Dave
--
http://mail.python.org/mailman/listinfo/python-list
Re: urllib2 meta-refresh
Artificial Life wrote: urllib2 does not seem to be able to handle META-REFRESH in an html document. I just get back the html to the page that is supposed to forward me to the intended page. Right - urllib2 is for working with protocols (like HTTP) to transfer data, whereas META-REFRESH is an application (browser) level "instruction" *in* that data. Compare this to a 302 HTTP response header (a simple redirect) - urllib2 can handle it because it is part of HTTP. Any way around this? Sure - META-REFRESH is an instruction to the browser, so have your code be the browser: scan the HTML for the tag, extract the URL, and send that new URL off to urllib2. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: intigrate the PyGame module with my Python
Warren Postma wrote: 1. Downloaded the windows binary for python 1.5.2 from python.org. Pygame uses Python 1.5.2 still!? :-) Oi. Nah, must have been a typo, as www.pygame.org lists Windows installers for Python 2.2, 2.3, and 2.4: http://www.pygame.org/download.shtml -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Apparently, I don't understand threading
[EMAIL PROTECTED] wrote: The following script does not behave as expected. I wanted to terminate the 'mythread' thread after 5 seconds. What I see is the threading.Thread call blocking until the 15 second sleep is done. Suggestions? [snip] foo = FooClass() mythread = threading.Thread(foo.stall()) I think you meant to write: mythread = threading.Thread(target=foo.stall) In the original version, you're _calling_ foo.stall at the time of thread creation. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Quixote+Nevow+LivePage
Jp Calderone wrote: On 3 Dec 2004 22:02:51 -0800, Mir Nazim <[EMAIL PROTECTED]> wrote: Q1) Is it possibe to use "Nevow + LivePage + Quixote" together in a web app. Live Page is really important for me as I am not into content oriented web apps. Q2) Is is nessary that LivePage only with Twisted or can work with any web server like Apache. I haven't used LivePage myself, but someone in the know tells me that LivePage requires an extra, non-HTTP connection to operate, so will pretty much only work with Twisted. Do you have a reference that says this - I'd like to know for sure. I did a quick perusal of the code and thought that all the communication was via the XmlHttpRequest functionality available in a lot of modern browsers (IE, Mozilla, Safari). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Distributing closed source modules
Jiri Barton wrote:
I'd like to be able to distribute some python modules of my system (plugins)
without the source. So far, I have done this by including only the *.pyc
files. However, I have recently found they are platform dependent and
python version dependent.
This approach has been very convenient because I don't have to mess up with
__import__ and the like - which seem to be kind of a pain when inter-module
dependencies are introduced.
Can some one point me in another direction of protecting the code? I know
and this whole thing just does not sound right to me either but I am forced
to do so.
Protecting code in any language is pretty tough and/or futile, but you can
Google the archives if you're interested in reading more on that.
Anyway, you can create a module on the fly like this (untested):
import new, sys
name = 'MyModule'
m = sys.modules[name] = new.module(name)
exec codeStr in m.__dict__
where codeStr is a string that contains the source code of your module (e.g.
from file('somemodule.py').read() ).
You can combine the above with whatever mechanism you come up with for
distributing the code itself. You could store it in an encrypted archive file,
you could download it on the fly from a remote server over a secure connection, etc.
-Dave
--
http://mail.python.org/mailman/listinfo/python-list
Re: Distributing closed source modules
Fuzzyman wrote: Dave Brueck wrote: It's certainly something lot's of people are interested in. I guess it depends who your audience is. If ytour code isn't for *mass* distribution - the chances of people putting a lot of effort into breaking it are greatly reduced. I don't htink it's necessarily futile. By "futile" I meant that, if the code ends up running on a user's machine, then a sufficiently motivated person could crack it wide open, regardless of implementation language - the only way to truly protect the code is to never let it out of your hands (i.e. it's accessible just via a web service). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Distributing closed source modules
Fuzzyman wrote: Dave Brueck wrote: By "futile" I meant that, if the code ends up running on a user's machine, then a sufficiently motivated person could crack it wide open, regardless of implementation language - the only way to truly protect the code is to never let it out of your hands (i.e. it's accessible just via a web service). I understand what you are saying - using hte word 'futilew' implies that code is *likely* to be broken, not that it is *theoretically possible* for it to be broken. If code has a small user base it is probable that there is plenty that can be done to make breaking the code a lot harder. There are also legitimate reasons why someone would want to do this. 'Futile' is definitely a misleading response :-)3 Not really. For all practical purposes, shipping .pyc files is probably sufficient for most of the software out there: (1) it's a high enough "fence" for almost all users, (2) for most programs, getting the source code and being able to do something with it are two very different things, and (3) for most programs, there really is no proprietary magic worth protecting. So, when somebody says it's not good enough, and they need something better, I have to admit I'm initially skeptical of their perceived need for "better" protection of the source code (there _are_ some cases where it should be protected, but they are much less common than people seem to think). One of two things is probably true in these cases: 1) The value of the source code is overestimated - yes, it's a nice program, but there's not really anything in there to warrant the higher development/deployment/debugging costs associated with more security. As such, nobody is really going to care enough to crack the code. And if anybody does, it's unlikely that they'll actually do anything with the code. Thus, the effort to secure the code more is futile - it's ineffective because the effort will never provide any benefit. OR 2) The code really does have some innovative, proprietary algorithm, like a video codec with wildly improved compression (that for some reason you've implemented in pure Python ;-) ). If the value of the code is really high, then no amount of security is going to prevent people from getting at it - trying to protect your code is futile because no matter how high a wall you create, sufficiently determined people will climb over it. Plus, protecting the source code may be the least of your worries (if they're willing to steal your code, they may just as well be willing to use your library illegally, etc.). It's a question that often comes up on comp.lang.python - and the reply is often "don't bother, it's not possible - and why do you want to do that anyway". This is a response that is likely to turn people towards other languages Perhaps the response could be framed better, but at the same time it _is_ a pretty honest response, and maybe Python really _isn't_ the language for such people. It's just like people who ask for curly braces - Python is not the language for them. So if I asked for braces, a lot of the c.l.py responses would be geared towards helping me understand that they aren't really needed, but if I insist that I have to have them, then maybe Python isn't for me. :) So, when the question comes up, I don't mind offering some suggestions, but the suggestions will always include the disclaimer that it's probably a waste of time & effort - IMO leaving that part out would be misleading. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Sockets
Dan wrote:
>On Thu, 7 Apr 2005 21:52:11 -0500, [EMAIL PROTECTED] wrote:
>>Python strings always carry their length, and can have embedded NULs.
>> s.write("\0\1\2\3")
>>should write 4 bytes to the socket 's'.
>
>I'm taking binary data from a database, so it's not really a Python
>string. Is there an easy way to convert it?
Unless I'm misunderstanding you, you don't need to. It's not that Python
normal strings are "encoded" in some special format, it's just that a
string object does not make a distrinction between what is traditionally
called "text" and "binary data".
Play around with this at a Python prompt to see for yourself. For example:
>>> img = file('foo.jpg', 'rb').read() # use a real image filename though
>>> img[:50]
'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x01\x00H\x00H\x00\x00\xff\xdb\x00C\x00
\x05\x03\x04\x04\x04\x03\x05\x04\x04\x04\x05\x05\x05\x06\x07\x0c\x08\x07\x07\x07
\x07\x0f\x0b\x0b\t'
>>>
So, the fact that you're getting the data from a database is probably
immaterial.
HTH,
Dave
--
http://mail.python.org/mailman/listinfo/python-list
Re: Are circular dependencies possible in Python?
Tim Tyler wrote: Like C, Python seems to insist I declare functions before calling them - rather than, say, scanning to the end of the current script when it can't immediately find what function I'm referring to. Yes and no. Yes, they have to exist before you can use them (that only makes sense), but no, they don't have to be placed in the same order in the source code like they do with C: >>> def A(count): ... print 'A', count ... B(count + 1) ... >>> def B(count): ... print 'B', count ... if count < 10: ... A(count + 1) ... >>> B(0) B 0 A 1 B 2 ... See - all that matters is that they exist before you call them. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: semicolons
Mage wrote: I amafraid of I will stop using semicolons in other languages after one or two months of python. However I see that python simply ignores the semicolons atd the end of the lines. What's your advice? I don't want to write full-of-typo php scripts but I see the logic of the python syntax too. Don't punish Python for the shortcomings of other languages! :) -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Can't Stop Process On Windows
Dan wrote: I have a python script running under Windows XP that I need to terminate from the keyboard. A control-c works fine under Linux, but not under Windows. I'm pretty sure that the culprit is 'select' that I'm using to multiplex socket i/o, which seems to be blocking the keyboard interrupt. Is there some way around this? Are you calling select with a long timeout? An alternative would be to call select with a much shorter timeout, but call it multiple times from a loop. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.4 killing commercial Windows Python development ?
Terry Reedy wrote: I guess I don't understand some people's determination to not have users install fully useable Python on their Windows machines. [snip] To put it another way, needing a Python interpreter to run .py files is no different from, for instance, needing a movie player to run .mpg files, and all Windows users are or need to become familiar with that general concept. Also, I think it a bit 'anti-social' to hide usage of Python. If all Python Windows programs ran with a normal, communally installed Python, then users would gradually get the idea that having Python installed is much like having Shockwave and other utility platforms installed, and that is is part of a 'fully loaded' Windows system to have a .py player installed. If there is something about the default install of Python on Windows that makes it less desireable or less easy than other platforms, then maybe that can be fixed. To make installation easier, maybe someone could write a small .exe that could be frozen with scripts or run with installers and that would detect the presence/absence of the needed Python version and offer an auto download and install if needed. I mostly agree with the sentiment, but it does break down a little in practice. At least currently it does - like you said, this is fixable, but nobody has signed up to fix it yet. The main thing that's needed is a zero-input Python distribution - a Python runtime, if you will - that (1) gets installed to a "good" place (2) does so without asking the user to do anything, (3) can coexist with different versions of the runtime, and (4) is easily detectable by applications wanting to use it. One other minor component is a small launcher executable, because on Windows it's non-trivial to find out which "python.exe" in the task manager is running which Python script. Anyway, each app would have a small launcher that bootstraps the actual Python script[1]. (Or, if there's some way to trick the task manager into displaying something besides "python.exe", that'd work too) In order to "fit" into the expectations of a typical Windows user, an application installer needs the ability to check for the presence of a particular version of the Python runtime, and then install it if it's not present, and do so without asking the user to do anything. With that much in place, a lot of the need for freezing Python apps goes away (definitely not all of it, but a lot of it). Here's stuff that still needs to be considered though: 1) A way for apps to specify their compatibility with different versions of the runtime - "I work with pyrt2.3.3 through 2.4.1" 2) A way for apps to register their dependency on that runtime, so that if the user uninstalls it, there is at least an indication of what programs might break. 3) (this is a nice-to-have, but not 100% required) A way to download additional libraries once and use them for multiple programs, e.g. the installer could see if ctypes is present, and if not, go get it. Later programs would take advantage of it already being installed. Like I said, not a 100% requirement, and some of the ongoing work with code CPAN-like code repositories would shoehorn into this. 4) The security implications, e.g. an innocent app installs pywin32 and enables Python client-side scripting in Internet Explorer, and later this is used as a big door for malicious users to use. Most of these tasks don't require a lot of work; indeed this has been on my "one of these days" lists for awhile. The main reasons it hasn't been done yet: 1) The code freezing tools like py2exe are *very* good and convenient (especially since loading code from zip archives was added - even if you include all of Python, it's only a few files on disk, and now they're all in the same directory) 2) Storage space and download speeds continue to increase at a rate much faster than the rate at which the size of Python is growing - a download of a few MB isn't too bad these days, who cares if your app takes 4MB on disk, and so what if it doesn't fit on a floppy, for example. 3) As soon as you get started on such a project, almost immediately you will be overwhelmed with a desire to create a CPAN-like system while you're at it, at which point your project's status will shift from "making good progress!" to "in quagmire, developers MIA". :) -Dave [1] I built a hacky one of these launcher exe's for one project, and it was fairly reusable for apps in the same vein. Basically, the exe would would look to see what its name was, and then load a Python module of the same name. So if you have MyGame.py, you'd just take the generic launcher.exe, copy it to the same directory as your .py files, and rename it to MyGame.exe. On launch, it'd load the interpreter, load MyGame.py, and pass it the command-line args. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.4 killing commercial Windows Python development ?
Thomas Heller wrote: Dave Brueck <[EMAIL PROTECTED]> writes: Terry Reedy wrote: If there is something about the default install of Python on Windows that makes it less desireable or less easy than other platforms, then maybe that can be fixed. To make installation easier, maybe someone could write a small .exe that could be frozen with scripts or run with installers and that would detect the presence/absence of the needed Python version and offer an auto download and install if needed. I mostly agree with the sentiment, but it does break down a little in practice. At least currently it does - like you said, this is fixable, but nobody has signed up to fix it yet. The main thing that's needed is a zero-input Python distribution - a Python runtime, if you will - that (1) gets installed to a "good" place (2) does so without asking the user to do anything, (3) can coexist with different versions of the runtime, and (4) is easily detectable by applications wanting to use it. The effbot.exe platform (or how it's called) ? Yep - something along those lines, plus some docs for app developers. I don't know if it installs all the stdlib, or just what effbot apps need, but I assume the former. One other minor component is a small launcher executable, because on Windows it's non-trivial to find out which "python.exe" in the task manager is running which Python script. Anyway, each app would have a small launcher that bootstraps the actual Python script[1]. (Or, if there's some way to trick the task manager into displaying something besides "python.exe", that'd work too) exemaker? Something similar to that, yes. You'd need an option to generate a console launcher or a Windows app launcher (maybe his latest version already has this?), and a way to figure out which version of the runtime to use (rather than specify the path ahead of time, you'd specify version requirements, and then at runtime use registry entries to figure out where that runtime is), but the general idea is the same. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Dr. Dobb's Python-URL! - weekly Python news and links (Apr 11)
Roel Schroeven wrote: Simon Brunning wrote: ... Not that it really matters, but does anybody know why the weekly Python news always arrives twice? Does it only happen to me, or does it happen to others too? It's not that it irritates me or anything, I'm just being curious. I just thought it was for emphasis. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: a more precise re for email addys
Does it really need to be a regular expression? Why not just write a
short function that breaks apart the input and validates each part?
def IsEmail(addr):
'Returns True if addr appears to be a valid email address'
# we don't allow stuff like [EMAIL PROTECTED]@biff.com
if addr.count('@') != 1:
return False
name, host = addr.split('@')
# verify the hostname (is an IP or has a valid TLD, etc.)
hostParts = host.split('.')
...
That way you'd have a nice, readable chunk of code that you could tweak
as needed (for example, maybe you'll find that the RFC is too liberal
so you'll end up needing to add additional rules to exclude "bad"
addresses).
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to catch these kind of bugs in Python?
asincero wrote: > Is there anyway to catch the following type of bug in Python code: > > message = 'This is a message' > if some_obscure_condition: >nessage = 'Some obscure condition occured.' > print message > > > In the above example, message should be set to 'Some obscure condition > occured.' if some_obscure_condition is True. But due to a lack of > sleep, and possibly even being drunk, the programmer has mistyped > message. These types of bugs would easily be caught in languages that > have a specific keyword or syntax for declaring variables before use. There are tools that help (e.g. pychecker), but there are a few things to consider: 1) If the programmer is sleepy/drunk, you're going to have other bugs too (logical errors, not handling all cases, etc.) 2) Other languages would catch *some* types of these bugs, but would still miss some of them (I can see a sleepy programmer also using the wrong variable instead of just mistyping the right one). So while a tool might assist, it's worth your while to also consider some strategies for tackling the above two problems and, in the process, the sleepy-programmer-mistype bugs will get caught as well. Some type of testing is probably the best answer - be it reusable unit tests or, at the very least, some interactive testing of code snippets (which Python makes really easy to do). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: plug-ins
Eric Nieuwland wrote:
> The app I'm working on keeps getting new transforms and I'm tired of
> adding them by hand. So here it goes:
> Can anyone provide me with clues/examples/references on how to create a
> plug-in framework?
The biggest task is defining the interface between your app and the plugins -
you need to decide at what points plugins can interact with your application,
what data from your application they'll need to have access to, etc. You'll
also
need to decide how your app becomes aware of new plugins - perhaps the data
specifies it, or a config file. IOW, most of the initial work is just deciding
on a convention you want to follow. Here is a simple example:
"Plugins are Python modules that are placed in the /plugins directory. A
plugin may optionally have an Init() function that is called when the plugin is
first loaded. A plugin is required to have a Process() function. This function
takes an input buffer and return a transformed output buffer."
Here's some quickie code to load a plugin (untested):
import new
def LoadPlugin(name):
'Loads a plugin and returns a plugin module object'
# TODO: Decide on and implement an error-handling convention
# Create a new module
m = new.module(name)
source = file('plugins/%s' % name).read()
exec source in m.__dict__
assert hasattr(m, 'Process'), 'Plugin %s has no Process function' % name
# Optionally run its Init function
if hasattr(m, 'Init'):
m.Init()
return m
If you need plugins at different times, you could have a plugin cache:
pluginCache = {} # name -> plugin module
def GetPlugin(name):
'Returns the named plugin from the cache, creating it if needed'
try:
return pluginCache[name]
except KeyError:
plugin = pluginCache[name] = LoadPlugin(name)
return plugin
You might use a plugin like this:
def Transform(name, data):
'Applies the named transform to the data and returns the result'
plugin = GetPlugin(name)
return plugin.Process(data)
In practice, your plugins will probably have a much richer interface. You may
even decide to have an "application" object of some sort that you can pass to
the plugins to provide services to them (logging facility, supporting routines
that many plugins need, etc.). Instead of a module-based approach, you may
instead decide that all plugins inherit from a plugin base class.
-Dave
--
http://mail.python.org/mailman/listinfo/python-list
Re: Structuring larger applications - ideas
[EMAIL PROTECTED] wrote: > I'm busy with a large application and feel it would eas my work if I > can specify dependencies on the granularity of packages, rather than > modules and classes. Eg: > > - By convention I do the one class per file thing. SO in python this > means one class per module - naming classes after their modules. (this > helps with version control, and several other little irritations, for > example) > - I'd like to specify once for a _package_ that it depends upon another > _package_. > - The following should then also be true for a module A in package X > (which depends upon package Y): > 1) X should be available in the namespaces of module A (in fact for > all modules in X) > 2) X.B should refer to X.B.B (I name classes after their modules). > > (2) Can be done easily by, eg putting the following in X.__init__.py: >from B import B > > What's the feeling in this group about the idea & plans to get closer > to accimplishing it? My two cents: one of the advantages of a language like Python is that the project reaches an unmanageable size more slowly than in many other languages. As such, be wary of "importing" conventions you have used in other languages. Many will apply, some will not. IMO, the one-class-per-module is a convention that in and of itself doesn't add much value (it drove me crazy in Java), and only accelerates your project towards being tough to manage - you don't want to artificially introduce complexity. If two classes are inherently tied together and closely related, is there an advantage to forcing them to live in separate files? The costs of doing so include manually tracking a relationship that would normally be implied and obvious, keeping file versions in synch (seems like this makes version control harder), and some extra code to bring them into each others' namespaces. Much like a class encapsulates many details and pieces of functionality into a larger, more powerful building block, a module often does the same thing on a larger scale (and a package on an even larger one). So if it makes sense in a few cases to restrict yourself to a single class per module, go for it, but I don't see the value in adopting it as a convention (maybe I'm missing something?). As for inter-package dependencies, what problem are you trying to solve? If you are trying to track dependencies for documentation purposes, you could just write a script to walk the source tree and detect the dependencies - it is information contained in the code itself, so rather than tracking it manually, you could just "ask" it. If you are trying to organize the overall structure of the project, then the consumer of that info is likely to be a person, so a piece of paper, a whiteboard, or a big fat docstring might be a better suited home for it. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Windows distribution suggestions?
>> One tip: make sure your three top-level scripts are as empty as
>> possible: just import some other module and call a function. All
>> your active code is then in some library.zip shared between the
>> three, and you need never change alice.exe, bob.exe, and carol.exe
>
> I think I understand what you're saying and it sounds like a very
> good idea. Thanks.
To elaborate, this approach has worked well for me when I've needed to
release multiple .exe's together:
First make a "dispatching" module like this (call it run.py)
if __name__ == '__main__':
app = sys.argv[0].lower()
if app.find('alice') != -1:
import alice
alice.main()
elif app.find('bob') != -1:
import bob
bob.main()
etc...
Next, use py2exe to create an executable for run.py, and then copy
run.exe to alice.exe, bob.exe, and carol.exe (the resulting .exe's will
be very small). The reasoning behind this is that this way only one
copy of all the DLLs, Python std library, and so forth need to be
included in your release.
And to second everyone else's suggestions: NSIS and InnoSetup are
great, free, and powerful. They have all the features you need to
automatically copy your files to the "proper" location on disk, create
desktop icons, and create an uninstaller entry - you'll just write a
small installation script telling what features you'd like to take
advantage of, but they figure out all the details and combine your
application files into a standalone installer (your app.exe).
It'll take a little work to get this all straightened out the first
time, but after that you can have a single build/release script so it's
painless from then on.
And for updates: the least problematic approach is to just release the
whole thing each time. IIRC, InnoSetup (and probably NSIS) include a
feature where you can detect if your app is running when the user tries
to install an update and you can prompt the user to shut down first.
You'd need to use ctypes or pywin32 to make your app detectable that
way, but it's only 1 or 2 lines of code. But, lemme know if you want to
go the route of updating only portions of the app - I've gone that
route too but it takes more work and has its own set of problems; I
might be able to save you a few headaches.
-Dave
--
http://mail.python.org/mailman/listinfo/python-list
Re: Reading image dimensions with PIL
Will McGugan wrote: > I'm writing an app that downloads images. It rejects images that are > under a certain size - whithout downloading them completely. I've > implemented this using PIL, by downloading the first K and trying to > create a PIL image with it. PIL raises an exception because the file is > incomplete, but the image object is initialised with the image > dimensions, which is what I need. It actualy works well enough, but I'm > concerened about side-effects - since it seems an unconventional way of > working with PIL. Can anyone see any problems with doing this? Or a > better method? If you're tossing images that are too _small_, is there any benefit to not downloading the whole image, checking it, and then throwing it away? Checking just the first 1K probably won't save you too much time unless you're over a modem. Are you using a byte-range HTTP request to pull down the images or just a normal GET (via e.g. urllib)? If you're not using a byte-range request, then all of the data is already on its way so maybe you could go ahead and get it all. But hey, if your current approach works... :) It _is_ a bit unconventional, so to reduce the risk you could test it on a decent mix of image types (normal JPEG, progressive JPEG, normal & progressive GIF, png, etc.) - just to make sure PIL is able to handle partial data for all different types you might encounter. Also, if PIL can't handle the partial data, can you reliably detect that scenario? If so, you could detect that case and use the download-it-all-and-check approach as a failsafe. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Python forum
Grant Edwards wrote: > On 2005-05-17, Jonas Melian <[EMAIL PROTECTED]> wrote: >>I'm going to say a suggestion, why don't you create a forum >>like the one of Ruby (http://www.rubyforums.com/)? for the >>novices this is a great help, better than a mail list [snip] >>But I think that a forum is great for the learning. The test >>is in gentoo's forums. They are a lot of experienced people >>answering questions without any problem and it goes very well > > > Except for the torture of using a web forum's UI. > > [In case you can't tell, I hate web forums. I've never seen a > single one with a suable UI.] Amen! Generally they are an abomination. To make matters worse, many forums that become popular are saddled with so many advertisements that moving from one message to another becomes a... grueling... lesson... in... patience. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: speeding up Python
Luis P. Mendes wrote: > I have a 1000 line python script that takes many hours to finish. It is > running with six inside 'for' loops. Hi Luis, Before going too much into optimizing your current code, you might want to take a step back and see if another approach to your problem might work instead. Python is great for this sort of algorithmic experimentation. So: what are you doing? In particular, why SIX nested loops? It's not uncommon for a different algorithm to yield performance improvements that are orders of magnitude greater than what just optimizing the code could achieve. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python suitable for a huge, enterprise size app?
john67 wrote: > The company I work for is about to embark on developing a commercial > application that will cost us tens-of-millions to develop. [snip] > Right now it looks like Java is the language of choice that the app > will be developed in. However, I have been looking and reading a lot > about Python recently and it seems to me that Python could handle it. > The big attraction to me is the developer productivity. It seems that > if Python can handle it, then we could gain a huge savings by using > Python instead of Java from a productivity standpoint alone. > > So, given the very general requirements in the first paragraph, do you > think that Python could handle it? Yes. Python has good database support, it works well on a wide range of platforms, and it's great at tying together different processes, machines, etc. - for example, it's fairly easy to get Python to access C code, dynamic libraries, system APIs, and external programs. It's easier to test than Java code as well, and the overal cost of change is lower. The concise nature of the language means that what would otherwise be a "huge" enterprise app in Java might just be a "large" enterprise app in Python - the project will not reach that unwieldy size as quickly, if ever. The Google archives of this group have this topic covered in various ways many times over - IMO Python is great for smallish apps, but its advantage over e.g. Java actually *increases* with the size of the app. If you go the Python route, two of the main obstacles will be: 1) mindshare - getting people on board, getting them to overcome biases one way or another, convincing them to really take the time to come up to speed on Python. 2) reducing the complexity of what you think you need to build. I don't know how to better describe this, but in Java, for example, I'd have whole bunches custom classes to do various tasks but the Python equivalent collapsed into a single module. Anyway, I predict that it'll take you some time to convince yourself that you simply won't need to build all of the same components as you otherwise would, or that they'll often be vastly simpler. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python suitable for a huge, enterprise size app?
EHP wrote: >>Python has good database support, it works well on a wide range of >>platforms, and it's great at tying together different processes, machines, >>etc. - for example, it's fairly easy to get Python to access C code, >>dynamic libraries, system APIs, and external programs. It's easier to test >>than Java code as well, and the overal cost of change is lower. > > > Python has good DB support - but only simple connectors (like JDBC). > ORM in Java (like Hibernate) are much better than ORM in Python (probably the > best is sqlobject). How you can write huge OO app without ORM ? I don't doubt that Hibernate is more mature, but having not used it, I can't really say how much better/worse it is than, say, sqlobject for Python. But, as you pointed out, there *are* ORM libraries for Python. FWIW, the only times I've been involved in database-centric projects with a cost in the "tens of millions", the database team steered clear of any sort of any automatic ORM layer altogether. Depending on your application, an ORM can fall into the same category as, say, EJB - a nifty idea that looks great on paper but can cause more problems than it solves. Just because the app itself is very OO, it doesn't always follow that the database level needs to be - there are lots and lots of problems for which normal RDBMS tables & joins are a pretty darn good fit. (I'm not trying to discount ORM layers or even object databases, just disagreeing with the notion that huge OO apps automatically require ORM) Have fun, -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python suitable for a huge, enterprise size app?
George Sakkis wrote: > Personally, although I find nothing comes close to the clarity and > flexibility that python offers, in this case I would go with java, if > for nothing else, to be on the safe side. Think about it: if the > project fails (and that's quite likely for huge projects, not dark > humour) and you've done it in python, everyone will blame python and > you who chose it; java OTOH belongs in the realm of "standard business > practices", so in a way "it's ok" to screw up, you did what everybody > else does. Yet another failed java enterprise project, nothing to see > here. lol - I got a kick out of this post and I agree with it: *if* you're pretty sure the project will ultimately fail, and you're planning for the best way to point fingers and avoid the ax when it happens, then by all means, Java is the right way to go. Java is the new "nobody ever got fired for buying IBM". -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python suitable for a huge, enterprise size app?
Paul Rubin wrote: > "Kay Schluehr" <[EMAIL PROTECTED]> writes: >>To answer Your initial question: there is probably no technical reason >>against Python as a language or the CPython runtime. Both are very >>stable and mature. > > > I'd like to agree with this but I just can't. Python is a great > language for prototyping and trying stuff out. It's fast and pleasant > to write. But you're right when you say that constant testing and > integration is absolutely necessary to develop anything substantial in > Python. [snip] > I'm in the middle of prototyping something in Python and it's going > pretty well despite occasional snags like those mentioned above. But > at the end of the day I don't think what we get will be solid enough > for use in production. We're planning to use the Python prototype to > get the feature set and protocols figured out, then reimplement in Java. > The OP might consider that approach. You bring up some good points and I imagine the bugs you've encountered are pretty frustrating, but I'm surprised at how different our experience has been. I know my comments about Python probably come across a "total fan boy" - but for us Python has been extremely positive. We currently use Python, in production, in the following ways: - custom HTTP web servers - custom HTTP proxies - behind-the-scenes log processors and other operational tools - several Zope-based applications, both internally and externally facing - both the client and server sides of a distributed work cluster - the client side of a media encoding farm (server is a Zope app) - a downloadable Windows client that our customer's customers use, that includes a Windows COM server (an ActiveX control we implemented using ctypes). Most of these have been in production for a year or more, with the oldest being about 4 years (not a long time, but certainly long enough to be considered stable). The toughest pieces initially were the Zope apps and the Windows COM stuff - Zope because of documentation and COM because, well, it's COM. During that time, the biggest problem we had with Python and its standard libraries was one issue with socket.recv and memory allocation when doing lots of receives. That lost us a few days of hunting for a memory leak, but we were able to resolve it by looking at the C source. Another problem was the need to call unlink (or something like that) on XML minidom elements in some circumstances. Apart from that, we haven't encountered much of anything in terms of bugs in the language or the standard library in *years*. In fact, I don't recall encountering a bug in the language itself. We've generally trailed the standard Python releases by about a year (although this week I moved the encoding farm to Python 2.4.1, which is half a year early). We don't use Tkinter. We don't use much outside the standard library: we make heavy use of ctypes, psycopg (Postgres), and lately CherryPy, but that's about it. So for us, the number of bugs in Python + stdlib has been really low. One thing from your experience that did resonate with me is that, except for ftplib and occasionally urllib (for basic, one-shot GETs), we don't use any of the standard library's "protocol" modules - partly because we had to implement our own HTTP libraries for performance and scalability reasons anyway, and partly because we had trouble figuring out e.g. all the ins and outs of urllib/urllib2/httplib. Overall it's been such a positive experience for us that nobody in the company - from grunt testers up to the CTO - has any reservations about using Python in production anymore (even though initially they all did). All of the developers have previous experience with using Java in production systems, and none seriously consider it for new projects at all. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python suitable for a huge, enterprise size app?
Paul Rubin wrote: > Dave Brueck <[EMAIL PROTECTED]> writes: > >>One thing from your experience that did resonate with me is that, >>except for ftplib and occasionally urllib (for basic, one-shot GETs), >>we don't use any of the standard library's "protocol" modules - partly >>because we had to implement our own HTTP libraries for performance and >>scalability reasons anyway, and partly because we had trouble figuring >>out e.g. all the ins and outs of urllib/urllib2/httplib. > > > What do you use for HTTPS? Hi Paul, m2crypto (plus some patches to make asynchronous SSL do what we needed). > And did you use the Cookie module in your > HTTP servers? You may have had problems without even being aware of > them (until recently if you used Cookie with its default settings, any > attacker could completely take over your server by sending you > carefully concoted cookies). Are you referring to the use of pickle for cookie serialization? In any case, we didn't use Cookie.py from the stdlib (on the servers, nearly everything related to URLs & HTTP was custom-built, with the exception of urlparse, for the aforemenioned reasons). -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python suitable for a huge, enterprise size app?
Paul Rubin wrote: > Dave Brueck <[EMAIL PROTECTED]> writes: > >>>What do you use for HTTPS? >> >>m2crypto (plus some patches to make asynchronous SSL do what we needed). > > > That seems to be a nice piece of code, but it's still at version 0.13; Version numbers are fairly relative, though. In another project we're using some proprietary, closed source libraries (unrelated to crypto) that are version 3 and they seem buggier and less stable than m2crypto. And don't get me started on Microsoft products (we've been using DirectShow *9* in some stuff, and due to bugs in DirectShow we were completely and utterly screwed despite what the documentation said; things just didn't work as they should, and Microsoft has confirmed that it's a bug that will be fixed in some future release - so we had to backtrack, ripping out code that should work just fine, and take another stab at getting DirectShow to cooperate). Version NINE (supposedly). > if something goes wrong, are you sure you want to explain that you > were using beta-test software to protect your customers' production > financial transactions? lol - what? We're not doing any financial transactions with it (IOW - we evaluated m2crypto for what we needed to do, and it's a good fit - that we didn't evaluate it in terms of what we don't need it to do doesn't bother me). Having said that - I think we probably *would* use it for production financial transactions - but that's more a matter of closed vs. open source than Python vs not. Besides, do you really think that, if something went wrong, you'd in the end have some meeting where you explain to your customer that you were using beta software? Of course not - it just doesn't work that way. Either they won't care and will drop you because of the problem (regardless of the source) or they want some broad details. > There's also been some traffic on the > python-crypto list about Zope encountering memory leaks with it. Ok... so? I mean, if there's a memory leak, and it's hurting us, we have options: we can go look in the source code, we can make Zope reboot itself often, we can hire somebody to fix it, we can see if the author wants to give us a support contract, etc. Memory leaks aren't exactly unique to Python - according to bugs.sun.com, there are currently 382 *open* bugs related to memory leaks in the JDK alone. If you're using Java in your "huge, enterprise size app" and get bit by one of those bugs, and if you're not a big enough company to get some ridiculously-priced support contract from Sun, what are your options? Again, this seems more like an open-vs-closed source issue, but to me it's another reason why I'd feel uncomfortable using Java in mission critical work. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: working with pointers
Michael wrote: > sorry, I'm used to working in c++ :-p > > if i do > a=2 > b=a > b=0 > then a is still 2!? > > so when do = mean a reference to the same object Always. > and when does it mean make a copy of the object?? Never. -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Socket Speed
[EMAIL PROTECTED] wrote: > Hi All > > I'm busy writing a python p2p program and would like some advice. > > I'm pushing about 300k/s and would like to know if there are any python > tricks I could pull to speed things up. I'm thinking about unrolling > some of the loops and recuding calls to my custom socket class and > just calling recv and send directly. What is the overhead for using > try and exception blocks ? any recomendations for recv/send sizes ? > On the server side I'm using SocketServer. I was planning to use > twisted > but I don't have the time just get to get into it, SocketServer does > the job. > > I'll post some code soon. Yeah - do post some code as 300k/s is probably *way* below the potential. In one project I'm working on, I do simple file transfers over a single socket with Python on both ends and I routinely get above 500Mbps (often into the high 700's/low 800's - pretty close to what I could hope for for gig over copper with TCP/IP) and the CPU utilization is way low - I assume disk on one end or the other of the connection is a bottleneck now. I use 64k send/receive buffers - above that it doesn't seem to help much for what I'm doing. In any case, before you make any local optimizations (loop unrolling, etc.), I'd do some more investigation to find the problem because something major is broken, and fixing it will probably do way more to improve performance than any optimization "tricks". -Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing backwards compatible code
Steven D'Aprano wrote: > I came across an interesting (as in the Chinese curse) problem today. I > had to modify a piece of code using generator expressions written with > Python 2.4 in mind to run under version 2.3, but I wanted the code to > continue to use the generator expression if possible. [snip] > What techniques do others use? About the only reliable one I know of is to not use syntax from the newer version, period, if the code needs to run on old versions. Your example was pretty straightforward, but unless the use case is always that simple it's too easy to have two separate versions of the code to maintain, and there's no way to enforce that changes/fixes to one will be duplicated in the other. So... if you have to take the time to make one that works with the older version, you might as well just use it exclusively. -Dave -- http://mail.python.org/mailman/listinfo/python-list
