Bugs item #1161031, was opened at 2005-03-10 19:34 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1161031&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Tony Meyer (anadelonbrin) Assigned to: A.M. Kuchling (akuchling) Summary: Neverending warnings from asyncore Initial Comment: Changes in asyncore from 2.3 to 2.4 mean that asyncore.poll() now passes all the sockets in the map to select.select() to be checked for errors, which is probably a good thing. If an error occurs, then handle_expt() is called, which by default logs the error. asyncore.dispatcher creates nonblocking sockets. When connect_ex() is called on a nonblocking socket, it will probably return EWOULDBLOCK (connecting takes time), which may mean the connection is successful, or may not (asyncore dispatcher keeps going assuming all is well). If the connection is not successful, and then asyncore.loop() is called, then select.select() will indicate that there is an error with the socket (can't connect) and the error will be logged. The trouble is that asyncore.loop then keeps going, and will log this error again. My not-that-fast system here gets about 10,000 logged messages per second with a single socket in the asyncore map. There are ways to avoid this: (1) if the socket is blocking when connect()ing (and then nonblocking afterwards) an error is raised if the connect fails. (2) Before calling asyncore.loop(), the caller can run through all the sockets, checking that they are ok. (3) handle_expt() can be overridden with a function that repairs or removes the socket from the map (etc) However, I'm not convinced that this is good behavior for asyncore to have, by default. On Windows, select.select() will only indicate an error when trying to connect (nonblocking) or if SO_OOBINLINE is disabled, but this may not be the case (i.e. errors can occur at other times) with other platforms, right? Unless the error is temporary, asyncore will by default start streaming (extremely fast) a lot of "warning: unhandled exception" (not very helpful an error message, either) messages. Even if the error only lasts a minute, that could easily result in 10,000 warnings being logged. Do any of the python developers agree that this is a flaw? I'm happy to work up a patch for whatever the desired solution is, if so. ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2005-06-09 11:41 Message: Logged In: YES user_id=11375 What change to asyncore caused the problem? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2005-06-02 22:02 Message: Logged In: YES user_id=31435 I agree the change in default behavior here was at least questionable, and spent more than a week of my own "dealing with consequences" of 2.4 asyncore changes in ZODB and Zope. Assigning to Andrew, since it looks like he made the change in question here. Andrew, why did this change? How can it be improved? I don't think Tony has mentioned it here, but when SpamBayes was first released with Python 2.4, it was a disaster because some users found their hard drives literally filled with gigabytes of mysterious "warning: unhandled exception" messages. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2005-06-02 21:38 Message: Logged In: YES user_id=552329 I am not at all unwilling (and this isn't a problem for me personally - the issue here is whether this is good for Python in general) to subclass. Deciding to subclass does *not* mean that you should have to replace all functions in the parent class. If you did, then there would be little point in subclassing at all. Sensible default behaviour should be provided as methods of classes. The current behaviour of the handle_expt() method is not sensible. It essentially forces the user to override that method, even if they have no interest in handling errors (e.g. and would normally just override handle_read and handle_write). This is *not* rare. You haven't seen any in years, because this was a change introduced in Python 2.4, which hasn't been released for even one year yet. I agree that the desired behaviour will be application specific. But what is the point of having default behaviour that will essentially crash the program/system running it? Having default behaviour be "pass" would be more useful. At the very least, this is a problem that many people (compared to the number that will use asyncore) will come across and should be reflected as such in the documentation. If you haven't replicated this problem on your system so that you understand it, please do. I am happy to provide a simple script to demonstrate, if necessary. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2005-05-31 15:34 Message: Logged In: YES user_id=341410 You seem to be unwilling to subclass asyncore.dispatcher to extend its functionality, and the only reason you have given as to why you are unwilling is "As much as possible a class should provide sensible methods, so that overriding is kept to a minimum." (I personally subclass dispatcher and its async_chat derivative qutie often) Now, in the case of the other standard socket server and client framework in the Python standard library, namely the SocketServer module and its derivatives, you will find extending the functionality of those classes is via subclassing and overriding methods as necessary. To me, when two 'competing' methods of generating socket servers and clients in the standard library offer the same method of extension of their base functionality, then perhaps that is what should be done. The fact that basically all of the standard library is subclassable (some C modules are exceptions to the rule, but should be fixed in Python 2.5), including types in the base language, further suggests to me that subclassing is the standard mechanism for extending the functionality of a class, regardless of its usefulness in its base state. In regards to the documentation, it seems to be that whenever an object has an error, the handle_expt() method is called (in spending two minutes reading the source). Whether or not those errors are rare, perhaps debatable (I've not seen any in years), but it seems to be application-specific as to what behavior the socket should have in the case of an error (you may want to close, I may want to report the error and reconnect, etc.). ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2005-05-31 03:42 Message: Logged In: YES user_id=552329 dispatcher is not at all unusable without subclassing. You can get data with recv() and send it with send() etc. It can be treated as a thin wrapper around socket objects. Yes, you will want to subclass it to get more useful behaviour than you can get from a basic socket. I don't see that this means that you should be required to override the handle_expt() function, though. As much as possible a class should provide sensible methods, so that overriding is kept to a minimum. At the very least, this is a documentation error, since the documentation states: """ handle_expt( ) Called when there is out of band (OOB) data for a socket connection. This will almost never happen, as OOB is tenuously supported and rarely used. """ "Almost never" is completely wrong. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2005-05-31 03:31 Message: Logged In: YES user_id=341410 I hate to point out the obvious, but dispatcher is wholly unusable without subclassing. How would you get data to/from a connection without replacing handle_read, handle_write? How do you handle the case when you want to connect to someone else or accept connections from someone else without overloading handle_connect or handle_accept? ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2005-05-31 03:15 Message: Logged In: YES user_id=552329 Yes this problem is easily solved by subclassing. However I firmly believe that it is a terrible default behaviour, and that it's likely to hit many asyncore users. A class shouldn't have to be subclassed to be usable (ignoring virtual classes and all that), and that really is the case here. The simplest solution would be to change the handler to not log the message. Or log the message once per socket or something. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2005-05-31 03:03 Message: Logged In: YES user_id=341410 Option 1 is not really an option in any case where a large number of connections are opened (so I don't believe should be the default). >From what I understand, certain methods are supposed to be overridden in a subclass if someone using a class wants different behavior. In this case, I believe you can perform either option 2 or 3 in your own code to avoid the thousands of logged lines; either by creating your own loop() function, or by creating a subclass of dispatcher and implementing a handle_expt() method that does as you desire. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1161031&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
