Bugs item #978833, was opened at 2004-06-24 11:57 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=978833&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 8 Submitted By: kxroberto (kxroberto) Assigned to: Nobody/Anonymous (nobody) Summary: SSL-ed sockets don't close correct? Initial Comment: When testing FTP over SSL I have strong doubt, that ssl-ed sockets are not closed correctly. (This doesn't show with https because nobody takes care about whats going on "after the party".) See the following : --- I need to run FTP over SSL from windows (not shitty sftp via ssh etc!) as explained on http://www.ford-hutchinson.com/~fh-1-pfh/ftps-ext.html (good variant 3: FTP_TLS ) I tried to learn from M2Crypto's ftpslib.py (uses OpenSSL - not Pythons SSL) and made a wrapper for ftplib.FTP using Pythons SSL. I wrap the cmd socket like: self.voidcmd('AUTH TLS') ssl = socket.ssl(self.sock, self.key_file, self.cert_file) import httplib self.sock = httplib.FakeSocket(self.sock, ssl) self.file = self.sock.makefile('rb') Everything works ok, if I don't SSL the data port connection, but only the If I SSL the data port connection too, it almosts work, but ... self.voidcmd('PBSZ 0') self.voidcmd('PROT P') wrap the data connection with SSL: ssl = socket.ssl(conn, self.key_file, self.cert_file) import httplib conn = httplib.FakeSocket(conn, ssl) than in retrbinary it hangs endless in the last 'return self.voidresp()'. all data of the retrieved file is already correctly in my basket! The ftp server just won't send the final '226 Transfer complete.' on the cmd socket. Why? def retrbinary(self, cmd, callback, blocksize=8192, rest=None): self.voidcmd('TYPE I') conn = self.transfercmd(cmd, rest) fp = conn.makefile('rb') while 1: #data = conn.recv(blocksize) data = fp.read() #blocksize) if not data: break callback(data) fp.close() conn.close() return self.voidresp() what could be reason? The server is a ProFTPD 1.2.9 Server. I debugged, that the underlying (Shared)socket of the conn object is really closed. (If I simly omit the self.voidresp(), I have one file in the box, but subsequent ftp communication on that connection is not anymore correct.) ---------------------------------------------------------------------- >Comment By: Martin v. Löwis (loewis) Date: 2006-07-03 14:03 Message: Logged In: YES user_id=21627 Can you please try the attached patch? It makes sure _socketobject.close really closes the socket, rather than relying on reference counting to close it. ---------------------------------------------------------------------- Comment By: kxroberto (kxroberto) Date: 2006-05-11 14:05 Message: Logged In: YES user_id=972995 Testing it with Python2.5a2, the problem is still there. Without the .shutdown(2) (or .shutdown(1)) patch to the httplib.SharedSocket (base for FakeSocket), the ftps example freezes on the cmd channel, because the SSL'ed data channel doesn't close/terminate --> FTPS server doesn't respond on the cmd channel. The ftps example is most specific to show the bug. Yet you can also easily blow up a HTTPS-server with this decent test code who only opens (bigger!) files and closes without reading everything: Python 2.5a2 (r25a2:45740, May 11 2006, 11:25:30) [GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Robert's Interactive Python - TAB=complete import sys,os,re,string,time,glob,thread,pdb >>> import urllib >>> l=[] >>> for i in range(10): ... f=urllib.urlopen('https://srv/big-Python-2.5a2.tgz') ... f.close() ... l.append(f) ... >>> => in the (apache) servers ssl_engine_log you can see that connections remain open (until apache times out after 2 minutes) and lots of extra apache daemons are started! => f.close() doesn't really close the connection (until it is __del__'ed ) Trying around I found that the original undeleted f.fp._ssl is most probably the cause and holds the real socket open. a f.fp._sock.close() doesn't close also - but only when del f.fp._ssl is done. (only a f.fp._sock._sock.close() would force the close). The original fp is held in closures of .readline(s)/__iter__/next... -- I now tried an alternative patch (instead of the shutdown(2)-patch), which also so far seems to cure everything . Maybe thats the right solution for the bug: --- httplib.py.orig 2006-05-11 11:25:32.000000000 +0200 +++ httplib.py 2006-05-11 13:45:07.000000000 +0200 @@ -970,6 +970,7 @@ self._shared.decref() self._closed = 1 self._shared = None + self._ssl = None class SSLFile(SharedSocketClient): """File-like object wrapping an SSL socket.""" @@ -1085,6 +1086,7 @@ def close(self): SharedSocketClient.close(self) self._sock = self.__class__._closedsocket() + self._ssl = None def makefile(self, mode, bufsize=None): if mode != 'r' and mode != 'rb': -------------- In another application with SSL'ed SMTP connections there arose similar problems. I also worked around the problem in smtplib.py so far in a similar style: def close(self): self.realsock.shutdown(2) self.realsock.close() --- Yet, the right patch maybe (not tested extensively so far): --- Lib-orig/smtplib.py 2006-05-03 13:10:40.000000000 +0200 +++ Lib/smtplib.py 2006-05-11 13:50:12.000000000 +0200 @@ -142,6 +142,7 @@ sendall = send def close(self): + self.sslobj = None self.realsock.close() class SSLFakeFile: @@ -161,7 +162,7 @@ return str def close(self): - pass + self.sslobj = None def quoteaddr(addr): """Quote a subset of the email addresses defined by RFC 821. ------------------ -robert ---------------------------------------------------------------------- Comment By: kxroberto (kxroberto) Date: 2005-09-24 21:45 Message: Logged In: YES user_id=972995 Now I managed to solve the problem for me with attached patch of httplib.py: a explicit shutdown ( 2 or 1 ) of the (faked) ssl'ed socket solves it. I still guess the ssl'ed socket (ssl dll) should do that auto on sock.close() Thus I leave it as a bug HERE. Right? [ I also have the hope, that this also solves the ssl-eof-errors with https (of some of my users; will see this in future) *** \usr\src\py24old/httplib.py Sat Sep 24 21:35:28 2005 --- httplib.py Sat Sep 24 21:37:48 2005 *************** class SharedSocket: *** 899,904 **** --- 899,905 ---- self._refcnt -= 1 assert self._refcnt >= 0 if self._refcnt == 0: + self.sock.shutdown(2) self.sock.close() def __del__(self): ---------------------------------------------------------------------- Comment By: kxroberto (kxroberto) Date: 2005-09-24 21:06 Message: Logged In: YES user_id=972995 I retried that again with py2.4.1. The problem/bug is still there. In attachment I supplied a full FTPS client test_ftpslib.py with simple test function - ready to run into the problem: At the end of ftp.retrlines 'return self.voidresp()' freezes : waiting eternally for response bytes at the command port. the same at the end of .storelines after the data is transfered on the data port. My preliminary guess is still, that python's ssl has a severe (EOF?) problem closing a socket/connection correctly. obviously this problem doesn't show up with https because everything is done on one connection - no dependency on a correct socket/connect close signal. (from other https communication with some proxies in between my users get ssl-eof-error errors also. I still can't solve that bug too. this shows python's ssl has a severe EOF bug with complex https also - or cannot handle certain situations correct.) I learned the FTPS/TLS client from M2crypto's ftpslib. the only difference in (transposed) logic, that I can see is, that M2crpyto uses an additional line "conn.set_session(self.sock.get_session())" during setup of the data port ssl connection. I don't know what it is, pythons ssl doesn't have sucht "session"-functions, but I think it has no severe meaning.? Robert ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-12-22 06:14 Message: Logged In: YES user_id=357491 Since I believe this was fixed with the patch Tino mentions and Roberto has not replied, I am closing as fixed. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-08-17 01:18 Message: Logged In: YES user_id=357491 Is this still a problem for you, Roberto, with Python 2.4a2? ---------------------------------------------------------------------- Comment By: Tino Lange (tinolange) Date: 2004-07-11 00:30 Message: Logged In: YES user_id=212920 Hi Roberto! Today a patch for _ssl.c was checked in (see #945642) that might solve your problem, too. Could you please grab the *next* alpha (this will be Python 2.4 Alpha 2) and test and report afterwards if it is solved? Thanks for your help! Tino ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=978833&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
