I'm reading files from an FTP server at the U.S. Securities and
Exchange Commission. This code has been running successfully for
years. Recently, they imposed a consistent connection delay
of 20 seconds at FTP connection, presumably because they're having
some denial of service attack. Python 2.7 urllib2 doesn't
seem to use the timeout specified. After 20 seconds, it
gives up and times out.
Here's the traceback:
Internal error in EDGAR update: <urlopen error ftp error: [Errno 110]
Connection timed out>
....
File "./edgar/edgarnetutil.py", line 53, in urlopen
return(urllib2.urlopen(url,timeout=timeout))
File "/opt/python27/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/opt/python27/lib/python2.7/urllib2.py", line 394, in open
response = self._open(req, data)
File "/opt/python27/lib/python2.7/urllib2.py", line 412, in _open
'_open', req)
File "/opt/python27/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/opt/python27/lib/python2.7/urllib2.py", line 1379, in ftp_open
fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout)
File "/opt/python27/lib/python2.7/urllib2.py", line 1400, in connect_ftp
fw = ftpwrapper(user, passwd, host, port, dirs, timeout)
File "/opt/python27/lib/python2.7/urllib.py", line 860, in __init__
self.init()
File "/opt/python27/lib/python2.7/urllib.py", line 866, in init
self.ftp.connect(self.host, self.port, self.timeout)
File "/opt/python27/lib/python2.7/ftplib.py", line 132, in connect
self.sock = socket.create_connection((self.host, self.port),
self.timeout)
File "/opt/python27/lib/python2.7/socket.py", line 571, in
create_connection
raise err
URLError: <urlopen error ftp error: [Errno 110] Connection timed out>
Periodic update completed in 21.1 seconds.
----------------------------------------------
Here's the relevant code:
TIMEOUTSECS = 60 ## give up waiting for server after 60 seconds
...
def urlopen(url,timeout=TIMEOUTSECS) :
if url.endswith(".gz") : # gzipped file, must decompress first
nd = urllib2.urlopen(url,timeout=timeout) # get connection
... # (NOT .gz FILE, DOESN'T TAKE THIS PATH)
else :
return(urllib2.urlopen(url,timeout=timeout)) # (OPEN FAILS)
TIMEOUTSECS used to be 20 seconds, and I increased it to 60. It didn't
help.
This isn't an OS problem. The above traceback was on a Linux system.
On Windows 7, it fails with
"URLError: <urlopen error ftp error: [Errno 10060] A connection attempt
failed because the connected party did not properly respond after a
period of time, or established connection failed because connected host
has failed to respond>"
But in both cases, the command line FTP client will work, after a
consistent 20 second delay before the login prompt. So the
Python timeout parameter isn't working.
John Nagle
--
http://mail.python.org/mailman/listinfo/python-list
