Hi Raphael,
I have done a bit of debugging to try to shed some light on why urllib
is failing.
First of all, the nautilus-dropbox package doesn't list a specific
version of python in it's dependencies. I run it on debian/sid and
ubuntu/quantal, on which the default python version is 2.7.3.
I normally run a combination of polipo and privoxy as my proxies, so to
rule out that it's a misconfiguration problem, I have specifically
installed tinyproxy and run my tests against that.
I have taken the important parts of the dropbox program that performs
the download and placed it in a separate script with connection
debugging turned on. The script specifically compares urllib and
urllib2, but I also ran the equivalent curl command that connects to
the dropbox server.
The output.log clearly shows that there is nothing wrong with the proxy
configuration, because curl can connect to the server and returns the
correct headers. The python script shows that urllib fails, while
urllib2 succeeds. Specifically, urllib results in the key error because
the proxy returns "HTTP/1.0 500 Unable to connect".
I haven't filed a bug report with python as yet, but this might be a
case of "it's not a bug, it's a feature" in urllib. I have done a few
internet searches and found that this is not an uncommon problem, and
people talk of workarounds and backports. Basically, urllib doesn't
support SSL connections through a proxy. See
http://bytes.com/topic/python/answers/20602-urllib-2-https-blues-try-pytunnel-python-tunnelling.
Best,
Carlos
On 12/17/2012 06:25 PM, Raphael Hertzog wrote:
Control: forwarded -1 [email protected]
Hello,
On Mon, 17 Dec 2012, Carlos Maddela wrote:
Dear Maintainer,
When https_proxy is set, the download of the Dropbox daemon fails
with a KeyError exception, as the returned headers do not contain a
'Content-Length'. I have attached the output log.
Did you file this bug against python too? Because if urllib strips the
field when urllib2 doesn't, it's a clear bug of urllib.
Are you sure it's not a misconfiguration or misfeature of your https proxy
that would drop the field? (And maybe urllib2 creating it if it's missing?).
The dropbox servers do return the Content-Length header.
I had a go at patching the python script to fix this. It seems that
using urllib2.urlopen() instead of urllib.urlopen() is a little more
promising, but I also had to disable non-blocking I/O. It works alright
for me though. I have attached this patch, along with some others to
get rid of compilation warnings.
That's a lot of changes for this bugfix... I'm ccing the upstream
authors at Dropbox. They might be interested to merge your changes. But
for now I won't take them in the current Debian package.
David and Rian, you can download the patches at
http://bugs.debian.org/696122 (I'll also bounce you Carlos original mail).
Cheers,
#!/usr/bin/python
import urllib, urllib2, httplib
httplib.HTTPConnection.debuglevel = 1
dropbox_url = 'http://www.dropbox.com/download?plat=lnx.x86_64'
user_agent = 'DropboxLinuxDownloader/1.4.0'
class DropboxURLopener(urllib.FancyURLopener):
version = user_agent
urllib._urlopener = DropboxURLopener()
class DebugHTTPHandler(urllib2.HTTPHandler):
def __init__(self):
urllib2.HTTPHandler.__init__(self)
self.set_http_debuglevel(1)
class DebugHTTPSHandler(urllib2.HTTPSHandler):
def __init__(self):
urllib2.HTTPSHandler.__init__(self)
self.set_http_debuglevel(1)
opener = urllib2.build_opener(DebugHTTPHandler(), DebugHTTPSHandler())
opener.addheaders = [('User-agent', user_agent)]
urllib2.install_opener(opener)
for lib in (urllib, urllib2):
try:
print('\nUsing %s...\n' % lib.__name__)
s = lib.urlopen(dropbox_url)
size = int(s.info()['content-length'])
print('\nThe length of the stream is: %d\n' % size)
except KeyError as e:
print('\nGot key error: %s\n' % e.message)
finally:
s.close()
Script started on Tue 18 Dec 2012 09:14:23 PM EST
carlos@carlitos:/tmp$ printenv | grep -i proxy
http_proxy=http://192.168.0.1:8888/
https_proxy=http://192.168.0.1:8888/
carlos@carlitos:/tmp$ curl -vIL -A "DropboxLinuxDownloader/1.4.0" "http://www.dropbox.com/download?plat=lnx.x86_64";
* About to connect() to proxy 192.168.0.1 port 8888 (#0)
* Trying 192.168.0.1...
* connected
* Connected to 192.168.0.1 (192.168.0.1) port 8888 (#0)
> HEAD http://www.dropbox.com/download?plat=lnx.x86_64 HTTP/1.1
> User-Agent: DropboxLinuxDownloader/1.4.0
> Host: www.dropbox.com
> Accept: */*
> Accept-Encoding: deflate, gzip
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 302 FOUND
HTTP/1.1 302 FOUND
< Via: 1.1 tinyproxy (tinyproxy/1.8.3)
Via: 1.1 tinyproxy (tinyproxy/1.8.3)
< Date: Tue, 18 Dec 2012 10:14:42 GMT
Date: Tue, 18 Dec 2012 10:14:42 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Server: nginx
Server: nginx
< pragma: no-cache
pragma: no-cache
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< location: https://www.dropbox.com/download?plat=lnx.x86_64
location: https://www.dropbox.com/download?plat=lnx.x86_64
< set-cookie: gvc=NjgwMDg4MDU4OTYyMDg5MzI2NjIwNjY2NTUzNjY5OTAyNTYxMTM%3D; expires=Sun, 17-Dec-2017 10:14:42 GMT; Path=/; httponly
set-cookie: gvc=NjgwMDg4MDU4OTYyMDg5MzI2NjIwNjY2NTUzNjY5OTAyNTYxMTM%3D; expires=Sun, 17-Dec-2017 10:14:42 GMT; Path=/; httponly
< set-cookie: bang=; Domain=dropbox.com; expires=Tue, 18-Dec-2012 10:14:42 GMT; Path=/; httponly
set-cookie: bang=; Domain=dropbox.com; expires=Tue, 18-Dec-2012 10:14:42 GMT; Path=/; httponly
< set-cookie: flash=; Domain=dropbox.com; expires=Tue, 18-Dec-2012 10:14:42 GMT; Path=/; httponly
set-cookie: flash=; Domain=dropbox.com; expires=Tue, 18-Dec-2012 10:14:42 GMT; Path=/; httponly
< cache-control: no-cache
cache-control: no-cache
<
* Connection #0 to host 192.168.0.1 left intact
* Issue another request to this URL: 'https://www.dropbox.com/download?plat=lnx.x86_64'
* About to connect() to proxy 192.168.0.1 port 8888 (#1)
* Trying 192.168.0.1...
* connected
* Connected to 192.168.0.1 (192.168.0.1) port 8888 (#1)
* Establish HTTP proxy tunnel to www.dropbox.com:443
> CONNECT www.dropbox.com:443 HTTP/1.1
> Host: www.dropbox.com:443
> User-Agent: DropboxLinuxDownloader/1.4.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
HTTP/1.0 200 Connection established
< Proxy-agent: tinyproxy/1.8.3
Proxy-agent: tinyproxy/1.8.3
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES256-SHA
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Dropbox, Inc.; CN=*.dropbox.com
* start date: 2011-12
* expire date: 2014-01
* common name: *.dropbox.com (matched)
* issuer: C=US; O
* SSL certificate verify ok.
> HEAD /download?plat=lnx.x86_64 HTTP/1.1
> User-Agent: DropboxLinuxDownloader/1.4.0
> Host: www.dropbox.com
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 302 FOUND
HTTP/1.1 302 FOUND
< Server: nginx
Server: nginx
< Date: Tue, 18 Dec 2012 10:14:45 GMT
Date: Tue, 18 Dec 2012 10:14:45 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Connection: keep-alive
Connection: keep-alive
< set-cookie: gvc=MTMwMjIzNjM4MjkyMDcwNjYxNzQ4ODM0NTQ5OTI3NDI0MjQ0MDAx; expires=Sun, 17-Dec-2017 10:14:45 GMT; Path=/; httponly
set-cookie: gvc=MTMwMjIzNjM4MjkyMDcwNjYxNzQ4ODM0NTQ5OTI3NDI0MjQ0MDAx; expires=Sun, 17-Dec-2017 10:14:45 GMT; Path=/; httponly
< set-cookie: t=bIAyOSFibTvxCaamYUqiwmQs; Domain=dropbox.com; expires=Thu, 17-Jan-2013 10:14:45 GMT; Path=/; secure; httponly
set-cookie: t=bIAyOSFibTvxCaamYUqiwmQs; Domain=dropbox.com; expires=Thu, 17-Jan-2013 10:14:45 GMT; Path=/; secure; httponly
< strict-transport-security: max-age=2592000; includeSubDomains
strict-transport-security: max-age=2592000; includeSubDomains
< location: http://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64-1.6.6.tar.gz
location: http://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64-1.6.6.tar.gz
< pragma: no-cache
pragma: no-cache
< cache-control: no-cache
cache-control: no-cache
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
<
* Connection #1 to host 192.168.0.1 left intact
* Issue another request to this URL: 'http://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64-1.6.6.tar.gz'
* Connection #0 seems to be dead!
* Closing connection #0
* About to connect() to proxy 192.168.0.1 port 8888 (#0)
* Trying 192.168.0.1...
* connected
* Connected to 192.168.0.1 (192.168.0.1) port 8888 (#0)
> HEAD http://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64-1.6.6.tar.gz HTTP/1.1
> User-Agent: DropboxLinuxDownloader/1.4.0
> Host: dl-web.dropbox.com
> Accept: */*
> Accept-Encoding: deflate, gzip
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Via: 1.1 tinyproxy (tinyproxy/1.8.3)
Via: 1.1 tinyproxy (tinyproxy/1.8.3)
< Date: Tue, 18 Dec 2012 10:14:49 GMT
Date: Tue, 18 Dec 2012 10:14:49 GMT
< Content-Type: application/x-tar
Content-Type: application/x-tar
< Server: nginx/1.2.3
Server: nginx/1.2.3
< pragma: public
pragma: public
< x-robots-tag: noindex,nofollow
x-robots-tag: noindex,nofollow
< accept-ranges: bytes
accept-ranges: bytes
< Content-Length: 19608575
Content-Length: 19608575
< cache-control: max-age=0
cache-control: max-age=0
< x-server-response-time: 1058
x-server-response-time: 1058
<
* Connection #0 to host 192.168.0.1 left intact
* Closing connection #1
* SSLv3, TLS alert, Client hello (1):
* Closing connection #0
carlos@carlitos:/tmp$ python -V
Python 2.7.3
carlos@carlitos:/tmp$ ./urllibdebug.py
Using urllib...
send: 'GET http://www.dropbox.com/download?plat=lnx.x86_64 HTTP/1.0\r\nHost: www.dropbox.com\r\nUser-Agent: DropboxLinuxDownloader/1.4.0\r\n\r\n'
reply: 'HTTP/1.1 302 FOUND\r\n'
header: Via: 1.0 tinyproxy (tinyproxy/1.8.3)
header: x-frame-options: SAMEORIGIN
header: Date: Tue, 18 Dec 2012 10:15:12 GMT
header: location: https://www.dropbox.com/download?plat=lnx.x86_64
header: Content-Type: text/html; charset=utf-8
header: set-cookie: gvc=MTQ3NzAwNjMyMDkyNDY3NzYxMjY1NzI4MDk5OTcyNzM0NjQ5Nzk5; expires=Sun, 17-Dec-2017 10:15:12 GMT; Path=/; httponly
header: set-cookie: bang=; Domain=dropbox.com; expires=Tue, 18-Dec-2012 10:15:12 GMT; Path=/; httponly
header: set-cookie: flash=; Domain=dropbox.com; expires=Tue, 18-Dec-2012 10:15:12 GMT; Path=/; httponly
header: Server: nginx
header: cache-control: no-cache
header: pragma: no-cache
send: 'GET https://www.dropbox.com/download?plat=lnx.x86_64 HTTP/1.0\r\nUser-Agent: DropboxLinuxDownloader/1.4.0\r\n\r\n'
reply: 'HTTP/1.0 500 Unable to connect\r\n'
header: Server: tinyproxy/1.8.3
header: Content-Type: text/html
header: Connection: close
Got key error: content-length
Using urllib2...
send: 'GET http://www.dropbox.com/download?plat=lnx.x86_64 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.dropbox.com\r\nConnection: close\r\nUser-Agent: DropboxLinuxDownloader/1.4.0\r\n\r\n'
reply: 'HTTP/1.1 302 FOUND\r\n'
header: Via: 1.1 tinyproxy (tinyproxy/1.8.3)
header: Date: Tue, 18 Dec 2012 10:16:17 GMT
header: Content-Type: text/html; charset=utf-8
header: Server: nginx
header: cache-control: no-cache
header: x-frame-options: SAMEORIGIN
header: location: https://www.dropbox.com/download?plat=lnx.x86_64
header: set-cookie: gvc=MzM5NjMwNTU3ODc3OTI0NzExNzA1MzIyNzIwOTE3MTQ1Mjc0NzIz; expires=Sun, 17-Dec-2017 10:16:17 GMT; Path=/; httponly
header: set-cookie: bang=; Domain=dropbox.com; expires=Tue, 18-Dec-2012 10:16:17 GMT; Path=/; httponly
header: set-cookie: flash=; Domain=dropbox.com; expires=Tue, 18-Dec-2012 10:16:17 GMT; Path=/; httponly
header: pragma: no-cache
send: 'CONNECT www.dropbox.com:443 HTTP/1.0\r\n'
send: '\r\n'
send: 'GET /download?plat=lnx.x86_64 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.dropbox.com\r\nConnection: close\r\nUser-Agent: DropboxLinuxDownloader/1.4.0\r\n\r\n'
reply: 'HTTP/1.1 302 FOUND\r\n'
header: Server: nginx
header: Date: Tue, 18 Dec 2012 10:16:20 GMT
header: Content-Type: text/html; charset=utf-8
header: Transfer-Encoding: chunked
header: Connection: close
header: set-cookie: gvc=MzA2NTgwNjIyOTQzMTI4OTAzNjU5ODM5Njc5MzY4NDUyNTUxOTE3; expires=Sun, 17-Dec-2017 10:16:20 GMT; Path=/; httponly
header: set-cookie: t=MANSVCsbbWz_W9iAadWSDZ7D; Domain=dropbox.com; expires=Thu, 17-Jan-2013 10:16:20 GMT; Path=/; secure; httponly
header: strict-transport-security: max-age=2592000; includeSubDomains
header: location: http://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64-1.6.6.tar.gz
header: pragma: no-cache
header: cache-control: no-cache
header: x-frame-options: SAMEORIGIN
send: 'GET http://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64-1.6.6.tar.gz HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: dl-web.dropbox.com\r\nConnection: close\r\nUser-Agent: DropboxLinuxDownloader/1.4.0\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Via: 1.1 tinyproxy (tinyproxy/1.8.3)
header: x-server-response-time: 1132
header: Date: Tue, 18 Dec 2012 10:16:23 GMT
header: x-robots-tag: noindex,nofollow
header: accept-ranges: bytes
header: Server: nginx/1.2.3
header: Content-Length: 19608575
header: Content-Type: application/x-tar
header: cache-control: max-age=0
header: pragma: public
The length of the stream is: 19608575
carlos@carlitos:/tmp$ exit
exit
Script done on Tue 18 Dec 2012 09:16:30 PM EST
