Ahh! By sheer coincidence, it was actually me that integrated the latest SOCKS/proxy support you see in urllib3 lol.
Basically, the current proxy handling is quite badly broken and required a bunch of fixes. I submitted a stable set of patches for integration here; https://github.com/shazow/urllib3/pull/68 This was later merged and branched by the core repo maintainer on the following branch; https://github.com/shazow/urllib3/tree/socks-proxy Sadly, the project we were working on that required proxy access was finished, and I wasn't able to contribute any more time. There are a few ways to hack this into working.. most of them involve monkey-patching the hell out of urllib3 ;) You could maybe try patching up 'path_url', with something really hacky like str.replace("http://localhost:8080";, "") (don't shoot me!) Let me know how you get on Cal On Tue, Oct 2, 2012 at 10:31 PM, Victor Hooi <victorh...@gmail.com> wrote: > heya, > > Thanks for the tips - you're probably right, I might need to whip out > wireshark or something and see what exactly is going on. > > However, one thing I did notice - I normally have the http_proxy > environment variable set, as we use a HTTP proxy at work. > > However, if I unset the http_proxy variable, Python requests suddenly > seems to start working again. > > I tried to set the no_proxy variable, and put in localhost and 127.0.0.1 > in there - however, Python requests doesn't seem to respect that? > > Cheers, > Victor > > > On Tuesday, 2 October 2012 22:48:26 UTC+10, Cal Leeming [Simplicity Media > Ltd] wrote: > >> Hi Victor, >> >> I've had my fair share of exposure with python requests - so thought I'd >> chime in. >> >> On first glance, this looks to be an issue with specifying the port >> number into python-requests, doing so seems to send the entire " >> http://localhost:8000/api/v1/**host/?name__regex=&format=json<http://localhost:8000/api/v1/host/?name__regex=&format=json> >> **" as the request. However, further analysis shows that might not be >> the case. >> >> Looking at the python requests code; >> https://github.com/**kennethreitz/requests/blob/** >> develop/requests/models.py<https://github.com/kennethreitz/requests/blob/develop/requests/models.py> >> >> >>> urlparse.urlparse("http://**localhost:8080/test/url?with=**params<http://localhost:8080/test/url?with=params> >> ") >> ParseResult(scheme='http', netloc='localhost:8080', path='/test/url', >> params='', query='with=params', fragment='') >> >> It then sends this directly into urllib3 using connection_from_url(); >> https://github.com/shazow/**urllib3/blob/master/urllib3/** >> connectionpool.py<https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py> >> >> This then calls the following; >> scheme, host, port = get_host(url) >> if scheme == 'https': >> return HTTPSConnectionPool(host, port=port, **kw) >> else: >> return HTTPConnectionPool(host, port=port, **kw) >> >> get_host -> parse_url() >> https://github.com/shazow/**urllib3/blob/master/urllib3/**util.py<https://github.com/shazow/urllib3/blob/master/urllib3/util.py> >> >> Tracing through urllib3 finally gets to parse_url(); >> >> >>> urllib3.util.parse_url("http:/**/localhost:8080/test/url?with=** >> params <http://localhost:8080/test/url?with=params>") >> Url(scheme='http', auth=None, host='localhost', port=8080, >> path='/test/url', query='with=params', fragment=None) >> >> So, lets look at path_url() instead; >> https://github.com/**kennethreitz/requests/blob/** >> develop/requests/models.py<https://github.com/kennethreitz/requests/blob/develop/requests/models.py> >> >> >>> lol = requests.get("http://**localhost:8000/api/v1/host/?** >> name__regex=&format=json<http://localhost:8000/api/v1/host/?name__regex=&format=json> >> ") >> >>> lol.request.path_url >> '/api/v1/host/?name__regex=&**format=json' >> >> Performing a test connection shows; >> >> foxx@test01.internal [~] > nc -p8000 -l >> GET /api/v1/host/?name__regex=&**format=json HTTP/1.1 >> Host: localhost:8000 >> Accept-Encoding: identity, deflate, compress, gzip >> Accept: */* >> User-Agent: python-requests/0.11.1 >> >> So, from what I can tell, python requests is functioning normally. >> >> Personally, I'd say get wireshark running, or use the nc trick shown >> above, perform 1 request using curl and 1 using python requests, then >> compare the request headers. >> >> Can't throw much more time at this, but hope the above helps >> >> Cal >> >> On Tue, Oct 2, 2012 at 8:54 AM, Victor Hooi <victo...@gmail.com> wrote: >> >>> Hi, >>> >>> I have a Django app that's serving up a RESTful API using tasty-pie. >>> >>> I'm using Django's development runserver to test. >>> >>> When I access it via a browser it works fine, and using Curl also works >>> fine: >>> >>> curl "http://localhost:8000/api/v1/**host/?name__regex=&format=** >>>> json <http://localhost:8000/api/v1/host/?name__regex=&format=json>" >>> >>> >>> On the console with runserver, I see: >>> >>> [02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=&**amp;format=json >>>> HTTP/1.1" 200 2845 >>> >>> >>> However, when I try to use the Python requests module ( >>> http://docs.python-requests.**org/en/latest/<http://docs.python-requests.org/en/latest/>), >>> I get a 404: >>> >>> >>> r = requests.get('http://**localhost:8000/api/v1/host/?** >>>> name__regex=&format=json<http://localhost:8000/api/v1/host/?name__regex=&format=json> >>>> ') >>>> >>> r >>>> <Response [404]> >>> >>> >>> or: >>> >>> >>> r = requests.get('http://**localhost:8000/api/v1/host/?** >>>> name__regex=&format=json<http://localhost:8000/api/v1/host/?name__regex=&format=json> >>>> ') >>>> >>> r >>>> <Response [404]> >>> >>> >>> or: >>> >>> >>> payload = { 'format': 'json'} >>>> >>> r = >>>> >>> requests.get('http://**localhost:8000/api/v1<http://localhost:8000/api/v1>', >>>> params=payload) >>>> >>> r >>>> <Response [404]> >>>> >>> r.url >>>> u'http://localhost:8000/api/**v1?format=json<http://localhost:8000/api/v1?format=json> >>>> ' >>> >>> >>> Also, on the Django runserver console, I see: >>> >>> [02/Oct/2012 17:25:01] "GET http://localhost:8000/api/v1/** >>>> host/?name__regex=&format=json<http://localhost:8000/api/v1/host/?name__regex=&format=json>HTTP/1.1" >>>> 404 161072 >>> >>> >>> For some reason, when I use requests, runserver prints out the whole >>> request URL, including localhost - but when I use the browser, or curl, it >>> only prints out the part *after* the hostname:port >>> >>> I'm assuming this is something to do with the encoding, user-agent or >>> request type it's sending? Why does runserver print out different URLs for >>> requests versus browser/curl? >>> >>> Cheers, >>> Victor >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django users" group. >>> To view this discussion on the web visit https://groups.google.com/d/** >>> msg/django-users/-/**ycLjP71ciAEJ<https://groups.google.com/d/msg/django-users/-/ycLjP71ciAEJ> >>> . >>> To post to this group, send email to django...@googlegroups.com. >>> To unsubscribe from this group, send email to django-users...@** >>> googlegroups.com. >>> >>> For more options, visit this group at http://groups.google.com/** >>> group/django-users?hl=en<http://groups.google.com/group/django-users?hl=en> >>> . >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/django-users/-/AebVvlYTg5EJ. > > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
