Hi Chris Proxies are on the application layer, TCP is a transport layer protocol below. Using the application layer to establish in-band transport layer connections is dodgy by its very nature. The multiple sockets to get through an HTTP proxy issue is not unique to Java. Here's a quick look at how several open source applications have dealt with the problem:
* prtunnel (http://joshbeam.com/software/prtunnel.php), a C application for tunneling through several proxy types, only supports no authentication and BASIC HTTP authentication. proxychains does the same. XChat does the same. * ntlmaps.sourceforge.net and similar apps work by acting as another proxy themselves. They can use as many sockets as necessary on the outgoing end to the second proxy. * pidgin (www.pidgin.im), an instant messaging app, has very good proxy support including NTLM authentication for HTTP proxies. It only uses a single socket and requires the proxy to use a persistent connection. proxytunnel (proxytunnel.sourceforge.net) does the same. I like pidgin's solution the best, and since it even works with NTLM, the other authentication types should be easy. Most HTTP 1.1 proxies should support persistent connections - after all, that feature benefits proxies the most. So we should be able to get away with one socket in most cases, maybe even for all commonly used HTTP proxies. I'll try to do some tests this weekend. Also, nothing stops a Socket from changing its impl. We would need a very clever impl that somehow remembers options that have been set and reapplies them to the new impl, but there's no theoretical reason why connect() cannot create as many impls as are necessary to get through the proxy. On the other hand, all Java's solutions for connecting through proxies are poor: SOCKS 4 is broken (http://mail.openjdk.java.net/pipermail/net-dev/2010-February/001580.html), server sockets are unsupported, datagram sockets are unsupported, and all java.nio channels are unsupported. But then proxies aren't so common any more: ADSL routers typically use NAT instead, and we don't provide any NAT traversal support for server sockets and datagram sockets using UPnP's IGD or Bonjour's NAT-PMP protocols. Maybe we should? Damjan On Wed, Mar 3, 2010 at 3:55 PM, Christopher Hegarty -Sun Microsystems Ireland <christopher.hega...@sun.com> wrote: > Hi Damjan, > > After spending more time looking at this, I still haven't found an elegant > solution for supporting authentication. > > You've correctly identified an issue with the fact we have no guaranteed of > a persistent connection with the proxy. Changing the socket identity is not > as simple as overriding getIn/OutputStream since other methods like > setting/getting options, close, shutdown, etc use the impls fd field. This > field needs to refer to the correct file descriptor object. > > There is quite a lot of logic in the http protocol handler to support > authentication and looking at it again it is closely tied to the handler > implementation itself. I would prefer to reuse this code if possible. > > Have you made any progress on this? If we want authentication it may be more > straight forward to revert back to your original proposal of using > HttpURLConnection directly and having its socket exposed. Though, I'm still > not convinced of this solution either. > > -Chris > > Damjan Jovanovic wrote: >> >> Hi Chris >> >> I think that without authentication, you'll just get another RFE >> asking for authentication. Many corporate proxies require >> authentication, especially for a liberal request like CONNECT. >> >> java.net.Socket seems to delegate its impl field's methods. If we have >> another, internal socket in HttpConnectSocketImpl, and we override >> getInputStream() and getOutputStream() in HttpConnectSocketImpl to >> return this inner socket's streams, we can open as many sockets as we >> like during authentication, and then just store the last one that >> connected as this internal socket? This seems like a way to "change >> socket identities" like I suggested. >> >> Thank you >> Damjan >> >> On Mon, Mar 1, 2010 at 4:29 PM, Christopher Hegarty - Sun Microsystems >> Ireland <christopher.hega...@sun.com> wrote: >>> >>> Hi Damjan, >>> >>> Sorry for the delayed response, I'm juggling too many things! >>> >>> I'm not sure how important Authentication is here. I initially pointed it >>> out, but hadn't done sufficient scoping so didn't encounter these issues. >>> Let me take another look at it, but I'm wondering if we should just >>> proceed >>> with it as is for now. Authentication is the reason that this RFE never >>> made >>> it back yet and has been sitting for a few years now. >> >> >>> -Chris. >>> >>> On 24/02/2010 17:28, Damjan Jovanovic wrote: >>>> >>>> On Mon, Feb 22, 2010 at 5:02 PM, Christopher Hegarty - Sun >>>> Microsystems Ireland<christopher.hega...@sun.com> wrote: >>>>> >>>>> Hi Damjan, >>>>> >>>>> Actually, I did some work on this back in 2006 (!), but never finished >>>>> it. I >>>>> brought the changes into a mercurial repository and created a webrev: >>>>> >>>>> http://cr.openjdk.java.net/~chegar/6370908/webrev.00/webrev/ >>>>> >>>>> Basically, this change provides the basic functionality, without any >>>>> frills, >>>>> authentication, etc. I think for tunneling sockets through a HTTP proxy >>>>> it >>>>> should be sufficient. Do you require authentication in your >>>>> environment? >>>>> >>>>> To have authentication supported we would need to restructure the HTTP >>>>> protocol handler in sun.net.www.protocol.http.HttpURLConnection, so >>>>> that >>>>> we >>>>> can take advantage of the authentication schemes it already supports. >>>>> Not >>>>> a >>>>> big deal, just needs to be done. >>>> >>>> The major problem with authentication is that there is no guarantee >>>> the HTTP connection will be persistent. Authentication generally >>>> requires making additional requests to the proxy, so we may need to >>>> connect to the proxy more than once. Each socket can only connect >>>> once, and the HttpConnectSocketImpl is a single immutable socket, so >>>> it cannot directly be used for authentication. >>>> >>>> A workaround may be to connect with a different socket for the initial >>>> request, and then connect through the HttpConnectSocketImpl's own >>>> socket for the final request. But then the problem becomes that if the >>>> proxy doesn't require authentication, the initial socket will succeed >>>> in connecting to the target host, and it will be too late to stop it >>>> and use HttpConnectSocketImpl's own socket... >>>> >>>> So we have the following options: >>>> 1. Only support proxies for which the entire request chain can be made >>>> over a single persistent connection. >>>> 2. Connect to each target host up to twice, once when we believe the >>>> proxy needs authentication but doesn't, and then again later with the >>>> real socket. This will probably kill protocols that use one shot >>>> connections, like FTP. >>>> 3. Find some way to change socket identities, then use as many sockets >>>> as it takes and somehow adopt the final socket into >>>> HttpConnectSocketImpl's own. >>>> >>>> Any suggestions? >>>> >>>>> -Chris. >>>> >>>> Damjan >>>> >>>>> On 21/02/2010 13:09, Damjan Jovanovic wrote: >>>>>> >>>>>> Hi >>>>>> >>>>>>> From http://bugs.sun.com/view_bug.do?bug_id=6370908 >>>>>> >>>>>> This RFE is basically about getting a TCP socket to tunnel through an >>>>>> HTTP proxy using the HTTP CONNECT request. >>>>>> >>>>>> I've found a hack to get this feature to work, using sun.net.* >>>>>> packages and lots of reflection. Would it be acceptable to use this >>>>>> solution (with some way to change socket identity) in a patch that >>>>>> adds a java.net.HttpSocketImpl class similar to the >>>>>> java.net.SocksSocketImpl class that's already used to tunnel through >>>>>> SOCKS proxies? If not, in what other way should such a patch be done? >>>>>> >>>>>> Thank you >>>>>> Damjan Jovanovic >>>>>> >>>>>> import java.net.*; >>>>>> import java.io.*; >>>>>> import java.lang.reflect.*; >>>>>> >>>>>> public class TunnelProxy { >>>>>> private static Socket connectThroughHTTPProxy(String proxyHost, >>>>>> int >>>>>> proxyPort, String destinationHost, int destinationPort) throws >>>>>> Exception >>>>>> { >>>>>> URL destinationURL = new URL("http://"; + destinationHost >>>>>> + >>>>>> ":" + >>>>>> destinationPort); >>>>>> sun.net.www.protocol.http.HttpURLConnection conn = >>>>>> new sun.net.www.protocol.http.HttpURLConnection( >>>>>> destinationURL, new >>>>>> java.net.Proxy(java.net.Proxy.Type.HTTP, new >>>>>> InetSocketAddress(proxyHost, proxyPort))); >>>>>> conn.setDoInput(true); >>>>>> conn.setDoOutput(true); >>>>>> conn.connect(); >>>>>> conn.doTunneling(); >>>>>> Field httpField = >>>>>> conn.getClass().getDeclaredField("http"); >>>>>> httpField.setAccessible(true); >>>>>> sun.net.www.http.HttpClient httpClient = >>>>>> (sun.net.www.http.HttpClient) httpField.get(conn); >>>>>> Field serverSocketField = >>>>>> sun.net.NetworkClient.class.getDeclaredField("serverSocket"); >>>>>> serverSocketField.setAccessible(true); >>>>>> Socket socket = (Socket) >>>>>> serverSocketField.get(httpClient); >>>>>> return socket; >>>>>> } >>>>>> >>>>>> public static void main(String[] args) throws Exception { >>>>>> System.setProperty("java.net.useSystemProxies", "true"); >>>>>> InputStream in = connectThroughHTTPProxy(args[0], >>>>>> Integer.parseInt(args[1]), args[2], >>>>>> Integer.parseInt(args[3])).getInputStream(); >>>>>> byte[] bytes = new byte[1024]; >>>>>> int bytesRead; >>>>>> while ((bytesRead = in.read(bytes)) != -1) { >>>>>> System.out.print(new String(bytes)); >>>>>> } >>>>>> } >>>>>> } >
