Hi Aleks, Yes, I have looked at your previous email. Those 3 URLs you mentioned below --
> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep i have done them already, otherwise, both of my SP's will not work. I had this setup from day one. As for your question on pointing shibboleth SP to proxy.example.com, I don't think that is doable, as shibboleth SP needs to access the actual resource ( i.e. jupyterhub instance in this case ), which is in the backend node. Are there any plans to support shibboleth with HAProxy? As you know, apache/nginx has both support for it. If my current setup is not acceptable by mgmt . , I will have to start looking at nginx on how to do this. The current setup directly allows the backend nodes to be accessible by users, which, I don't think is the best thing. --imam On Sat, Oct 27, 2018 at 10:34 AM Aleksandar Lazic <[email protected]> wrote: > Hi. > > Thanks for the infos. Have you also seen the other answers in the previous > mail? > > Regards > Aleks > > ------------------------------ > *Von:* Imam Toufique <[email protected]> > *Gesendet:* 27. Oktober 2018 19:06:50 MESZ > *An:* [email protected] > *CC:* Igor Cicimov <[email protected]>, haproxy < > [email protected]> > *Betreff:* Re: apache proxy pass rules in HAproxy > > Hi Aleks, > Yes, I should have done last in my last email post. Sorry about that. > > haproxy version: > > [root@crsplabnet2 haproxy]# haproxy -vv > HA-Proxy version 1.8.14-52e4d43 2018/09/20 > Copyright 2000-2018 Willy Tarreau <[email protected]> > > Build options : > TARGET = linux2628 > CPU = generic > CC = gcc > CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement > -fwrapv -fno-strict-overflow -Wno-unused-label > OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017 > Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 > Built with transparent proxy support using: IP_TRANSPARENT > IPV6_TRANSPARENT IP_FREEBIND > Encrypted password support via crypt(3): yes > Built with multi-threading support. > Built with PCRE version : 8.32 2012-11-30 > Running on PCRE version : 8.32 2012-11-30 > PCRE library supports JIT : no (USE_PCRE_JIT not set) > Built with zlib version : 1.2.7 > Running on zlib version : 1.2.7 > Compression algorithms supported : identity("identity"), > deflate("deflate"), raw-deflate("deflate"), gzip("gzip") > Built with network namespace support. > > Available polling systems : > epoll : pref=300, test result OK > poll : pref=200, test result OK > select : pref=150, test result OK > Total: 3 (3 usable), will use epoll. > > Available filters : > [SPOE] spoe > [COMP] compression > [TRACE] trace > > ----------------------------- > Shibboleth version ( running in both of my backend nodes, as HAproxy can't > interact with shib , to do authentication offload in HAProxy ) > > [root@web1 ~]# rpm -qa |grep hibb > shibboleth-3.0.2-1.1.x86_64 > > ------------------------------- > > Apache version ( running in both backend nodes ONLY ) : > > [root@web1 ~]# rpm -qa |grep httpd > httpd-devel-2.4.6-80.el7.centos.x86_64 > httpd-tools-2.4.6-80.el7.centos.x86_64 > httpd-2.4.6-80.el7.centos.x86_64 > > ----------------------------------- > Linux version in all systems: > > Centos 7.5 , running stock kernel 3.10.0-862.el7.x86_64 > > here is a workflow and some background info: > > background info: > A. the application that we are trying to use: jupyterhub -- > https://github.com/jupyterhub > B. We have shibboleth IDP that all our internal sites do authentication > from. > C. This application Jupyterhub spawns its own ( internally built ) proxy > instance and binds that to port 8000, as you saw in the <Location> config > section in apache config above. > D. Jupyterhub can directly authenticate with shibboleth ( some additional > work is needed ), but since we are putting this behind HAProxy, running > Shibboleth SP in the backend node ( where the jupyterhub instance runs ) is > needed. therefore, I decided to wrap jupytrehub within apache as apache > has built-in support for shibboleth ( mod_shib , that gets installed with > shibboleth RPM ). > > Here is what I was thinking expected workflow would be: > > > > Client [ public IP ] connects ---> HAproxy [ public IP ] connects ---> > backend node [ private IP:80 ] > > Note: Shibboleth is running in the backend node with its public IP: 443, > as it loves to work with SSL and I needed an in-common SSL cert installed > to get the SP working correctly. > > > > So, for example, in the browser, I go to: > > -- proxy.example.com ( HAProxy system ) > -- from the backend, shibboleth responds to me with a login screen. > -- I authenticate with username/password > -- Shibboleth hands my username as an attribute called "UserID". to > Apache, and I forward that "UserID" to jupyterhub. Apache pulls the > jupyterhub instance, presents to the user. > > Here is what I expect: > > Type: > > URL in the browser: 'proxy.exmaple.com/jhub' > > > After authentication, the browse URL shown to connect directly to the > backend node, such as: > > https://backendnode1.example.com/jhub/itoufiqu/tree? > > whereas, I would expect it to keep the proxy URL to be: > https://proxy.exmaple.com/jhub/itoufiqu/tree? > > Interesting thing is, this redirect to the backend machine, connects the > client directly to the backend machine over the public IP of the backend > machine. I think this is due to SSO being done by shibboleth. Since > Shibboleth fails authentication from within the internal LAN ( as the IDP > could not talk to the SP in the private network), I had to get a public > IP. In this process, HAProxy loses all accounting data to this session, > except the first connection attempt. > > This setup could be good or bad, I am not sure. My initial thoughts were, > all traffic passes through HAProxy. In retrospect, HAProxy is not being > taxed at all, there is a direct connection between the client and the > backend node over a public IP ( of the backend node). But, what I don't > understand is, if a connection attempt is made through HAProxy, why would > it allow the connection to be handed off to the backend node directly ( > client -> backend node )? > > Thoughts? > > > On Sat, Oct 27, 2018 at 4:31 AM Aleksandar Lazic <[email protected]> > wrote: > >> Hi Imam. >> >> It would be helpfull to know your used versions: >> >> haproxy -vv >> apache httpd version >> shibboleth version >> >> A small workflow picture like: >> >> Client -> haproxy -> apache httpd -> shibboleth ? >> >> Am 27.10.2018 um 07:44 schrieb Imam Toufique: >> > Hi Igor, >> > >> > Thanks very much for offering to help! I will do this in sections, >> hopefully, I >> > can keep this from being too cluttered. >> > >> > haproxy.cfg: >> > >> -------------------------------------------------------------------------------------- >> > global >> > #log /dev/log local0 debug >> > #log /dev/log local1 debug >> > log 127.0.0.1 local2 >> > chroot /var/lib/haproxy >> > stats timeout 30s >> > user haproxy >> > group haproxy >> > tune.ssl.default-dh-param 2048 >> > daemon >> > >> > defaults >> > log global >> > mode http >> > option tcplog >> > option dontlognull >> > timeout connect 5000 >> > timeout client 50000 >> > timeout server 50000 >> > timeout tunnel 9h >> > option tcp-check >> > >> > frontend http_front >> > bind :80 >> > bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >> > stats uri /haproxy?stats >> > default_backend web1_cluster >> > option httplog >> > log global >> > #option dontlognull >> > log /dev/log local0 debug >> >> The 2 log entries are redundant, imho. >> >> > mode http >> >> Is set at default block. Please take a look at >> >> https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/ >> >> > option forwardfor # forward IP >> > http-request set-header X-Forwarded-Port %[dst_port] >> > http-request add-header X-Forwarded-Proto https if { ssl_fc } >> >> I personally would use here also set-header instead of add. >> >> > redirect scheme https if !{ ssl_fc } >> > >> > acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav >> >> This should be only the host name. >> >> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.6-hdr >> and >> >> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.6-req.hdr >> >> I would add a addition acl. >> >> acl host_web2 hdr(host) -i crsplab2.oit.uci.edu >> acl path_web2 path_beg -i /webdav >> >> > use_backend webdav_cluster if host_web2 >> >> an add it to the use_backend line >> >> use_backend webdav_cluster if host_web2 path_web2 >> >> > acl host_web3 path_beg /jhub >> > use_backend web3_cluster if host_web3 >> > >> > >> > backend webdav_cluster >> > balance roundrobin >> >> I would add here the following line. >> >> cookie SRV1 insert indirect nocache >> > server web1 10.1.100.156:8080 check inter 2000 cookie w1 >> > server web2 10.1.100.160:8080 check inter 2000 cookie w2 >> > >> > backend web3_cluster >> >> I would add here the following line. >> >> cookie SRV2 insert indirect nocache >> > server publicIP:443 check ssl verify none inter 2000 cookie w1 >> > >> ----------------------------------------------------------------------------------------------------- >> > Note: I have a single backend node, as it was easy to test with just >> one node, >> > instead of making changes to 2 nodes at a time. >> > >> > Here is my apache config: >> > >> > in httpd.conf, only change I have made is ( the rest is a stock centos >> 7.5 >> > httpd.conf ): >> > ------------------------------------- >> > ServerName 10.1.100.160:80 ( Internal IP of the backend node) >> > Redirect permanent /jhub https://crsplabweb1.domain.com/jhub >> > ------------------------------------- >> > >> > in my ssl.conf, where I access the jupyterhub instance running in >> 127.0.0.1:8000 >> > <http://127.0.0.1:8000> . Also, note that the backend is running >> shibboleth >> > SP. One of the issues I encountered is, If I did not have SSL , i was >> getting a >> > browser warning for not having SSL. >> Can you set up shibboleth in that manner that he answers with >> proxy.domain.com? >> As we don't know which version is in place I send you just as a hint here >> some >> links. >> >> https://wiki.shibboleth.net/confluence/display/SHIB2/SPReverseProxy >> >> http://shibboleth.1660669.n2.nabble.com/shibboleth-sp-behind-an-HAproxy-td5960149.html >> >> https://duckduckgo.com/?q=shibboleth+behind+haproxy >> >> >> > Here is my ssl.conf: >> > >> > >> -------------------------------------------------------------------------- >> > Listen 443 https >> > SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog >> > SSLSessionCache shmcb:/run/httpd/sslcache(512000) >> > SSLSessionCacheTimeout 300 >> > SSLRandomSeed startup file:/dev/urandom 256 >> > SSLRandomSeed connect builtin >> > SSLCryptoDevice builtin >> > >> > <VirtualHost _default_:443> >> > >> > UseCanonicalName on >> > ServerName crsplabweb1.domain.com:443 >> >> Maybe you can change this to proxy.domain.com and add a ServerAlias >> >> ServerName proxy.domain.com >> ServerAlias crsplabweb1 crsplabweb1.domain.com >> >> https://httpd.apache.org/docs/2.4/mod/core.html#serveralias >> >> > ErrorLog logs/ssl_error_log >> > TransferLog logs/ssl_access_log >> > LogLevel warn >> > >> > SSLEngine on >> > >> > SSLProtocol all -SSLv2 -SSLv3 >> > SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA >> > SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer >> > SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key >> > SSLCertificateChainFile >> /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c >> > >> > <Files ~ "\.(cgi|shtml|phtml|php3?)$"> >> > SSLOptions +StdEnvVars >> > </Files> >> > <Directory "/var/www/cgi-bin"> >> > SSLOptions +StdEnvVars >> > </Directory> >> > >> > <Location /jhub> >> > ProxyPass http://127.0.0.1:8000/jhub >> > ProxyPassReverse http://127.0.0.1:8000/jhub >> > RequestHeader unset Accept-Encoding >> > ProxyPreserveHost on >> > AuthType shibboleth >> > ShibRequestSetting requireSession 1 >> > Require shibboleth >> > ShibUseHeaders On >> > ShibBasicHijack On >> > RewriteEngine On >> > RequestHeader set X-Remote-User %{REMOTE_USER}s >> > </Location> >> > >> > <LocationMatch >> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >> > ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3 >> > ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3 >> > </LocationMatch> >> > >> > BrowserMatch "MSIE [2-5]" \ >> > nokeepalive ssl-unclean-shutdown \ >> > downgrade-1.0 force-response-1.0 >> > >> > CustomLog logs/ssl_request_log \ >> > "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" >> > </VirtualHost> >> > >> ---------------------------------------------------------------------------------- >> > >> > Thanks >> >> Regards >> Aleks >> >> > On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov < >> [email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Hi Imam, >> > >> > On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Hi, >> > >> > I came up with the following config, things seem to be working >> now, for >> > the most part. >> > >> > frontend http_front >> > bind :80 >> > bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >> > stats uri /haproxy?stats >> > default_backend web1_cluster >> > option httplog >> > log global >> > #option dontlognull >> > log /dev/log local0 debug >> > mode http >> > option forwardfor # forward IP >> > http-request set-header X-Forwarded-Port %[dst_port] >> > http-request add-header X-Forwarded-Proto https if { ssl_fc } >> > redirect scheme https if !{ ssl_fc } >> > acl host_web3 path_beg /jhub >> > use_backend web3_cluster if host_web3 >> > >> > web3_cluster >> > >> > backend web3_cluster >> > mode http >> > balance source >> > server crsplabweb1.domain.com publicIP:443 check ssl verify >> none inter 2000 cookie w1 >> > >> > The above config gets me to the backend node -- where I have a >> > jupyterhub instance running + . Shibboleth SP running for >> > authentication. As I could not get shibboleth SP to work by >> staying in >> > my private network, I had to set up a public IP for the backend >> node, >> > get SSL certs - so shibboleth authentication could be done. I >> am sure >> > there is a better approach to this, but I don't know what it >> is. I will >> > be trying out SNAT to see if that will allow me to keep using >> my private >> > IP for the backend nodes. If any of you know how to do SNAT, >> please >> > chime in, it would be worth the time/effort to try it out. >> > >> > Now, the interesting thing I have noticed with the above setup >> -- when I >> > connect to HAProxy, let's say with https://proxy.domain.com , I >> > authenticate with shibboleth, and then the URL in the browser >> points to >> > the backend node. >> > >> > For example: >> > >> > my proxy address: https://proxy.domain.com/jhub >> > >> > after I connect to the backend, the URL turns into - >> > https://crsplabweb1.domain.com/jhub/tree? >> > >> > ...and everything works thereafter. >> > >> > I tried the rewrite method that Igor has suggested before, that >> did not >> > make any difference. But what I noticed is, after I connect, >> no traffic >> > go through the proxy anymore, my client ( i.e. laptop) connects >> directly >> > to the backend server. Not sure if this good or bad though (?) >> , but, I >> > am not sure how to configure this so that I will go through a >> proxy but >> > still be connected in the backend via a private IP and I can ( >> still ) >> > authenticate via shibboleth. >> > >> > So, when I change the 'web3_cluster' backend to : >> > >> > server crsplabweb1 privateIP:80 inter 2000 cookie w1 >> > >> > and, I set backend apache to accept connection on port 80, then >> I break >> > shibboleth authentication. >> > >> > Any inputs here? >> > >> > thanks, guys! >> > >> > >> > I think it is time for you to provide the full HAP and Apache >> configs so we >> > can see what is going on (please obfuscate any sensitive data). >> Also the use >> > of the "cookie w1" is not clear since you are not setting it in HAP >> and is >> > kinda redundant for single backend setup. >> > >> > >> > >> > On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov >> > <[email protected] <mailto: >> [email protected]>> >> > wrote: >> > >> > >> > >> > On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov >> > <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > >> > >> > On Thu, 25 Oct 2018 6:13 pm Imam Toufique < >> [email protected] >> > <mailto:[email protected]>> wrote: >> > >> > so I almost got this to work, based on the >> situation I am >> > in. To elaborate just a bit, my setup involves a >> shibboleth >> > SP that I need to authenticate my application. >> Since I >> > can't set up the HA proxy node with shibboleth SP - >> I had to >> > wrap my application in the backend with apache so I >> can pass >> > REMOTE_USER to the application. the application I >> have is - >> > jupyterhub and it start with its own proxy. Long >> story >> > short, here is my current setup: >> > >> > frontend >> > bind :80 >> > bind :443 ssl crt /etc/haproxy/crsplab2_1.pem >> > stats uri /haproxy?stats >> > default_backend web1_cluster >> > option httplog >> > log global >> > #option dontlognull >> > log /dev/log local0 debug >> > mode http >> > option forwardfor # forward IP >> > http-request set-header X-Forwarded-Port >> %[dst_port] >> > http-request add-header X-Forwarded-Proto https >> if { ssl_fc } >> > redirect scheme https if !{ ssl_fc } >> > >> > acl host_web3 path_beg /jhub >> > use_backend web3_cluster if host_web3 >> > >> > backend >> > server web1.oit.uci.edu <http://web1.oit.uci.edu> >> > 128.110.80.5:80 <http://128.110.80.5:80> check >> > >> > this works for the most part. But I am confused >> with a >> > problem. when I get to my application, my backend >> IP address >> > shows up in the browser URL. >> > >> > for example, I see this in my browser: >> > >> > http://128.110.80.5/jhub/user/itoufiqu/tree? >> > >> > whereas, I was expecting that it would show the >> original >> > URL, such as: >> > >> > http://crsplab2.domain.com/jhub/user/itoufiqu/tree >> > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>? >> ( >> > where crsplab2.domain.com < >> http://crsplab2.domain.com> is >> > the URL to get HAproxy ) >> > >> > >> > You need to tell your backend app that it runs behind >> reverse >> > proxy with ssl termination and that it's domain/url >> > is https://crsplab2.domain.com >> > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>. >> How you do >> > that depends on the backend app you are using but most >> of them >> > like apache2, tomcat etc. have specific configs that >> you can >> > find in their documentation. For example if your >> backend is >> > apache2 I bet you don't have the DomainName set in the >> config in >> > which case it defaults to the host ip address. >> > >> > >> > You can also try: >> > >> > rspirep ^Location:\ http://(.*):80(.*) Location:\ >> > https://crsplab2.domain.com >> > <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2 >> if { >> > ssl_fc } >> > >> > to fix the URL but note that this will not save you from >> hard coded >> > url's in the returned html pages the way apache does. >> > >> > >> > >> > While I am no expert in HA proxy world, I think >> this might >> > due to the fact that my backend does not have SSL >> and >> > HAproxy frontend does have SSL. At this point, I >> would >> > avoid that IP address showing up in the browser. >> what is >> > the best way to accomplish this? >> > >> > thanks for your continues help! >> > >> > >> > >> > >> > >> > >> > On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic >> > <[email protected] <mailto:[email protected]>> >> wrote: >> > >> > Hi. >> > >> > Am 23.10.2018 um 09:04 schrieb Imam Toufique: >> > > I am looking for some help on how to write the >> > following apache proxypass rules >> > > in HAproxy. Not to mention I am at a bit of >> loss with >> > my first try :-) . Here >> > > are my current proxypass rules: >> > > >> > > ProxyPass http://10.1.100.156:8000/jhub >> > > ProxyPassReverse >> http://10.1.100.156:8000/jhub >> > >> > Well ProxyPass and ProxyPassReverse do a lot of >> thinks >> > not just rewrites, as >> > mentioned in the doc >> > >> > >> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass >> > >> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse >> > >> > >> > > <LocationMatch >> > >> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >> > > ProxyPassMatch ws:// >> 10.1.100.156:8000/jhub/$1/$2$3 >> > <http://10.1.100.156:8000/jhub/$1/$2$3> >> > > ProxyPassReverse >> > ws://10.1.100.156:8000/jhub/$1/$2$3 >> > <http://10.1.100.156:8000/jhub/$1/$2$3> >> > > </LocationMatch> >> > > >> > > As I am not well versed in the massive HAproxy >> > configuration guide, if any of >> > > you can give me a hand with this, I would >> very much >> > appreciate it. >> > >> > I'm also not "that" expert but I would try the >> > following, untested. >> > >> > ### >> > defaults >> > mode http >> > log global >> > >> > #... maybe some other settings >> > timeout tunnel 10h >> > >> > frontend https_001 >> > >> > #... maybe some other settings >> > >> > acl websocket path_beg /jhub >> > >> > #... maybe some other acls >> > >> > use_backend websocket_001 if websocket >> > >> > backend websocket_001 >> > >> > reqrep "^([^\ :]*) >> > >> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >> > "/jhub/\1/\2\3" >> > >> > # You will need to replace the first column >> with the >> > response from the >> > # backend response >> > # rspirep "^Location: >> > >> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >> > "Location: >> > /jhub/\1/\2\3" >> > # OR >> > # http-response replace-header Location >> > >> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >> > "/jhub/\1/\2\3" >> > >> > # add some checks >> > >> > server ws_01 10.1.100.156:8000 >> > <http://10.1.100.156:8000> check >> > ### >> > >> > Here are some links which may help you also. >> > >> > >> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ >> > >> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep >> > >> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep >> > >> > I would run haproxy in Debug mode and see how >> the >> > request pass haproxy and adopt >> > the config. >> > >> > It would be nice when you show us the working >> conf ;-) >> > >> > It would be nice to have a >> > >> > http-request replace-uri <match-regex> >> <replace-fmt> >> > >> > to replace the reqrep. >> > >> > > > -- > Regards, > *Imam Toufique* > *213-700-5485* > -- Regards, *Imam Toufique* *213-700-5485*
