Hi Igor, Thank you so much, I will definitely try your suggestions, but I am not sure how it will help my situation. shibboleth SP looks for, let's suppose, https://crsplabweb2.example.com/Shibboleth.sso - for it it's single sign-on. for apache or nginx to talk to the SP, SP needs to run in the same node ( as far as I know ). So, I am not sure how shibboleth will be able to communicate with the HAP for its SSO calls.
--imam On Sun, Oct 28, 2018 at 5:21 PM Igor Cicimov <[email protected]> wrote: > Hi Imam, > > On Sat, Oct 27, 2018 at 4:42 PM Imam Toufique <[email protected]> wrote: > >> Hi Igor, >> >> Thanks very much for offering to help! I will do this in sections, >> hopefully, I can keep this from being too cluttered. >> >> haproxy.cfg: >> >> -------------------------------------------------------------------------------------- >> global >> #log /dev/log local0 debug >> #log /dev/log local1 debug >> log 127.0.0.1 local2 >> chroot /var/lib/haproxy >> stats timeout 30s >> user haproxy >> group haproxy >> tune.ssl.default-dh-param 2048 >> daemon >> >> defaults >> log global >> mode http >> option tcplog >> option dontlognull >> timeout connect 5000 >> timeout client 50000 >> timeout server 50000 >> timeout tunnel 9h >> option tcp-check >> >> frontend http_front >> bind :80 >> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >> stats uri /haproxy?stats >> default_backend web1_cluster >> option httplog >> log global >> #option dontlognull >> log /dev/log local0 debug >> mode http >> option forwardfor # forward IP >> http-request set-header X-Forwarded-Port %[dst_port] >> http-request add-header X-Forwarded-Proto https if { ssl_fc } >> redirect scheme https if !{ ssl_fc } >> >> acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav >> use_backend webdav_cluster if host_web2 >> >> acl host_web3 path_beg /jhub >> use_backend web3_cluster if host_web3 >> >> >> backend webdav_cluster >> balance roundrobin >> server web1 10.1.100.156:8080 check inter 2000 cookie w1 >> server web2 10.1.100.160:8080 check inter 2000 cookie w2 >> >> backend web3_cluster >> server publicIP:443 check ssl verify none inter 2000 cookie w1 >> >> ----------------------------------------------------------------------------------------------------- >> Note: I have a single backend node, as it was easy to test with just one >> node, instead of making changes to 2 nodes at a time. >> >> Here is my apache config: >> >> in httpd.conf, only change I have made is ( the rest is a stock centos >> 7.5 httpd.conf ): >> ------------------------------------- >> ServerName 10.1.100.160:80 ( Internal IP of the backend node) >> Redirect permanent /jhub https://crsplabweb1.domain.com/jhub >> ------------------------------------- >> >> in my ssl.conf, where I access the jupyterhub instance running in >> 127.0.0.1:8000 . Also, note that the backend is running shibboleth SP. >> One of the issues I encountered is, If I did not have SSL , i was getting a >> browser warning for not having SSL. >> >> Here is my ssl.conf: >> >> -------------------------------------------------------------------------- >> Listen 443 https >> SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog >> SSLSessionCache shmcb:/run/httpd/sslcache(512000) >> SSLSessionCacheTimeout 300 >> SSLRandomSeed startup file:/dev/urandom 256 >> SSLRandomSeed connect builtin >> SSLCryptoDevice builtin >> >> <VirtualHost _default_:443> >> >> UseCanonicalName on >> ServerName crsplabweb1.domain.com:443 >> >> ErrorLog logs/ssl_error_log >> TransferLog logs/ssl_access_log >> LogLevel warn >> >> SSLEngine on >> >> SSLProtocol all -SSLv2 -SSLv3 >> SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA >> SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer >> SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key >> SSLCertificateChainFile >> /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c >> >> <Files ~ "\.(cgi|shtml|phtml|php3?)$"> >> SSLOptions +StdEnvVars >> </Files> >> <Directory "/var/www/cgi-bin"> >> SSLOptions +StdEnvVars >> </Directory> >> >> <Location /jhub> >> ProxyPass http://127.0.0.1:8000/jhub >> ProxyPassReverse http://127.0.0.1:8000/jhub >> RequestHeader unset Accept-Encoding >> ProxyPreserveHost on >> AuthType shibboleth >> ShibRequestSetting requireSession 1 >> Require shibboleth >> ShibUseHeaders On >> ShibBasicHijack On >> RewriteEngine On >> RequestHeader set X-Remote-User %{REMOTE_USER}s >> </Location> >> >> <LocationMatch >> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >> ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3 >> ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3 >> </LocationMatch> >> >> BrowserMatch "MSIE [2-5]" \ >> nokeepalive ssl-unclean-shutdown \ >> downgrade-1.0 force-response-1.0 >> >> CustomLog logs/ssl_request_log \ >> "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" >> </VirtualHost> >> >> ---------------------------------------------------------------------------------- >> >> Thanks >> > > Your problem is that you are not using the Forwarded headers set by HAP in > Apache thus you get http response instead ssl. > > First for haproxy create a directory where you will keep all your SSL > certs, lets say /etc/haproxy/ssl.d/, and put the crsplab2.oit.uci.edu and > crsplabweb1.domain.com certificates inside. More details on setting SSL > certificates in Haproxy can be found here: > https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crt > > The config will then look something like this: > > frontend http_front > bind *:80 > bind *:443 ssl crt /etc/haproxy/ssl.d/ no-sslv3 no-tls-tickets ... > > backend web3_cluster > server shibboleth1 10.1.100.160:80 check inter 2000 > > On the apache side remove the ssl settings (since now HAP will be > terminating SSL) and set a SSL redirect, something like this: > > <VirtualHost *:80> > ServerName crsplabweb1.domain.com > ServerAlias www.crsplabweb1.domain.com > > SetEnvIfNoCase X-Forwarded-Proto https HTTPS=on > # Insure the pages requested over ssl are always over ssl > RewriteEngine On > RewriteCond %{HTTP_X_Forwarded_Proto} ^https$ > RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R,L] > ... > </VirtualHost> > Let me know if any further questions. > > >> On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov < >> [email protected]> wrote: >> >>> Hi Imam, >>> >>> On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I came up with the following config, things seem to be working now, for >>>> the most part. >>>> >>>> frontend http_front >>>> bind :80 >>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>>> stats uri /haproxy?stats >>>> default_backend web1_cluster >>>> option httplog >>>> log global >>>> #option dontlognull >>>> log /dev/log local0 debug >>>> mode http >>>> option forwardfor # forward IP >>>> http-request set-header X-Forwarded-Port %[dst_port] >>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>> redirect scheme https if !{ ssl_fc } >>>> acl host_web3 path_beg /jhub >>>> use_backend web3_cluster if host_web3 >>>> >>>> web3_cluster >>>> >>>> backend web3_cluster >>>> mode http >>>> balance source >>>> server crsplabweb1.domain.com publicIP:443 check ssl verify none >>>> inter 2000 cookie w1 >>>> >>>> The above config gets me to the backend node -- where I have a >>>> jupyterhub instance running + . Shibboleth SP running for authentication. >>>> As I could not get shibboleth SP to work by staying in my private network, >>>> I had to set up a public IP for the backend node, get SSL certs - so >>>> shibboleth authentication could be done. I am sure there is a better >>>> approach to this, but I don't know what it is. I will be trying out SNAT >>>> to see if that will allow me to keep using my private IP for the backend >>>> nodes. If any of you know how to do SNAT, please chime in, it would be >>>> worth the time/effort to try it out. >>>> >>>> Now, the interesting thing I have noticed with the above setup -- when >>>> I connect to HAProxy, let's say with https://proxy.domain.com , I >>>> authenticate with shibboleth, and then the URL in the browser points to the >>>> backend node. >>>> >>>> For example: >>>> >>>> my proxy address: https://proxy.domain.com/jhub >>>> >>>> after I connect to the backend, the URL turns into - >>>> https://crsplabweb1.domain.com/jhub/tree? >>>> >>>> ...and everything works thereafter. >>>> >>>> I tried the rewrite method that Igor has suggested before, that did not >>>> make any difference. But what I noticed is, after I connect, no traffic go >>>> through the proxy anymore, my client ( i.e. laptop) connects directly to >>>> the backend server. Not sure if this good or bad though (?) , but, I am not >>>> sure how to configure this so that I will go through a proxy but still be >>>> connected in the backend via a private IP and I can ( still ) authenticate >>>> via shibboleth. >>>> >>>> So, when I change the 'web3_cluster' backend to : >>>> >>>> server crsplabweb1 privateIP:80 inter 2000 cookie w1 >>>> >>>> and, I set backend apache to accept connection on port 80, then I break >>>> shibboleth authentication. >>>> >>>> Any inputs here? >>>> >>>> thanks, guys! >>>> >>>> >>> I think it is time for you to provide the full HAP and Apache configs so >>> we can see what is going on (please obfuscate any sensitive data). Also the >>> use of the "cookie w1" is not clear since you are not setting it in HAP >>> and is kinda redundant for single backend setup. >>> >>> >>>> >>>> On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov < >>>> [email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Thu, 25 Oct 2018 6:13 pm Imam Toufique <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> so I almost got this to work, based on the situation I am in. To >>>>>>> elaborate just a bit, my setup involves a shibboleth SP that I need to >>>>>>> authenticate my application. Since I can't set up the HA proxy node >>>>>>> with >>>>>>> shibboleth SP - I had to wrap my application in the backend with apache >>>>>>> so >>>>>>> I can pass REMOTE_USER to the application. the application I have is - >>>>>>> jupyterhub and it start with its own proxy. Long story short, here is >>>>>>> my >>>>>>> current setup: >>>>>>> >>>>>>> frontend >>>>>>> bind :80 >>>>>>> bind :443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>>> stats uri /haproxy?stats >>>>>>> default_backend web1_cluster >>>>>>> option httplog >>>>>>> log global >>>>>>> #option dontlognull >>>>>>> log /dev/log local0 debug >>>>>>> mode http >>>>>>> option forwardfor # forward IP >>>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>>> redirect scheme https if !{ ssl_fc } >>>>>>> >>>>>>> acl host_web3 path_beg /jhub >>>>>>> use_backend web3_cluster if host_web3 >>>>>>> >>>>>>> backend >>>>>>> server web1.oit.uci.edu 128.110.80.5:80 check >>>>>>> >>>>>>> this works for the most part. But I am confused with a problem. >>>>>>> when I get to my application, my backend IP address shows up in the >>>>>>> browser >>>>>>> URL. >>>>>>> >>>>>>> for example, I see this in my browser: >>>>>>> >>>>>>> http://128.110.80.5/jhub/user/itoufiqu/tree? >>>>>>> >>>>>>> whereas, I was expecting that it would show the original URL, such >>>>>>> as: >>>>>>> >>>>>>> http://crsplab2.domain.com/jhub/user/itoufiqu/tree? ( where >>>>>>> crsplab2.domain.com is the URL to get HAproxy ) >>>>>>> >>>>>> >>>>>> You need to tell your backend app that it runs behind reverse proxy >>>>>> with ssl termination and that it's domain/url is >>>>>> https://crsplab2.domain.com >>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>. How you do >>>>>> that depends on the backend app you are using but most of them like >>>>>> apache2, tomcat etc. have specific configs that you can find in their >>>>>> documentation. For example if your backend is apache2 I bet you don't >>>>>> have >>>>>> the DomainName set in the config in which case it defaults to the host ip >>>>>> address. >>>>>> >>>>> >>>>> You can also try: >>>>> >>>>> rspirep ^Location:\ http://(.*):80(.*) Location:\ https:// >>>>> crsplab2.domain.com >>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2 if { >>>>> ssl_fc } >>>>> >>>>> to fix the URL but note that this will not save you from hard coded >>>>> url's in the returned html pages the way apache does. >>>>> >>>>> >>>>>> >>>>>>> While I am no expert in HA proxy world, I think this might due to >>>>>>> the fact that my backend does not have SSL and HAproxy frontend does >>>>>>> have >>>>>>> SSL. At this point, I would avoid that IP address showing up in the >>>>>>> browser. what is the best way to accomplish this? >>>>>>> >>>>>>> thanks for your continues help! >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi. >>>>>>>> >>>>>>>> Am 23.10.2018 um 09:04 schrieb Imam Toufique: >>>>>>>> > I am looking for some help on how to write the following apache >>>>>>>> proxypass rules >>>>>>>> > in HAproxy. Not to mention I am at a bit of loss with my first >>>>>>>> try :-) . Here >>>>>>>> > are my current proxypass rules: >>>>>>>> > >>>>>>>> > ProxyPass http://10.1.100.156:8000/jhub >>>>>>>> > ProxyPassReverse http://10.1.100.156:8000/jhub >>>>>>>> >>>>>>>> Well ProxyPass and ProxyPassReverse do a lot of thinks not just >>>>>>>> rewrites, as >>>>>>>> mentioned in the doc >>>>>>>> >>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass >>>>>>>> >>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse >>>>>>>> >>>>>>>> >>>>>>>> > <LocationMatch >>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>>>>>>> > ProxyPassMatch ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>> > ProxyPassReverse ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>> > </LocationMatch> >>>>>>>> > >>>>>>>> > As I am not well versed in the massive HAproxy configuration >>>>>>>> guide, if any of >>>>>>>> > you can give me a hand with this, I would very much appreciate it. >>>>>>>> >>>>>>>> I'm also not "that" expert but I would try the following, untested. >>>>>>>> >>>>>>>> ### >>>>>>>> defaults >>>>>>>> mode http >>>>>>>> log global >>>>>>>> >>>>>>>> #... maybe some other settings >>>>>>>> timeout tunnel 10h >>>>>>>> >>>>>>>> frontend https_001 >>>>>>>> >>>>>>>> #... maybe some other settings >>>>>>>> >>>>>>>> acl websocket path_beg /jhub >>>>>>>> >>>>>>>> #... maybe some other acls >>>>>>>> >>>>>>>> use_backend websocket_001 if websocket >>>>>>>> >>>>>>>> backend websocket_001 >>>>>>>> >>>>>>>> reqrep "^([^\ :]*) >>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>> "/jhub/\1/\2\3" >>>>>>>> >>>>>>>> # You will need to replace the first column with the response >>>>>>>> from the >>>>>>>> # backend response >>>>>>>> # rspirep "^Location: >>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>> "Location: >>>>>>>> /jhub/\1/\2\3" >>>>>>>> # OR >>>>>>>> # http-response replace-header Location >>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>> "/jhub/\1/\2\3" >>>>>>>> >>>>>>>> # add some checks >>>>>>>> >>>>>>>> server ws_01 10.1.100.156:8000 check >>>>>>>> ### >>>>>>>> >>>>>>>> Here are some links which may help you also. >>>>>>>> >>>>>>>> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ >>>>>>>> >>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep >>>>>>>> >>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep >>>>>>>> >>>>>>>> I would run haproxy in Debug mode and see how the request pass >>>>>>>> haproxy and adopt >>>>>>>> the config. >>>>>>>> >>>>>>>> It would be nice when you show us the working conf ;-) >>>>>>>> >>>>>>>> It would be nice to have a >>>>>>>> >>>>>>>> http-request replace-uri <match-regex> <replace-fmt> >>>>>>>> >>>>>>>> to replace the reqrep. >>>>>>>> >>>>>>>> > thanks >>>>>>>> >>>>>>>> Hth >>>>>>>> Aleks >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> *Imam Toufique* >>>>>>> *213-700-5485* >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> Igor Cicimov | DevOps >>>>> >>>>> >>>>> p. +61 (0) 433 078 728 >>>>> e. [email protected] <http://encompasscorporation.com/> >>>>> w*.* www.encompasscorporation.com >>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> *Imam Toufique* >>>> *213-700-5485* >>>> >>> >>> >>> -- >>> Igor Cicimov | DevOps >>> >>> >>> p. +61 (0) 433 078 728 >>> e. [email protected] <http://encompasscorporation.com/> >>> w*.* www.encompasscorporation.com >>> a. Level 4, 65 York Street, Sydney 2000 >>> >> >> >> -- >> Regards, >> *Imam Toufique* >> *213-700-5485* >> > > > -- > Igor Cicimov | DevOps > > > p. +61 (0) 433 078 728 > e. [email protected] <http://encompasscorporation.com/> > w*.* www.encompasscorporation.com > a. Level 4, 65 York Street, Sydney 2000 > -- Regards, *Imam Toufique* *213-700-5485*
