Well you need to point crsplabweb2.example.com to the haproxy IP that's the whole point of it running behind a proxy. Or am I missing something?
On Mon, Oct 29, 2018 at 1:28 PM Imam Toufique <[email protected]> wrote: > Hi Igor, > > Thank you so much, I will definitely try your suggestions, but I am not > sure how it will help my situation. shibboleth SP looks for, let's > suppose, https://crsplabweb2.example.com/Shibboleth.sso - for it it's > single sign-on. for apache or nginx to talk to the SP, SP needs to run in > the same node ( as far as I know ). So, I am not sure how shibboleth will > be able to communicate with the HAP for its SSO calls. > > --imam > > > > On Sun, Oct 28, 2018 at 5:21 PM Igor Cicimov < > [email protected]> wrote: > >> Hi Imam, >> >> On Sat, Oct 27, 2018 at 4:42 PM Imam Toufique <[email protected]> >> wrote: >> >>> Hi Igor, >>> >>> Thanks very much for offering to help! I will do this in sections, >>> hopefully, I can keep this from being too cluttered. >>> >>> haproxy.cfg: >>> >>> -------------------------------------------------------------------------------------- >>> global >>> #log /dev/log local0 debug >>> #log /dev/log local1 debug >>> log 127.0.0.1 local2 >>> chroot /var/lib/haproxy >>> stats timeout 30s >>> user haproxy >>> group haproxy >>> tune.ssl.default-dh-param 2048 >>> daemon >>> >>> defaults >>> log global >>> mode http >>> option tcplog >>> option dontlognull >>> timeout connect 5000 >>> timeout client 50000 >>> timeout server 50000 >>> timeout tunnel 9h >>> option tcp-check >>> >>> frontend http_front >>> bind :80 >>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>> stats uri /haproxy?stats >>> default_backend web1_cluster >>> option httplog >>> log global >>> #option dontlognull >>> log /dev/log local0 debug >>> mode http >>> option forwardfor # forward IP >>> http-request set-header X-Forwarded-Port %[dst_port] >>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>> redirect scheme https if !{ ssl_fc } >>> >>> acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav >>> use_backend webdav_cluster if host_web2 >>> >>> acl host_web3 path_beg /jhub >>> use_backend web3_cluster if host_web3 >>> >>> >>> backend webdav_cluster >>> balance roundrobin >>> server web1 10.1.100.156:8080 check inter 2000 cookie w1 >>> server web2 10.1.100.160:8080 check inter 2000 cookie w2 >>> >>> backend web3_cluster >>> server publicIP:443 check ssl verify none inter 2000 cookie w1 >>> >>> ----------------------------------------------------------------------------------------------------- >>> Note: I have a single backend node, as it was easy to test with just one >>> node, instead of making changes to 2 nodes at a time. >>> >>> Here is my apache config: >>> >>> in httpd.conf, only change I have made is ( the rest is a stock centos >>> 7.5 httpd.conf ): >>> ------------------------------------- >>> ServerName 10.1.100.160:80 ( Internal IP of the backend node) >>> Redirect permanent /jhub https://crsplabweb1.domain.com/jhub >>> ------------------------------------- >>> >>> in my ssl.conf, where I access the jupyterhub instance running in >>> 127.0.0.1:8000 . Also, note that the backend is running shibboleth >>> SP. One of the issues I encountered is, If I did not have SSL , i was >>> getting a browser warning for not having SSL. >>> >>> Here is my ssl.conf: >>> >>> >>> -------------------------------------------------------------------------- >>> Listen 443 https >>> SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog >>> SSLSessionCache shmcb:/run/httpd/sslcache(512000) >>> SSLSessionCacheTimeout 300 >>> SSLRandomSeed startup file:/dev/urandom 256 >>> SSLRandomSeed connect builtin >>> SSLCryptoDevice builtin >>> >>> <VirtualHost _default_:443> >>> >>> UseCanonicalName on >>> ServerName crsplabweb1.domain.com:443 >>> >>> ErrorLog logs/ssl_error_log >>> TransferLog logs/ssl_access_log >>> LogLevel warn >>> >>> SSLEngine on >>> >>> SSLProtocol all -SSLv2 -SSLv3 >>> SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA >>> SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer >>> SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key >>> SSLCertificateChainFile >>> /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c >>> >>> <Files ~ "\.(cgi|shtml|phtml|php3?)$"> >>> SSLOptions +StdEnvVars >>> </Files> >>> <Directory "/var/www/cgi-bin"> >>> SSLOptions +StdEnvVars >>> </Directory> >>> >>> <Location /jhub> >>> ProxyPass http://127.0.0.1:8000/jhub >>> ProxyPassReverse http://127.0.0.1:8000/jhub >>> RequestHeader unset Accept-Encoding >>> ProxyPreserveHost on >>> AuthType shibboleth >>> ShibRequestSetting requireSession 1 >>> Require shibboleth >>> ShibUseHeaders On >>> ShibBasicHijack On >>> RewriteEngine On >>> RequestHeader set X-Remote-User %{REMOTE_USER}s >>> </Location> >>> >>> <LocationMatch >>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>> ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3 >>> ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3 >>> </LocationMatch> >>> >>> BrowserMatch "MSIE [2-5]" \ >>> nokeepalive ssl-unclean-shutdown \ >>> downgrade-1.0 force-response-1.0 >>> >>> CustomLog logs/ssl_request_log \ >>> "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" >>> </VirtualHost> >>> >>> ---------------------------------------------------------------------------------- >>> >>> Thanks >>> >> >> Your problem is that you are not using the Forwarded headers set by HAP >> in Apache thus you get http response instead ssl. >> >> First for haproxy create a directory where you will keep all your SSL >> certs, lets say /etc/haproxy/ssl.d/, and put the crsplab2.oit.uci.edu >> and crsplabweb1.domain.com certificates inside. More details on setting >> SSL certificates in Haproxy can be found here: >> https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crt >> >> The config will then look something like this: >> >> frontend http_front >> bind *:80 >> bind *:443 ssl crt /etc/haproxy/ssl.d/ no-sslv3 no-tls-tickets ... >> >> backend web3_cluster >> server shibboleth1 10.1.100.160:80 check inter 2000 >> >> On the apache side remove the ssl settings (since now HAP will be >> terminating SSL) and set a SSL redirect, something like this: >> >> <VirtualHost *:80> >> ServerName crsplabweb1.domain.com >> ServerAlias www.crsplabweb1.domain.com >> >> SetEnvIfNoCase X-Forwarded-Proto https HTTPS=on >> # Insure the pages requested over ssl are always over ssl >> RewriteEngine On >> RewriteCond %{HTTP_X_Forwarded_Proto} ^https$ >> RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R,L] >> ... >> </VirtualHost> >> Let me know if any further questions. >> >> >>> On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov < >>> [email protected]> wrote: >>> >>>> Hi Imam, >>>> >>>> On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I came up with the following config, things seem to be working now, >>>>> for the most part. >>>>> >>>>> frontend http_front >>>>> bind :80 >>>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>> stats uri /haproxy?stats >>>>> default_backend web1_cluster >>>>> option httplog >>>>> log global >>>>> #option dontlognull >>>>> log /dev/log local0 debug >>>>> mode http >>>>> option forwardfor # forward IP >>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>> redirect scheme https if !{ ssl_fc } >>>>> acl host_web3 path_beg /jhub >>>>> use_backend web3_cluster if host_web3 >>>>> >>>>> web3_cluster >>>>> >>>>> backend web3_cluster >>>>> mode http >>>>> balance source >>>>> server crsplabweb1.domain.com publicIP:443 check ssl verify none >>>>> inter 2000 cookie w1 >>>>> >>>>> The above config gets me to the backend node -- where I have a >>>>> jupyterhub instance running + . Shibboleth SP running for authentication. >>>>> As I could not get shibboleth SP to work by staying in my private network, >>>>> I had to set up a public IP for the backend node, get SSL certs - so >>>>> shibboleth authentication could be done. I am sure there is a better >>>>> approach to this, but I don't know what it is. I will be trying out SNAT >>>>> to see if that will allow me to keep using my private IP for the backend >>>>> nodes. If any of you know how to do SNAT, please chime in, it would be >>>>> worth the time/effort to try it out. >>>>> >>>>> Now, the interesting thing I have noticed with the above setup -- when >>>>> I connect to HAProxy, let's say with https://proxy.domain.com , I >>>>> authenticate with shibboleth, and then the URL in the browser points to >>>>> the >>>>> backend node. >>>>> >>>>> For example: >>>>> >>>>> my proxy address: https://proxy.domain.com/jhub >>>>> >>>>> after I connect to the backend, the URL turns into - >>>>> https://crsplabweb1.domain.com/jhub/tree? >>>>> >>>>> ...and everything works thereafter. >>>>> >>>>> I tried the rewrite method that Igor has suggested before, that did >>>>> not make any difference. But what I noticed is, after I connect, no >>>>> traffic go through the proxy anymore, my client ( i.e. laptop) connects >>>>> directly to the backend server. Not sure if this good or bad though (?) , >>>>> but, I am not sure how to configure this so that I will go through a >>>>> proxy but still be connected in the backend via a private IP and I can ( >>>>> still ) authenticate via shibboleth. >>>>> >>>>> So, when I change the 'web3_cluster' backend to : >>>>> >>>>> server crsplabweb1 privateIP:80 inter 2000 cookie w1 >>>>> >>>>> and, I set backend apache to accept connection on port 80, then I >>>>> break shibboleth authentication. >>>>> >>>>> Any inputs here? >>>>> >>>>> thanks, guys! >>>>> >>>>> >>>> I think it is time for you to provide the full HAP and Apache configs >>>> so we can see what is going on (please obfuscate any sensitive data). Also >>>> the use of the "cookie w1" is not clear since you are not setting it >>>> in HAP and is kinda redundant for single backend setup. >>>> >>>> >>>>> >>>>> On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, 25 Oct 2018 6:13 pm Imam Toufique <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> so I almost got this to work, based on the situation I am in. To >>>>>>>> elaborate just a bit, my setup involves a shibboleth SP that I need to >>>>>>>> authenticate my application. Since I can't set up the HA proxy node >>>>>>>> with >>>>>>>> shibboleth SP - I had to wrap my application in the backend with >>>>>>>> apache so >>>>>>>> I can pass REMOTE_USER to the application. the application I have is - >>>>>>>> jupyterhub and it start with its own proxy. Long story short, here is >>>>>>>> my >>>>>>>> current setup: >>>>>>>> >>>>>>>> frontend >>>>>>>> bind :80 >>>>>>>> bind :443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>>>> stats uri /haproxy?stats >>>>>>>> default_backend web1_cluster >>>>>>>> option httplog >>>>>>>> log global >>>>>>>> #option dontlognull >>>>>>>> log /dev/log local0 debug >>>>>>>> mode http >>>>>>>> option forwardfor # forward IP >>>>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>>>> redirect scheme https if !{ ssl_fc } >>>>>>>> >>>>>>>> acl host_web3 path_beg /jhub >>>>>>>> use_backend web3_cluster if host_web3 >>>>>>>> >>>>>>>> backend >>>>>>>> server web1.oit.uci.edu 128.110.80.5:80 check >>>>>>>> >>>>>>>> this works for the most part. But I am confused with a problem. >>>>>>>> when I get to my application, my backend IP address shows up in the >>>>>>>> browser >>>>>>>> URL. >>>>>>>> >>>>>>>> for example, I see this in my browser: >>>>>>>> >>>>>>>> http://128.110.80.5/jhub/user/itoufiqu/tree? >>>>>>>> >>>>>>>> whereas, I was expecting that it would show the original URL, such >>>>>>>> as: >>>>>>>> >>>>>>>> http://crsplab2.domain.com/jhub/user/itoufiqu/tree? ( where >>>>>>>> crsplab2.domain.com is the URL to get HAproxy ) >>>>>>>> >>>>>>> >>>>>>> You need to tell your backend app that it runs behind reverse proxy >>>>>>> with ssl termination and that it's domain/url is >>>>>>> https://crsplab2.domain.com >>>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>. How you do >>>>>>> that depends on the backend app you are using but most of them like >>>>>>> apache2, tomcat etc. have specific configs that you can find in their >>>>>>> documentation. For example if your backend is apache2 I bet you don't >>>>>>> have >>>>>>> the DomainName set in the config in which case it defaults to the host >>>>>>> ip >>>>>>> address. >>>>>>> >>>>>> >>>>>> You can also try: >>>>>> >>>>>> rspirep ^Location:\ http://(.*):80(.*) Location:\ https:// >>>>>> crsplab2.domain.com >>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2 if { >>>>>> ssl_fc } >>>>>> >>>>>> to fix the URL but note that this will not save you from hard coded >>>>>> url's in the returned html pages the way apache does. >>>>>> >>>>>> >>>>>>> >>>>>>>> While I am no expert in HA proxy world, I think this might due to >>>>>>>> the fact that my backend does not have SSL and HAproxy frontend does >>>>>>>> have >>>>>>>> SSL. At this point, I would avoid that IP address showing up in the >>>>>>>> browser. what is the best way to accomplish this? >>>>>>>> >>>>>>>> thanks for your continues help! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi. >>>>>>>>> >>>>>>>>> Am 23.10.2018 um 09:04 schrieb Imam Toufique: >>>>>>>>> > I am looking for some help on how to write the following apache >>>>>>>>> proxypass rules >>>>>>>>> > in HAproxy. Not to mention I am at a bit of loss with my first >>>>>>>>> try :-) . Here >>>>>>>>> > are my current proxypass rules: >>>>>>>>> > >>>>>>>>> > ProxyPass http://10.1.100.156:8000/jhub >>>>>>>>> > ProxyPassReverse http://10.1.100.156:8000/jhub >>>>>>>>> >>>>>>>>> Well ProxyPass and ProxyPassReverse do a lot of thinks not just >>>>>>>>> rewrites, as >>>>>>>>> mentioned in the doc >>>>>>>>> >>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass >>>>>>>>> >>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse >>>>>>>>> >>>>>>>>> >>>>>>>>> > <LocationMatch >>>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>>>>>>>> > ProxyPassMatch ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>>> > ProxyPassReverse ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>>> > </LocationMatch> >>>>>>>>> > >>>>>>>>> > As I am not well versed in the massive HAproxy configuration >>>>>>>>> guide, if any of >>>>>>>>> > you can give me a hand with this, I would very much appreciate >>>>>>>>> it. >>>>>>>>> >>>>>>>>> I'm also not "that" expert but I would try the following, untested. >>>>>>>>> >>>>>>>>> ### >>>>>>>>> defaults >>>>>>>>> mode http >>>>>>>>> log global >>>>>>>>> >>>>>>>>> #... maybe some other settings >>>>>>>>> timeout tunnel 10h >>>>>>>>> >>>>>>>>> frontend https_001 >>>>>>>>> >>>>>>>>> #... maybe some other settings >>>>>>>>> >>>>>>>>> acl websocket path_beg /jhub >>>>>>>>> >>>>>>>>> #... maybe some other acls >>>>>>>>> >>>>>>>>> use_backend websocket_001 if websocket >>>>>>>>> >>>>>>>>> backend websocket_001 >>>>>>>>> >>>>>>>>> reqrep "^([^\ :]*) >>>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>> "/jhub/\1/\2\3" >>>>>>>>> >>>>>>>>> # You will need to replace the first column with the response >>>>>>>>> from the >>>>>>>>> # backend response >>>>>>>>> # rspirep "^Location: >>>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>> "Location: >>>>>>>>> /jhub/\1/\2\3" >>>>>>>>> # OR >>>>>>>>> # http-response replace-header Location >>>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>> "/jhub/\1/\2\3" >>>>>>>>> >>>>>>>>> # add some checks >>>>>>>>> >>>>>>>>> server ws_01 10.1.100.156:8000 check >>>>>>>>> ### >>>>>>>>> >>>>>>>>> Here are some links which may help you also. >>>>>>>>> >>>>>>>>> >>>>>>>>> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ >>>>>>>>> >>>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep >>>>>>>>> >>>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep >>>>>>>>> >>>>>>>>> I would run haproxy in Debug mode and see how the request pass >>>>>>>>> haproxy and adopt >>>>>>>>> the config. >>>>>>>>> >>>>>>>>> It would be nice when you show us the working conf ;-) >>>>>>>>> >>>>>>>>> It would be nice to have a >>>>>>>>> >>>>>>>>> http-request replace-uri <match-regex> <replace-fmt> >>>>>>>>> >>>>>>>>> to replace the reqrep. >>>>>>>>> >>>>>>>>> > thanks >>>>>>>>> >>>>>>>>> Hth >>>>>>>>> Aleks >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> *Imam Toufique* >>>>>>>> *213-700-5485* >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Igor Cicimov | DevOps >>>>>> >>>>>> >>>>>> p. +61 (0) 433 078 728 >>>>>> e. [email protected] <http://encompasscorporation.com/> >>>>>> w*.* www.encompasscorporation.com >>>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> *Imam Toufique* >>>>> *213-700-5485* >>>>> >>>> >>>> >>>> -- >>>> Igor Cicimov | DevOps >>>> >>>> >>>> p. +61 (0) 433 078 728 >>>> e. [email protected] <http://encompasscorporation.com/> >>>> w*.* www.encompasscorporation.com >>>> a. Level 4, 65 York Street, Sydney 2000 >>>> >>> >>> >>> -- >>> Regards, >>> *Imam Toufique* >>> *213-700-5485* >>> >> >> >> -- >> Igor Cicimov | DevOps >> >> >> p. +61 (0) 433 078 728 >> e. [email protected] <http://encompasscorporation.com/> >> w*.* www.encompasscorporation.com >> a. Level 4, 65 York Street, Sydney 2000 >> > > > -- > Regards, > *Imam Toufique* > *213-700-5485* > -- Igor Cicimov | DevOps p. +61 (0) 433 078 728 e. [email protected] <http://encompasscorporation.com/> w*.* www.encompasscorporation.com a. Level 4, 65 York Street, Sydney 2000
