" Well you need to point crsplabweb2.example.com to the haproxy IP that's the whole point of it running behind a proxy. Or am I missing something? "
Well, I am not sure what you meant by that comment above. On Sun, Oct 28, 2018 at 8:07 PM Igor Cicimov <[email protected]> wrote: > Well you need to point crsplabweb2.example.com to the haproxy IP that's > the whole point of it running behind a proxy. Or am I missing something? > > On Mon, Oct 29, 2018 at 1:28 PM Imam Toufique <[email protected]> wrote: > >> Hi Igor, >> >> Thank you so much, I will definitely try your suggestions, but I am not >> sure how it will help my situation. shibboleth SP looks for, let's >> suppose, https://crsplabweb2.example.com/Shibboleth.sso - for it it's >> single sign-on. for apache or nginx to talk to the SP, SP needs to run in >> the same node ( as far as I know ). So, I am not sure how shibboleth will >> be able to communicate with the HAP for its SSO calls. >> >> --imam >> >> >> >> On Sun, Oct 28, 2018 at 5:21 PM Igor Cicimov < >> [email protected]> wrote: >> >>> Hi Imam, >>> >>> On Sat, Oct 27, 2018 at 4:42 PM Imam Toufique <[email protected]> >>> wrote: >>> >>>> Hi Igor, >>>> >>>> Thanks very much for offering to help! I will do this in sections, >>>> hopefully, I can keep this from being too cluttered. >>>> >>>> haproxy.cfg: >>>> >>>> -------------------------------------------------------------------------------------- >>>> global >>>> #log /dev/log local0 debug >>>> #log /dev/log local1 debug >>>> log 127.0.0.1 local2 >>>> chroot /var/lib/haproxy >>>> stats timeout 30s >>>> user haproxy >>>> group haproxy >>>> tune.ssl.default-dh-param 2048 >>>> daemon >>>> >>>> defaults >>>> log global >>>> mode http >>>> option tcplog >>>> option dontlognull >>>> timeout connect 5000 >>>> timeout client 50000 >>>> timeout server 50000 >>>> timeout tunnel 9h >>>> option tcp-check >>>> >>>> frontend http_front >>>> bind :80 >>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>>> stats uri /haproxy?stats >>>> default_backend web1_cluster >>>> option httplog >>>> log global >>>> #option dontlognull >>>> log /dev/log local0 debug >>>> mode http >>>> option forwardfor # forward IP >>>> http-request set-header X-Forwarded-Port %[dst_port] >>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>> redirect scheme https if !{ ssl_fc } >>>> >>>> acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav >>>> use_backend webdav_cluster if host_web2 >>>> >>>> acl host_web3 path_beg /jhub >>>> use_backend web3_cluster if host_web3 >>>> >>>> >>>> backend webdav_cluster >>>> balance roundrobin >>>> server web1 10.1.100.156:8080 check inter 2000 cookie w1 >>>> server web2 10.1.100.160:8080 check inter 2000 cookie w2 >>>> >>>> backend web3_cluster >>>> server publicIP:443 check ssl verify none inter 2000 cookie w1 >>>> >>>> ----------------------------------------------------------------------------------------------------- >>>> Note: I have a single backend node, as it was easy to test with just >>>> one node, instead of making changes to 2 nodes at a time. >>>> >>>> Here is my apache config: >>>> >>>> in httpd.conf, only change I have made is ( the rest is a stock centos >>>> 7.5 httpd.conf ): >>>> ------------------------------------- >>>> ServerName 10.1.100.160:80 ( Internal IP of the backend node) >>>> Redirect permanent /jhub https://crsplabweb1.domain.com/jhub >>>> ------------------------------------- >>>> >>>> in my ssl.conf, where I access the jupyterhub instance running in >>>> 127.0.0.1:8000 . Also, note that the backend is running shibboleth >>>> SP. One of the issues I encountered is, If I did not have SSL , i was >>>> getting a browser warning for not having SSL. >>>> >>>> Here is my ssl.conf: >>>> >>>> >>>> -------------------------------------------------------------------------- >>>> Listen 443 https >>>> SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog >>>> SSLSessionCache shmcb:/run/httpd/sslcache(512000) >>>> SSLSessionCacheTimeout 300 >>>> SSLRandomSeed startup file:/dev/urandom 256 >>>> SSLRandomSeed connect builtin >>>> SSLCryptoDevice builtin >>>> >>>> <VirtualHost _default_:443> >>>> >>>> UseCanonicalName on >>>> ServerName crsplabweb1.domain.com:443 >>>> >>>> ErrorLog logs/ssl_error_log >>>> TransferLog logs/ssl_access_log >>>> LogLevel warn >>>> >>>> SSLEngine on >>>> >>>> SSLProtocol all -SSLv2 -SSLv3 >>>> SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA >>>> SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer >>>> SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key >>>> SSLCertificateChainFile >>>> /etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c >>>> >>>> <Files ~ "\.(cgi|shtml|phtml|php3?)$"> >>>> SSLOptions +StdEnvVars >>>> </Files> >>>> <Directory "/var/www/cgi-bin"> >>>> SSLOptions +StdEnvVars >>>> </Directory> >>>> >>>> <Location /jhub> >>>> ProxyPass http://127.0.0.1:8000/jhub >>>> ProxyPassReverse http://127.0.0.1:8000/jhub >>>> RequestHeader unset Accept-Encoding >>>> ProxyPreserveHost on >>>> AuthType shibboleth >>>> ShibRequestSetting requireSession 1 >>>> Require shibboleth >>>> ShibUseHeaders On >>>> ShibBasicHijack On >>>> RewriteEngine On >>>> RequestHeader set X-Remote-User %{REMOTE_USER}s >>>> </Location> >>>> >>>> <LocationMatch >>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>>> ProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3 >>>> ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3 >>>> </LocationMatch> >>>> >>>> BrowserMatch "MSIE [2-5]" \ >>>> nokeepalive ssl-unclean-shutdown \ >>>> downgrade-1.0 force-response-1.0 >>>> >>>> CustomLog logs/ssl_request_log \ >>>> "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" >>>> </VirtualHost> >>>> >>>> ---------------------------------------------------------------------------------- >>>> >>>> Thanks >>>> >>> >>> Your problem is that you are not using the Forwarded headers set by HAP >>> in Apache thus you get http response instead ssl. >>> >>> First for haproxy create a directory where you will keep all your SSL >>> certs, lets say /etc/haproxy/ssl.d/, and put the crsplab2.oit.uci.edu >>> and crsplabweb1.domain.com certificates inside. More details on setting >>> SSL certificates in Haproxy can be found here: >>> https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.1-crt >>> >>> The config will then look something like this: >>> >>> frontend http_front >>> bind *:80 >>> bind *:443 ssl crt /etc/haproxy/ssl.d/ no-sslv3 no-tls-tickets ... >>> >>> backend web3_cluster >>> server shibboleth1 10.1.100.160:80 check inter 2000 >>> >>> On the apache side remove the ssl settings (since now HAP will be >>> terminating SSL) and set a SSL redirect, something like this: >>> >>> <VirtualHost *:80> >>> ServerName crsplabweb1.domain.com >>> ServerAlias www.crsplabweb1.domain.com >>> >>> SetEnvIfNoCase X-Forwarded-Proto https HTTPS=on >>> # Insure the pages requested over ssl are always over ssl >>> RewriteEngine On >>> RewriteCond %{HTTP_X_Forwarded_Proto} ^https$ >>> RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R,L] >>> ... >>> </VirtualHost> >>> Let me know if any further questions. >>> >>> >>>> On Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov < >>>> [email protected]> wrote: >>>> >>>>> Hi Imam, >>>>> >>>>> On Sat, Oct 27, 2018 at 9:37 AM Imam Toufique <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I came up with the following config, things seem to be working now, >>>>>> for the most part. >>>>>> >>>>>> frontend http_front >>>>>> bind :80 >>>>>> bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>> stats uri /haproxy?stats >>>>>> default_backend web1_cluster >>>>>> option httplog >>>>>> log global >>>>>> #option dontlognull >>>>>> log /dev/log local0 debug >>>>>> mode http >>>>>> option forwardfor # forward IP >>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>> redirect scheme https if !{ ssl_fc } >>>>>> acl host_web3 path_beg /jhub >>>>>> use_backend web3_cluster if host_web3 >>>>>> >>>>>> web3_cluster >>>>>> >>>>>> backend web3_cluster >>>>>> mode http >>>>>> balance source >>>>>> server crsplabweb1.domain.com publicIP:443 check ssl verify none >>>>>> inter 2000 cookie w1 >>>>>> >>>>>> The above config gets me to the backend node -- where I have a >>>>>> jupyterhub instance running + . Shibboleth SP running for >>>>>> authentication. >>>>>> As I could not get shibboleth SP to work by staying in my private >>>>>> network, >>>>>> I had to set up a public IP for the backend node, get SSL certs - so >>>>>> shibboleth authentication could be done. I am sure there is a better >>>>>> approach to this, but I don't know what it is. I will be trying out SNAT >>>>>> to see if that will allow me to keep using my private IP for the backend >>>>>> nodes. If any of you know how to do SNAT, please chime in, it would be >>>>>> worth the time/effort to try it out. >>>>>> >>>>>> Now, the interesting thing I have noticed with the above setup -- >>>>>> when I connect to HAProxy, let's say with https://proxy.domain.com >>>>>> , I authenticate with shibboleth, and then the URL in the browser points >>>>>> to >>>>>> the backend node. >>>>>> >>>>>> For example: >>>>>> >>>>>> my proxy address: https://proxy.domain.com/jhub >>>>>> >>>>>> after I connect to the backend, the URL turns into - >>>>>> https://crsplabweb1.domain.com/jhub/tree? >>>>>> >>>>>> ...and everything works thereafter. >>>>>> >>>>>> I tried the rewrite method that Igor has suggested before, that did >>>>>> not make any difference. But what I noticed is, after I connect, no >>>>>> traffic go through the proxy anymore, my client ( i.e. laptop) connects >>>>>> directly to the backend server. Not sure if this good or bad though (?) , >>>>>> but, I am not sure how to configure this so that I will go through a >>>>>> proxy but still be connected in the backend via a private IP and I can ( >>>>>> still ) authenticate via shibboleth. >>>>>> >>>>>> So, when I change the 'web3_cluster' backend to : >>>>>> >>>>>> server crsplabweb1 privateIP:80 inter 2000 cookie w1 >>>>>> >>>>>> and, I set backend apache to accept connection on port 80, then I >>>>>> break shibboleth authentication. >>>>>> >>>>>> Any inputs here? >>>>>> >>>>>> thanks, guys! >>>>>> >>>>>> >>>>> I think it is time for you to provide the full HAP and Apache configs >>>>> so we can see what is going on (please obfuscate any sensitive data). Also >>>>> the use of the "cookie w1" is not clear since you are not setting it >>>>> in HAP and is kinda redundant for single backend setup. >>>>> >>>>> >>>>>> >>>>>> On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, 25 Oct 2018 6:13 pm Imam Toufique <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> so I almost got this to work, based on the situation I am in. To >>>>>>>>> elaborate just a bit, my setup involves a shibboleth SP that I need to >>>>>>>>> authenticate my application. Since I can't set up the HA proxy node >>>>>>>>> with >>>>>>>>> shibboleth SP - I had to wrap my application in the backend with >>>>>>>>> apache so >>>>>>>>> I can pass REMOTE_USER to the application. the application I have is >>>>>>>>> - >>>>>>>>> jupyterhub and it start with its own proxy. Long story short, here >>>>>>>>> is my >>>>>>>>> current setup: >>>>>>>>> >>>>>>>>> frontend >>>>>>>>> bind :80 >>>>>>>>> bind :443 ssl crt /etc/haproxy/crsplab2_1.pem >>>>>>>>> stats uri /haproxy?stats >>>>>>>>> default_backend web1_cluster >>>>>>>>> option httplog >>>>>>>>> log global >>>>>>>>> #option dontlognull >>>>>>>>> log /dev/log local0 debug >>>>>>>>> mode http >>>>>>>>> option forwardfor # forward IP >>>>>>>>> http-request set-header X-Forwarded-Port %[dst_port] >>>>>>>>> http-request add-header X-Forwarded-Proto https if { ssl_fc } >>>>>>>>> redirect scheme https if !{ ssl_fc } >>>>>>>>> >>>>>>>>> acl host_web3 path_beg /jhub >>>>>>>>> use_backend web3_cluster if host_web3 >>>>>>>>> >>>>>>>>> backend >>>>>>>>> server web1.oit.uci.edu 128.110.80.5:80 check >>>>>>>>> >>>>>>>>> this works for the most part. But I am confused with a problem. >>>>>>>>> when I get to my application, my backend IP address shows up in the >>>>>>>>> browser >>>>>>>>> URL. >>>>>>>>> >>>>>>>>> for example, I see this in my browser: >>>>>>>>> >>>>>>>>> http://128.110.80.5/jhub/user/itoufiqu/tree? >>>>>>>>> >>>>>>>>> whereas, I was expecting that it would show the original URL, such >>>>>>>>> as: >>>>>>>>> >>>>>>>>> http://crsplab2.domain.com/jhub/user/itoufiqu/tree? ( where >>>>>>>>> crsplab2.domain.com is the URL to get HAproxy ) >>>>>>>>> >>>>>>>> >>>>>>>> You need to tell your backend app that it runs behind reverse proxy >>>>>>>> with ssl termination and that it's domain/url is >>>>>>>> https://crsplab2.domain.com >>>>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>. How you do >>>>>>>> that depends on the backend app you are using but most of them like >>>>>>>> apache2, tomcat etc. have specific configs that you can find in their >>>>>>>> documentation. For example if your backend is apache2 I bet you don't >>>>>>>> have >>>>>>>> the DomainName set in the config in which case it defaults to the host >>>>>>>> ip >>>>>>>> address. >>>>>>>> >>>>>>> >>>>>>> You can also try: >>>>>>> >>>>>>> rspirep ^Location:\ http://(.*):80(.*) Location:\ https:// >>>>>>> crsplab2.domain.com >>>>>>> <http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2 if { >>>>>>> ssl_fc } >>>>>>> >>>>>>> to fix the URL but note that this will not save you from hard coded >>>>>>> url's in the returned html pages the way apache does. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> While I am no expert in HA proxy world, I think this might due to >>>>>>>>> the fact that my backend does not have SSL and HAproxy frontend does >>>>>>>>> have >>>>>>>>> SSL. At this point, I would avoid that IP address showing up in the >>>>>>>>> browser. what is the best way to accomplish this? >>>>>>>>> >>>>>>>>> thanks for your continues help! >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi. >>>>>>>>>> >>>>>>>>>> Am 23.10.2018 um 09:04 schrieb Imam Toufique: >>>>>>>>>> > I am looking for some help on how to write the following apache >>>>>>>>>> proxypass rules >>>>>>>>>> > in HAproxy. Not to mention I am at a bit of loss with my first >>>>>>>>>> try :-) . Here >>>>>>>>>> > are my current proxypass rules: >>>>>>>>>> > >>>>>>>>>> > ProxyPass http://10.1.100.156:8000/jhub >>>>>>>>>> > ProxyPassReverse http://10.1.100.156:8000/jhub >>>>>>>>>> >>>>>>>>>> Well ProxyPass and ProxyPassReverse do a lot of thinks not just >>>>>>>>>> rewrites, as >>>>>>>>>> mentioned in the doc >>>>>>>>>> >>>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass >>>>>>>>>> >>>>>>>>>> https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > <LocationMatch >>>>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"> >>>>>>>>>> > ProxyPassMatch ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>>>> > ProxyPassReverse ws://10.1.100.156:8000/jhub/$1/$2$3 >>>>>>>>>> > </LocationMatch> >>>>>>>>>> > >>>>>>>>>> > As I am not well versed in the massive HAproxy configuration >>>>>>>>>> guide, if any of >>>>>>>>>> > you can give me a hand with this, I would very much appreciate >>>>>>>>>> it. >>>>>>>>>> >>>>>>>>>> I'm also not "that" expert but I would try the following, >>>>>>>>>> untested. >>>>>>>>>> >>>>>>>>>> ### >>>>>>>>>> defaults >>>>>>>>>> mode http >>>>>>>>>> log global >>>>>>>>>> >>>>>>>>>> #... maybe some other settings >>>>>>>>>> timeout tunnel 10h >>>>>>>>>> >>>>>>>>>> frontend https_001 >>>>>>>>>> >>>>>>>>>> #... maybe some other settings >>>>>>>>>> >>>>>>>>>> acl websocket path_beg /jhub >>>>>>>>>> >>>>>>>>>> #... maybe some other acls >>>>>>>>>> >>>>>>>>>> use_backend websocket_001 if websocket >>>>>>>>>> >>>>>>>>>> backend websocket_001 >>>>>>>>>> >>>>>>>>>> reqrep "^([^\ :]*) >>>>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>> "/jhub/\1/\2\3" >>>>>>>>>> >>>>>>>>>> # You will need to replace the first column with the response >>>>>>>>>> from the >>>>>>>>>> # backend response >>>>>>>>>> # rspirep "^Location: >>>>>>>>>> /jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>> "Location: >>>>>>>>>> /jhub/\1/\2\3" >>>>>>>>>> # OR >>>>>>>>>> # http-response replace-header Location >>>>>>>>>> "/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)" >>>>>>>>>> "/jhub/\1/\2\3" >>>>>>>>>> >>>>>>>>>> # add some checks >>>>>>>>>> >>>>>>>>>> server ws_01 10.1.100.156:8000 check >>>>>>>>>> ### >>>>>>>>>> >>>>>>>>>> Here are some links which may help you also. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/ >>>>>>>>>> >>>>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep >>>>>>>>>> >>>>>>>>>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep >>>>>>>>>> >>>>>>>>>> I would run haproxy in Debug mode and see how the request pass >>>>>>>>>> haproxy and adopt >>>>>>>>>> the config. >>>>>>>>>> >>>>>>>>>> It would be nice when you show us the working conf ;-) >>>>>>>>>> >>>>>>>>>> It would be nice to have a >>>>>>>>>> >>>>>>>>>> http-request replace-uri <match-regex> <replace-fmt> >>>>>>>>>> >>>>>>>>>> to replace the reqrep. >>>>>>>>>> >>>>>>>>>> > thanks >>>>>>>>>> >>>>>>>>>> Hth >>>>>>>>>> Aleks >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> *Imam Toufique* >>>>>>>>> *213-700-5485* >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Igor Cicimov | DevOps >>>>>>> >>>>>>> >>>>>>> p. +61 (0) 433 078 728 >>>>>>> e. [email protected] <http://encompasscorporation.com/> >>>>>>> w*.* www.encompasscorporation.com >>>>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> *Imam Toufique* >>>>>> *213-700-5485* >>>>>> >>>>> >>>>> >>>>> -- >>>>> Igor Cicimov | DevOps >>>>> >>>>> >>>>> p. +61 (0) 433 078 728 >>>>> e. [email protected] <http://encompasscorporation.com/> >>>>> w*.* www.encompasscorporation.com >>>>> a. Level 4, 65 York Street, Sydney 2000 >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> *Imam Toufique* >>>> *213-700-5485* >>>> >>> >>> >>> -- >>> Igor Cicimov | DevOps >>> >>> >>> p. +61 (0) 433 078 728 >>> e. [email protected] <http://encompasscorporation.com/> >>> w*.* www.encompasscorporation.com >>> a. Level 4, 65 York Street, Sydney 2000 >>> >> >> >> -- >> Regards, >> *Imam Toufique* >> *213-700-5485* >> > > > -- > Igor Cicimov | DevOps > > > p. +61 (0) 433 078 728 > e. [email protected] <http://encompasscorporation.com/> > w*.* www.encompasscorporation.com > a. Level 4, 65 York Street, Sydney 2000 > -- Regards, *Imam Toufique* *213-700-5485*
