It looks like some reverse proxies send bad IP addresses in the X-Forwarded-For header. There is a proposed fix for this here: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5454
On Sat, Jul 23, 2016 at 8:23 PM, Stan Hu <sta...@gmail.com> wrote: > See https://gitlab.com/gitlab-org/gitlab-ce/issues/20194#note_13277376. > Feel free to comment there. > > On Sat, Jul 23, 2016 at 8:19 PM, James Σ <james.r.santi...@gmail.com> > wrote: > >> I'm getting the exact same behavior. Have gitlab behind an IIS reverse >> proxy (2012R2). After upgrading to 8.10.0 on ubuntu server 14.04 I get a >> 500 error and the same "app error: invalid address" in the unicorn stderr >> log. I've also tried modifying the trusted_proxy and >> real_ip_trusted_addresses configs without any luck. The gitlab-rake >> gitlab:check command shows no issues with my install. >> >> I wish you could tell what is sending an IP address, or what the address >> is that's failing the check. I assume it's maybe an empty string or a >> non-ip address that is getting passed to it. It's the very first entry in >> the unicorn stderr log so it's unclear what is driving the error. >> >> >> On Saturday, July 23, 2016 at 3:30:17 AM UTC+9, bcitech...@gmail.com >> wrote: >>> >>> Here are the parts of my *gitlab.rb* that are different from default: >>> >>> external_url 'https://git.x.com' >>> >>> nginx['listen_port'] = 8383 >>> nginx['listen_https'] = false >>> nginx['proxy_set_headers'] = { >>> "X-Forwarded-Proto" => "https", >>> "X-Forwarded-Ssl" => "on" >>> } >>> >>> These are done because the gitlab server is behind a reverse proxy IIS >>> box. These settings worked prior to 8.9.x. >>> >>> To your suggestion I looked up the trusted_proxies documentation and >>> found this page: >>> http://docs.gitlab.com/omnibus/settings/nginx.html#configuring-gitlab-trusted_proxies-and-the-nginx-real_ip-module >>> >>> I then added the IP address of the reverse proxy server to the nginx >>> trusted addresses but still get the same error. >>> >>> *nginx['real_ip_trusted_addresses'] = [ '10.1.0.185' ]* >>> *nginx['real_ip_header'] = 'X-Real-IP'* >>> *nginx['real_ip_recursive'] = 'on'* >>> >>> I tried adding the ipv6 address of the reverse proxy, as well as setting >>> them both in* gitlab_rails['trusted_proxies'] = []* >>> >>> Still getting the same error. >>> >>> On Friday, July 22, 2016 at 10:24:01 AM UTC-4, bcitech...@gmail.com >>> wrote: >>>> >>>> Getting the following error thrown when trying to view gitlab after >>>> updating to 8.10 >>>> >>>> ==> /var/log/gitlab/unicorn/current <== >>>> 2016-07-22_13:42:35.00512 starting new unicorn master >>>> 2016-07-22_13:43:25.87961 adopted new unicorn master 38005 >>>> 2016-07-22_13:46:34.70750 Received TERM from runit, sending to process >>>> group (-PID) >>>> 2016-07-22_13:46:34.71314 Terminated >>>> 2016-07-22_13:46:34.71315 forwarding TERM to unicorn master 38005 >>>> 2016-07-22_13:46:34.71315 wrapper for unicorn master 38005 exiting >>>> 2016-07-22_13:46:34.79448 adopted existing unicorn master 38005 >>>> 2016-07-22_13:46:35.79570 wrapper for unicorn master 38005 exiting >>>> 2016-07-22_13:46:35.80292 starting new unicorn master >>>> 2016-07-22_13:47:09.22842 adopted new unicorn master 38646 >>>> >>>> >>>> ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== >>>> E, [2016-07-22T09:51:24.020782 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in >>>> `each' >>>> E, [2016-07-22T09:51:24.020870 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in >>>> `call' >>>> E, [2016-07-22T09:51:24.020964 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:580:in >>>> `process_client' >>>> E, [2016-07-22T09:51:24.021052 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in >>>> `process_client' >>>> E, [2016-07-22T09:51:24.021146 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:674:in >>>> `worker_loop' >>>> E, [2016-07-22T09:51:24.021284 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:529:in >>>> `spawn_missing_workers' >>>> E, [2016-07-22T09:51:24.021414 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:140:in >>>> `start' >>>> E, [2016-07-22T09:51:24.021508 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/bin/unicorn:126:in >>>> `<top (required)>' >>>> E, [2016-07-22T09:51:24.021596 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `load' >>>> E, [2016-07-22T09:51:24.021695 #38662] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `<main>' >>>> >>>> >>>> ==> /var/log/gitlab/unicorn/unicorn_stdout.log <== >>>> >>>> >>>> ==> /var/log/gitlab/unicorn/state <== >>>> >>>> >>>> ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== >>>> E, [2016-07-22T09:51:45.169949 #38665] ERROR -- : app error: invalid >>>> address (IPAddr::InvalidAddressError) >>>> E, [2016-07-22T09:51:45.198460 #38665] ERROR -- : >>>> /opt/gitlab/embedded/lib/ruby/2.1.0/ipaddr.rb:559:in `in6_addr' >>>> E, [2016-07-22T09:51:45.198567 #38665] ERROR -- : >>>> /opt/gitlab/embedded/lib/ruby/2.1.0/ipaddr.rb:496:in `initialize' >>>> E, [2016-07-22T09:51:45.198615 #38665] ERROR -- : >>>> /opt/gitlab/embedded/lib/ruby/2.1.0/ipaddr.rb:514:in `new' >>>> E, [2016-07-22T09:51:45.198673 #38665] ERROR -- : >>>> /opt/gitlab/embedded/lib/ruby/2.1.0/ipaddr.rb:514:in `coerce_other' >>>> E, [2016-07-22T09:51:45.198719 #38665] ERROR -- : >>>> /opt/gitlab/embedded/lib/ruby/2.1.0/ipaddr.rb:170:in `include?' >>>> E, [2016-07-22T09:51:45.198764 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gitlab-rails/config/initializers/trusted_proxies.rb:9:in >>>> `block in trusted_proxy?' >>>> E, [2016-07-22T09:51:45.198820 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gitlab-rails/config/initializers/trusted_proxies.rb:9:in >>>> `each' >>>> E, [2016-07-22T09:51:45.198866 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gitlab-rails/config/initializers/trusted_proxies.rb:9:in >>>> `any?' >>>> E, [2016-07-22T09:51:45.198912 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gitlab-rails/config/initializers/trusted_proxies.rb:9:in >>>> `trusted_proxy?' >>>> E, [2016-07-22T09:51:45.198963 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/request.rb:365:in >>>> `block in reject_trusted_ip_addresses' >>>> E, [2016-07-22T09:51:45.199009 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/request.rb:365:in >>>> `reject' >>>> E, [2016-07-22T09:51:45.199054 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/request.rb:365:in >>>> `reject_trusted_ip_addresses' >>>> E, [2016-07-22T09:51:45.199110 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/request.rb:356:in >>>> `ip' >>>> E, [2016-07-22T09:51:45.199157 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.7/lib/action_dispatch/http/request.rb:226:in >>>> `ip' >>>> E, [2016-07-22T09:51:45.199209 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/rack/logger.rb:53:in >>>> `started_request_message' >>>> E, [2016-07-22T09:51:45.199255 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/rack/logger.rb:37:in >>>> `block in call_app' >>>> E, [2016-07-22T09:51:45.199300 #38665] ERROR -- : >>>> /opt/gitlab/embedded/lib/ruby/2.1.0/logger.rb:372:in `add' >>>> E, [2016-07-22T09:51:45.199345 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-session_store-1.0.0/lib/active_record/session_store/extension/logger_silencer.rb:38:in >>>> `add_with_threadsafety' >>>> E, [2016-07-22T09:51:45.199390 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/logger.rb:77:in >>>> `add' >>>> E, [2016-07-22T09:51:45.199447 #38665] ERROR -- : >>>> /opt/gitlab/embedded/lib/ruby/2.1.0/logger.rb:434:in `info' >>>> E, [2016-07-22T09:51:45.199492 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/rack/logger.rb:37:in >>>> `call_app' >>>> E, [2016-07-22T09:51:45.199546 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/rack/logger.rb:20:in >>>> `block in call' >>>> E, [2016-07-22T09:51:45.199592 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/tagged_logging.rb:68:in >>>> `block in tagged' >>>> E, [2016-07-22T09:51:45.199637 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/tagged_logging.rb:26:in >>>> `tagged' >>>> E, [2016-07-22T09:51:45.199697 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/tagged_logging.rb:68:in >>>> `tagged' >>>> E, [2016-07-22T09:51:45.199748 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/rack/logger.rb:20:in >>>> `call' >>>> E, [2016-07-22T09:51:45.199794 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/request_store-1.3.0/lib/request_store/middleware.rb:9:in >>>> `call' >>>> E, [2016-07-22T09:51:45.199839 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.7/lib/action_dispatch/middleware/request_id.rb:21:in >>>> `call' >>>> E, [2016-07-22T09:51:45.199885 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in >>>> `call' >>>> E, [2016-07-22T09:51:45.199933 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in >>>> `call' >>>> E, [2016-07-22T09:51:45.199986 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/lock.rb:17:in >>>> `call' >>>> E, [2016-07-22T09:51:45.200032 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in >>>> `call' >>>> E, [2016-07-22T09:51:45.200084 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sentry-raven-1.1.0/lib/raven/integrations/rack.rb:53:in >>>> `call' >>>> E, [2016-07-22T09:51:45.200130 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/engine.rb:518:in >>>> `call' >>>> E, [2016-07-22T09:51:45.200175 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/application.rb:165:in >>>> `call' >>>> E, [2016-07-22T09:51:45.200220 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/railtie.rb:194:in >>>> `public_send' >>>> E, [2016-07-22T09:51:45.200271 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.7/lib/rails/railtie.rb:194:in >>>> `method_missing' >>>> E, [2016-07-22T09:51:45.200316 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in >>>> `block in call' >>>> E, [2016-07-22T09:51:45.200370 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in >>>> `each' >>>> E, [2016-07-22T09:51:45.200417 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in >>>> `call' >>>> E, [2016-07-22T09:51:45.200462 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:580:in >>>> `process_client' >>>> E, [2016-07-22T09:51:45.200507 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in >>>> `process_client' >>>> E, [2016-07-22T09:51:45.200561 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:674:in >>>> `worker_loop' >>>> E, [2016-07-22T09:51:45.200607 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:529:in >>>> `spawn_missing_workers' >>>> E, [2016-07-22T09:51:45.200652 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:140:in >>>> `start' >>>> E, [2016-07-22T09:51:45.200697 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/bin/unicorn:126:in >>>> `<top (required)>' >>>> E, [2016-07-22T09:51:45.200745 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `load' >>>> E, [2016-07-22T09:51:45.200795 #38665] ERROR -- : >>>> /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `<main>' >>>> >>>> >>>> >>>> >>>> >>>> Looks like the error is in the ippaddr.rb and throws on the in6_addr >>>> function. >>>> >>>> I've run *gitlab-ctl reconfigure* and *gitlab-ctl restart* a few >>>> times.Never received this error while on 8.9.x >>>> >>>> Anyone have ideas on how to further determine what is causing this >>>> error? I have to imagine it has something to do with ipv6, is there any way >>>> to just disable ipv6 for Gitlab and use only ipv4? >>>> >>>> Thanks for any help. >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "GitLab" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to gitlabhq+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/gitlabhq/bb6557ac-65bd-44a2-8f8a-d4307ad67a52%40googlegroups.com >> <https://groups.google.com/d/msgid/gitlabhq/bb6557ac-65bd-44a2-8f8a-d4307ad67a52%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "GitLab" group. To unsubscribe from this group and stop receiving emails from it, send an email to gitlabhq+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/gitlabhq/CAMBWrQ%3D5vrkbMdyPgDrJTVoXTNYCi31uu%3DLAtu-fMMySC5AdfQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
