bump. On Tue, Dec 6, 2016 at 12:45 PM, Alex Elman <zombied.proc...@gmail.com> wrote:
> I'm seeing a performance issue on my GitLab EE omnibus instance. Each > branch DELETE gets processed by a unicorn worker. I have a 3 minute timeout > on the unicorn workers and for some reason deletes often take more than 3 > minutes. There are automated build tools in my environment that routinely > execute batch deletes on old branches. When these batched deletes exceed > the number of available unicorn workers, all of the workers get tied up > doing the DELETE and GitLab becomes unavailable for a full 180 seconds. Is > there something we can tune on my side to improve this situation? Does a > future version of GitLab have improvements? I have plans to upgrade to > v8.14 in a few weeks. > > > Started DELETE "/api/v3/projects/8987/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/9067/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/9541/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/9833/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/9107/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/8967/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/9506/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/9047/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/9147/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:38 -0600 > Started DELETE "/api/v3/projects/9187/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:39 -0600 > Started DELETE "/api/v3/projects/9087/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:40 -0600 > Started DELETE "/api/v3/projects/9471/repository/branches/develop" for > 192.168.1.1 at 2016-12-06 12:05:41 -0600 > > 3 minutes later.... > > E, [2016-12-06T12:08:39.197446 #8636] ERROR -- : worker=2 PID:4084 timeout > (181s > 180s), killing > E, [2016-12-06T12:08:39.197520 #8636] ERROR -- : worker=5 PID:11150 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.197591 #8636] ERROR -- : worker=3 PID:19262 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.197664 #8636] ERROR -- : worker=1 PID:30788 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.197735 #8636] ERROR -- : worker=11 PID:28165 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.197829 #8636] ERROR -- : worker=14 PID:26486 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.197915 #8636] ERROR -- : worker=0 PID:25084 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.197996 #8636] ERROR -- : worker=13 PID:27070 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.198073 #8636] ERROR -- : worker=8 PID:4466 timeout > (181s > 180s), killing > E, [2016-12-06T12:08:39.333829 #8636] ERROR -- : reaped #<Process::Status: > pid 4466 SIGKILL (signal 9)> worker=8 > E, [2016-12-06T12:08:39.334044 #8636] ERROR -- : worker=9 PID:18003 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.334121 #8636] ERROR -- : worker=2 PID:4084 timeout > (181s > 180s), killing > E, [2016-12-06T12:08:39.334189 #8636] ERROR -- : worker=5 PID:11150 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.334257 #8636] ERROR -- : worker=3 PID:19262 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.334323 #8636] ERROR -- : worker=1 PID:30788 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.334394 #8636] ERROR -- : worker=11 PID:28165 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.334456 #8636] ERROR -- : worker=14 PID:26486 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.334520 #8636] ERROR -- : worker=0 PID:25084 > timeout (181s > 180s), killing > E, [2016-12-06T12:08:39.334587 #8636] ERROR -- : worker=13 PID:27070 > timeout (181s > 180s), killing > > System information > System: RedHatEnterpriseServer 6.7 > Current User: git > Using RVM: no > Ruby Version: 2.1.8p440 > Gem Version: 2.5.1 > Bundler Version:1.10.6 > Rake Version: 10.5.0 > Sidekiq Version:4.1.4 > > GitLab information > Version: 8.10.13-ee > Revision: 6ece4f9 > Directory: /opt/gitlab/embedded/service/gitlab-rails > DB Adapter: PostgreSQL > DB Version: 9.2.17 > URL: https://mygitlab > HTTP Clone URL: https://mygitlab/some-group/some-project.git > SSH Clone URL: git@mygitlab:some-group/some-project.git > Elasticsearch: no > Geo: no > Using LDAP: yes > Using Omniauth: yes > Omniauth Providers: saml > > GitLab Shell > Version: 3.2.1 > Repository storage paths: > - default: /var/opt/gitlab/git-data/repositories > Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/ > Git: /opt/gitlab/embedded/bin/git > > > -- > Alex Elman > -- Alex Elman -- You received this message because you are subscribed to the Google Groups "GitLab" group. To unsubscribe from this group and stop receiving emails from it, send an email to gitlabhq+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/gitlabhq/CAD-8G_oYYNxoKf-RPXv-ENXnoB9HOp0NVr6gZ%3DkXkT8KHRSP_g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
