I'm seeing a performance issue on my GitLab EE omnibus instance. Each branch DELETE gets processed by a unicorn worker. I have a 3 minute timeout on the unicorn workers and for some reason deletes often take more than 3 minutes. There are automated build tools in my environment that routinely execute batch deletes on old branches. When these batched deletes exceed the number of available unicorn workers, all of the workers get tied up doing the DELETE and GitLab becomes unavailable for a full 180 seconds. Is there something we can tune on my side to improve this situation? Does a future version of GitLab have improvements? I have plans to upgrade to v8.14 in a few weeks.
Started DELETE "/api/v3/projects/8987/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/9067/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/9541/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/9833/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/9107/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/8967/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/9506/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/9047/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/9147/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:38 -0600 Started DELETE "/api/v3/projects/9187/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:39 -0600 Started DELETE "/api/v3/projects/9087/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:40 -0600 Started DELETE "/api/v3/projects/9471/repository/branches/develop" for 192.168.1.1 at 2016-12-06 12:05:41 -0600 3 minutes later.... E, [2016-12-06T12:08:39.197446 #8636] ERROR -- : worker=2 PID:4084 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.197520 #8636] ERROR -- : worker=5 PID:11150 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.197591 #8636] ERROR -- : worker=3 PID:19262 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.197664 #8636] ERROR -- : worker=1 PID:30788 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.197735 #8636] ERROR -- : worker=11 PID:28165 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.197829 #8636] ERROR -- : worker=14 PID:26486 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.197915 #8636] ERROR -- : worker=0 PID:25084 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.197996 #8636] ERROR -- : worker=13 PID:27070 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.198073 #8636] ERROR -- : worker=8 PID:4466 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.333829 #8636] ERROR -- : reaped #<Process::Status: pid 4466 SIGKILL (signal 9)> worker=8 E, [2016-12-06T12:08:39.334044 #8636] ERROR -- : worker=9 PID:18003 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.334121 #8636] ERROR -- : worker=2 PID:4084 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.334189 #8636] ERROR -- : worker=5 PID:11150 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.334257 #8636] ERROR -- : worker=3 PID:19262 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.334323 #8636] ERROR -- : worker=1 PID:30788 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.334394 #8636] ERROR -- : worker=11 PID:28165 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.334456 #8636] ERROR -- : worker=14 PID:26486 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.334520 #8636] ERROR -- : worker=0 PID:25084 timeout (181s > 180s), killing E, [2016-12-06T12:08:39.334587 #8636] ERROR -- : worker=13 PID:27070 timeout (181s > 180s), killing System information System: RedHatEnterpriseServer 6.7 Current User: git Using RVM: no Ruby Version: 2.1.8p440 Gem Version: 2.5.1 Bundler Version:1.10.6 Rake Version: 10.5.0 Sidekiq Version:4.1.4 GitLab information Version: 8.10.13-ee Revision: 6ece4f9 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.2.17 URL: https://mygitlab HTTP Clone URL: https://mygitlab/some-group/some-project.git SSH Clone URL: git@mygitlab:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 3.2.1 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/ Git: /opt/gitlab/embedded/bin/git -- Alex Elman -- You received this message because you are subscribed to the Google Groups "GitLab" group. To unsubscribe from this group and stop receiving emails from it, send an email to gitlabhq+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/gitlabhq/CAD-8G_rCfupsciAak1nSj4Zj0MdQkJV7g3C8qFJJj%3DeuwXRxDg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
