Looking into the problem a bit more, gitlab-ctl tail shows an error in unicorn:
==> /var/log/gitlab/unicorn/unicorn_stderr.log <== E, [2016-05-19T09:36:35.343295 #22609] ERROR -- : worker=0 PID:32187 timeout (61s > 60s), killing ==> /var/log/gitlab/gitlab-workhorse/current <== 2016-05-19_13:36:35.37377 2016/05/19 09:36:35 error: proxyRoundTripper: GET "/alazar/libats/merge_requests/13" failed with: "EOF" ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== E, [2016-05-19T09:36:35.373976 #22609] ERROR -- : reaped #<Process::Status: pid 32187 SIGKILL (signal 9)> worker=0 The full log of gitlab-ctl tail is attached. On Wednesday, May 18, 2016 at 5:04:42 PM UTC-4, Romain Deterre wrote: > > Hi, > > I'm running an instance of GitLab CE version 8.7.5-ce.0.el7 on a CentOS 7 > server, and everything works well except for on merge request's page that > cannot be opened. > > production.log shows only this when a browser tries to open the page: > > Started GET "/alazar/libats/merge_requests/13" for 127.0.0.1 at 2016-05-18 > 16:55:28 -0400 > Processing by Projects::MergeRequestsController#show as HTML > Parameters: {"namespace_id"=>"alazar", "project_id"=>"libats", > "id"=>"13"} > Started GET "/alazar/libats/merge_requests/13" for 127.0.0.1 at 2016-05-18 > 16:56:29 -0400 > Processing by Projects::MergeRequestsController#show as HTML > Parameters: {"namespace_id"=>"alazar", "project_id"=>"libats", > "id"=>"13"} > Started GET "/alazar/libats/merge_requests" for 127.0.0.1 at 2016-05-18 > 16:58:09 -0400 > Processing by Projects::MergeRequestsController#index as HTML > Parameters: {"namespace_id"=>"alazar", "project_id"=>"libats"} > Completed 200 OK in 219ms (Views: 139.0ms | ActiveRecord: 20.1ms) > Started GET "/alazar/libats/merge_requests/13" for 127.0.0.1 at 2016-05-18 > 16:58:12 -0400 > Processing by Projects::MergeRequestsController#show as HTML > Parameters: {"namespace_id"=>"alazar", "project_id"=>"libats", > "id"=>"13"} > Started GET "/alazar/libats/merge_requests/13" for 127.0.0.1 at 2016-05-18 > 16:59:13 -0400 > Processing by Projects::MergeRequestsController#show as HTML > Parameters: {"namespace_id"=>"alazar", "project_id"=>"libats", > "id"=>"13"} > > gitlab-rake:check shows all green lights. > > I tried stopping and starting GitLab with sudo gitlab-ctl stop then sudo > gitlab-ctl start, but it made no difference. I also tried to log in with > various user accounts, but I saw no change. > > Is there a way to open this merge request's page, or even to get rid of it > to create a new one from the same branch? > > Thanks > -- You received this message because you are subscribed to the Google Groups "GitLab" group. To unsubscribe from this group and stop receiving emails from it, send an email to gitlabhq+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/gitlabhq/5709d7d3-3bf4-4d58-a7c1-a2071c651f88%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:09.047586 #41496] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04999 I, [2016-05-19T09:35:09.048032 #41496] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rdeterre/testing> for user with key key-12. I, [2016-05-19T09:35:10.090077 #41586] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.05458 I, [2016-05-19T09:35:10.090747 #41586] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rdeterre/testing.git> for user with key key-12. I, [2016-05-19T09:35:17.716829 #41686] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04286 I, [2016-05-19T09:35:17.717020 #41686] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/ats-windows.git> for user with key key-12. I, [2016-05-19T09:35:18.749972 #41784] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04550 I, [2016-05-19T09:35:18.750201 #41784] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/ats-windows.git> for user with key key-12. I, [2016-05-19T09:35:23.176913 #41876] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 3.38518 I, [2016-05-19T09:35:23.177103 #41876] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rdeterre/ats-windows.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Parameters: {"namespace_id"=>"alazar", "project_id"=>"libats", "id"=>"13"} Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:09 -0400 Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:10 -0400 Started GET "/alazar/support-tickets/notes?target_id=239&target_type=issue&last_fetched_at=1463664793" for 127.0.0.1 at 2016-05-19 09:35:13 -0400 Processing by Projects::NotesController#index as JSON Parameters: {"target_id"=>"239", "target_type"=>"issue", "last_fetched_at"=>"1463664793", "namespace_id"=>"alazar", "project_id"=>"support-tickets"} Completed 200 OK in 90ms (Views: 0.5ms | ActiveRecord: 8.3ms) Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:17 -0400 Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:18 -0400 Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:19 -0400 ==> /var/log/gitlab/gitlab-rails/application.log <== ==> /var/log/gitlab/gitlab-rails/satellites.log <== ==> /var/log/gitlab/gitlab-rails/githost.log <== ==> /var/log/gitlab/redis/state <== ==> /var/log/gitlab/redis/current <== 2016-05-19_13:26:25.02778 [22585] 19 May 09:26:25.027 * 10 changes in 300 seconds. Saving... 2016-05-19_13:26:25.02893 [22585] 19 May 09:26:25.028 * Background saving started by pid 40326 2016-05-19_13:26:25.07585 [40326] 19 May 09:26:25.075 * DB saved on disk 2016-05-19_13:26:25.07659 [40326] 19 May 09:26:25.076 * RDB: 5 MB of memory used by copy-on-write 2016-05-19_13:26:25.12947 [22585] 19 May 09:26:25.129 * Background saving terminated with success 2016-05-19_13:31:26.04730 [22585] 19 May 09:31:26.046 * 10 changes in 300 seconds. Saving... 2016-05-19_13:31:26.04733 [22585] 19 May 09:31:26.047 * Background saving started by pid 40792 2016-05-19_13:31:26.10021 [40792] 19 May 09:31:26.099 * DB saved on disk 2016-05-19_13:31:26.10077 [40792] 19 May 09:31:26.100 * RDB: 4 MB of memory used by copy-on-write 2016-05-19_13:31:26.14824 [22585] 19 May 09:31:26.148 * Background saving terminated with success ==> /var/log/gitlab/postgresql/state <== ==> /var/log/gitlab/postgresql/current <== 2016-05-18_20:47:53.14056 FATAL: terminating connection due to administrator command 2016-05-18_20:47:53.14100 LOG: autovacuum launcher shutting down 2016-05-18_20:47:53.14100 FATAL: terminating connection due to administrator command 2016-05-18_20:47:53.14169 FATAL: terminating connection due to administrator command 2016-05-18_20:47:53.14170 FATAL: terminating connection due to administrator command 2016-05-18_20:47:53.14288 LOG: shutting down 2016-05-18_20:47:53.15919 LOG: database system is shut down 2016-05-18_20:48:06.04022 LOG: database system was shut down at 2016-05-18 20:47:53 GMT 2016-05-18_20:48:06.04949 LOG: database system is ready to accept connections 2016-05-18_20:48:06.04951 LOG: autovacuum launcher started ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== I, [2016-05-19T09:33:58.101919 #41370] INFO -- : worker=1 spawned pid=41370 I, [2016-05-19T09:33:58.102482 #41370] INFO -- : worker=1 ready E, [2016-05-19T09:34:09.107387 #22609] ERROR -- : worker=4 PID:41288 timeout (61s > 60s), killing E, [2016-05-19T09:34:09.132788 #22609] ERROR -- : reaped #<Process::Status: pid 41288 SIGKILL (signal 9)> worker=4 I, [2016-05-19T09:34:09.140336 #41385] INFO -- : worker=4 spawned pid=41385 I, [2016-05-19T09:34:09.140783 #41385] INFO -- : worker=4 ready E, [2016-05-19T09:35:10.199483 #22609] ERROR -- : worker=2 PID:37827 timeout (61s > 60s), killing E, [2016-05-19T09:35:10.248594 #22609] ERROR -- : reaped #<Process::Status: pid 37827 SIGKILL (signal 9)> worker=2 I, [2016-05-19T09:35:10.256961 #41621] INFO -- : worker=2 spawned pid=41621 I, [2016-05-19T09:35:10.257661 #41621] INFO -- : worker=2 ready ==> /var/log/gitlab/unicorn/unicorn_stdout.log <== ==> /var/log/gitlab/unicorn/state <== ==> /var/log/gitlab/unicorn/current <== 2016-05-18_20:47:55.24449 Received TERM from runit, sending to process group (-PID) 2016-05-18_20:47:55.24623 Terminated 2016-05-18_20:47:55.24656 forwarding TERM to unicorn master 35305 2016-05-18_20:47:55.24657 forwarding CONT to unicorn master 35305 2016-05-18_20:47:55.24658 wrapper for unicorn master 35305 exiting 2016-05-18_20:48:07.22437 starting new unicorn master 2016-05-18_20:48:23.62603 adopted new unicorn master 22609 ==> /var/log/gitlab/sidekiq/state <== ==> /var/log/gitlab/sidekiq/current <== 2016-05-19_13:20:12.97727 2016-05-19T13:20:12.977Z 22589 TID-aagt0 INFO: Cron Jobs - add job with name: repository_check_worker 2016-05-19_13:20:13.02868 2016-05-19T13:20:13.028Z 22589 TID-9ln2k RepositoryCheck::BatchWorker JID-938f7e0eaf2c925c1facdda0 INFO: done: 0.059 sec 2016-05-19_13:26:12.56409 2016-05-19T13:26:12.563Z 22589 TID-a6iw4 NewNoteWorker JID-4d034858ea9b4fd56a2581f0 INFO: start 2016-05-19_13:26:13.27676 2016-05-19T13:26:13.276Z 22589 TID-9mtnw ActionMailer::DeliveryJob JID-1305fff5d2e847aa432347cf INFO: start 2016-05-19_13:26:13.63718 2016-05-19T13:26:13.636Z 22589 TID-a6iw4 NewNoteWorker JID-4d034858ea9b4fd56a2581f0 INFO: done: 1.073 sec 2016-05-19_13:26:14.39821 2016-05-19T13:26:14.397Z 22589 TID-9mtnw ActionMailer::DeliveryJob JID-1305fff5d2e847aa432347cf INFO: done: 1.121 sec 2016-05-19_13:27:41.81594 2016-05-19T13:27:41.815Z 22589 TID-9ln2k NewNoteWorker JID-9c8b42baf122ec0fc0085c46 INFO: start 2016-05-19_13:27:42.00140 2016-05-19T13:27:42.001Z 22589 TID-a9veg ActionMailer::DeliveryJob JID-3bcb07a607b3000effb76a5d INFO: start 2016-05-19_13:27:42.18551 2016-05-19T13:27:42.184Z 22589 TID-9ln2k NewNoteWorker JID-9c8b42baf122ec0fc0085c46 INFO: done: 0.369 sec 2016-05-19_13:27:42.20124 2016-05-19T13:27:42.201Z 22589 TID-a9veg ActionMailer::DeliveryJob JID-3bcb07a607b3000effb76a5d INFO: done: 0.2 sec ==> /var/log/gitlab/nginx/current <== ==> /var/log/gitlab/nginx/error.log <== ==> /var/log/gitlab/nginx/gitlab_access.log <== 10.3.30.109 - - [19/May/2016:09:32:45 -0400] "GET /autocomplete/users.json?search=&per_page=20&active=true&project_id=45¤t_user=true HTTP/1.1" 200 638 "http://ats-gitlab/alazar/libats/issues" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 10.3.30.109 - - [19/May/2016:09:32:45 -0400] "GET /autocomplete/users.json?search=&per_page=20&active=true&project_id=45¤t_user=true HTTP/1.1" 200 638 "http://ats-gitlab/alazar/libats/issues" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 10.3.30.109 - - [19/May/2016:09:32:51 -0400] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 2204 "http://ats-gitlab/dashboard/merge_requests?scope=all&sort=id_desc&state=opened&utf8=%E2%9C%93&assignee_id=4" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 10.3.30.109 - - [19/May/2016:09:33:02 -0400] "GET /alazar/libats HTTP/1.1" 200 7320 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 10.3.30.109 - - [19/May/2016:09:33:04 -0400] "GET /alazar/libats/merge_requests HTTP/1.1" 200 6806 "http://ats-gitlab/alazar/libats" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 10.3.30.105 - - [19/May/2016:09:33:13 -0400] "GET /alazar/support-tickets/notes?target_id=239&target_type=issue&last_fetched_at=1463664673 HTTP/1.1" 200 72 "http://ats-gitlab/alazar/support-tickets/issues/35" "Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" 10.3.30.109 - - [19/May/2016:09:33:58 -0400] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 10.3.30.109 - - [19/May/2016:09:34:09 -0400] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 2204 "http://ats-gitlab/alazar/libats/merge_requests" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 10.3.30.109 - - [19/May/2016:09:35:10 -0400] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 2204 "http://ats-gitlab/alazar/libats/merge_requests" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 10.3.30.105 - - [19/May/2016:09:35:13 -0400] "GET /alazar/support-tickets/notes?target_id=239&target_type=issue&last_fetched_at=1463664793 HTTP/1.1" 200 72 "http://ats-gitlab/alazar/support-tickets/issues/35" "Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" ==> /var/log/gitlab/nginx/gitlab_error.log <== ==> /var/log/gitlab/logrotate/state <== ==> /var/log/gitlab/logrotate/current <== 2016-05-18_20:47:52.29190 Received TERM from runit, sending to process group (-PID) ==> /var/log/gitlab/gitlab-workhorse/state <== ==> /var/log/gitlab/gitlab-workhorse/current <== 2016-05-19_13:33:58.08930 2016/05/19 09:33:58 error: proxyRoundTripper: GET "/alazar/libats/merge_requests/13" failed with: "EOF" 2016-05-19_13:33:58.08944 2016/05/19 09:33:58 ErrorPage: serving predefined error page: 502 2016-05-19_13:33:58.08953 ats-gitlab @ - - [2016-05-19 09:32:57.630355717 -0400 EDT] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 2192 "" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 60.459131 2016-05-19_13:34:09.13262 2016/05/19 09:34:09 error: proxyRoundTripper: GET "/alazar/libats/merge_requests/13" failed with: "EOF" 2016-05-19_13:34:09.13272 2016/05/19 09:34:09 ErrorPage: serving predefined error page: 502 2016-05-19_13:34:09.13301 ats-gitlab @ - - [2016-05-19 09:33:08.406705395 -0400 EDT] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 2192 "http://ats-gitlab/alazar/libats/merge_requests" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 60.726066 2016-05-19_13:35:10.24775 2016/05/19 09:35:10 error: proxyRoundTripper: GET "/alazar/libats/merge_requests/13" failed with: "EOF" 2016-05-19_13:35:10.24825 2016/05/19 09:35:10 ErrorPage: serving predefined error page: 502 2016-05-19_13:35:10.24827 ats-gitlab @ - - [2016-05-19 09:34:09.144893041 -0400 EDT] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 2192 "http://ats-gitlab/alazar/libats/merge_requests" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 61.103087 2016-05-19_13:35:13.39666 ats-gitlab @ - - [2016-05-19 09:35:13.29819449 -0400 EDT] "GET /alazar/support-tickets/notes?target_id=239&target_type=issue&last_fetched_at=1463664793 HTTP/1.1" 200 41 "http://ats-gitlab/alazar/support-tickets/issues/35" "Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" 0.098308 ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:26 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:29.791575 #42016] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 3.38311 I, [2016-05-19T09:35:29.791806 #42016] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/ats-sdk.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:30 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:30.813486 #42122] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04802 I, [2016-05-19T09:35:30.813685 #42122] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/ats-sdk.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:31 -0400 Started GET "/alazar/libats/merge_requests/13" for 127.0.0.1 at 2016-05-19 09:35:34 -0400 Processing by Projects::MergeRequestsController#show as HTML Parameters: {"namespace_id"=>"alazar", "project_id"=>"libats", "id"=>"13"} ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:35.270345 #42211] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 3.54227 I, [2016-05-19T09:35:35.270607 #42211] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rdeterre/ats-sdk.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:38 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:38.681029 #42314] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.10159 I, [2016-05-19T09:35:38.681254 #42314] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/lawrence.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:39 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:39.762281 #42410] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.03449 I, [2016-05-19T09:35:39.762679 #42410] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/lawrence.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:40 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:40.861477 #42499] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.05107 I, [2016-05-19T09:35:40.861704 #42499] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rdeterre/lawrence.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:44 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:44.254804 #42583] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.10468 I, [2016-05-19T09:35:44.255097 #42583] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/ats-vi.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:45 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:45.272473 #42676] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04366 I, [2016-05-19T09:35:45.272724 #42676] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/ats-vi.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:48 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:48.465647 #42768] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.06099 I, [2016-05-19T09:35:48.465874 #42768] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/ats-utilities.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:49 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:49.579165 #42848] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.05703 I, [2016-05-19T09:35:49.579460 #42848] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/ats-utilities.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:52 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:52.875174 #42952] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.06103 I, [2016-05-19T09:35:52.875462 #42952] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/atnapi.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:53 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:53.910591 #43034] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04956 I, [2016-05-19T09:35:53.911207 #43034] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/atnapi.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:54 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:54.934156 #43113] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.05334 I, [2016-05-19T09:35:54.934435 #43113] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rdeterre/atnapi.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:58 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:58.188445 #43217] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.05941 I, [2016-05-19T09:35:58.188655 #43217] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/alazardso.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:35:59 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:35:59.201813 #43299] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04142 I, [2016-05-19T09:35:59.202028 #43299] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/alazardso.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:36:00 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:36:00.211821 #43396] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04368 I, [2016-05-19T09:36:00.212032 #43396] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rdeterre/alazardso.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:36:03 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:36:03.475333 #43506] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04755 I, [2016-05-19T09:36:03.475538 #43506] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/documentation.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:36:04 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:36:04.454842 #43595] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.03716 I, [2016-05-19T09:36:04.455497 #43595] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/documentation.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:36:07 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:36:07.673332 #43691] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.05656 I, [2016-05-19T09:36:07.673563 #43691] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/libats.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:36:08 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:36:08.694571 #43782] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04859 I, [2016-05-19T09:36:08.694968 #43782] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rkhatibi/libats.git> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:36:09 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:36:09.721950 #43872] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04641 I, [2016-05-19T09:36:09.722206 #43872] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/rdeterre/libats> for user with key key-12. ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2016-05-19 09:36:12 -0400 ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== I, [2016-05-19T09:36:12.939382 #43977] INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.05262 I, [2016-05-19T09:36:12.939631 #43977] INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/alazar/plxapi> for user with key key-12. ==> /var/log/gitlab/redis/current <== 2016-05-19_13:36:27.08615 [22585] 19 May 09:36:27.085 * 10 changes in 300 seconds. Saving... 2016-05-19_13:36:27.08728 [22585] 19 May 09:36:27.086 * Background saving started by pid 44071 2016-05-19_13:36:27.11407 [44071] 19 May 09:36:27.113 * DB saved on disk 2016-05-19_13:36:27.11410 [44071] 19 May 09:36:27.113 * RDB: 4 MB of memory used by copy-on-write 2016-05-19_13:36:27.18702 [22585] 19 May 09:36:27.186 * Background saving terminated with success ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== E, [2016-05-19T09:36:35.343295 #22609] ERROR -- : worker=0 PID:32187 timeout (61s > 60s), killing ==> /var/log/gitlab/gitlab-workhorse/current <== 2016-05-19_13:36:35.37377 2016/05/19 09:36:35 error: proxyRoundTripper: GET "/alazar/libats/merge_requests/13" failed with: "EOF" ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== E, [2016-05-19T09:36:35.373976 #22609] ERROR -- : reaped #<Process::Status: pid 32187 SIGKILL (signal 9)> worker=0 ==> /var/log/gitlab/gitlab-workhorse/current <== 2016-05-19_13:36:35.37423 2016/05/19 09:36:35 ErrorPage: serving predefined error page: 502 2016-05-19_13:36:35.37427 ats-gitlab @ - - [2016-05-19 09:35:34.948525762 -0400 EDT] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 2192 "http://ats-gitlab/alazar/libats/merge_requests" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" 60.425572 ==> /var/log/gitlab/nginx/gitlab_access.log <== 10.3.30.109 - - [19/May/2016:09:36:35 -0400] "GET /alazar/libats/merge_requests/13 HTTP/1.1" 502 2204 "http://ats-gitlab/alazar/libats/merge_requests" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== I, [2016-05-19T09:36:35.381863 #44080] INFO -- : worker=0 spawned pid=44080 I, [2016-05-19T09:36:35.382521 #44080] INFO -- : worker=0 ready ==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/alazar/support-tickets/notes?target_id=239&target_type=issue&last_fetched_at=1463664913" for 127.0.0.1 at 2016-05-19 09:37:13 -0400 Processing by Projects::NotesController#index as JSON Parameters: {"target_id"=>"239", "target_type"=>"issue", "last_fetched_at"=>"1463664913", "namespace_id"=>"alazar", "project_id"=>"support-tickets"} Completed 200 OK in 404ms (Views: 0.8ms | ActiveRecord: 26.5ms) ==> /var/log/gitlab/gitlab-workhorse/current <== 2016-05-19_13:37:14.14352 ats-gitlab @ - - [2016-05-19 09:37:13.406464413 -0400 EDT] "GET /alazar/support-tickets/notes?target_id=239&target_type=issue&last_fetched_at=1463664913 HTTP/1.1" 200 41 "http://ats-gitlab/alazar/support-tickets/issues/35" "Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" 0.736895 ==> /var/log/gitlab/nginx/gitlab_access.log <== 10.3.30.105 - - [19/May/2016:09:37:14 -0400] "GET /alazar/support-tickets/notes?target_id=239&target_type=issue&last_fetched_at=1463664913 HTTP/1.1" 200 72 "http://ats-gitlab/alazar/support-tickets/issues/35" "Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36"