It looks to me the worker abruptly terminates due to lack of memory. How
much RAM does your machine have?
On Wed, Feb 17, 2016 at 11:42 AM, Michael Albert <m...@askkodiak.com> wrote:
> Not exactly sure what to look for in the sidekiq log, but the only thing
> standing out to me are a bunch of "Cannot allocate memory - ps" warnings.
> Hopefully helpful?
>
> 2016-02-16_22:14:44.75819 2016-02-16T22:14:44.757Z 11097 TID-aznpw
> ProjectServiceWorker JID-174ef3e92c00664ffe75439a INFO: start
> 2016-02-16_22:14:45.87580 2016-02-16T22:14:45.875Z 11097 TID-aznpw
> ProjectServiceWorker JID-174ef3e92c00664ffe75439a INFO: fail: 1.117 sec
> 2016-02-16_22:14:45.87607 2016-02-16T22:14:45.875Z 11097 TID-aznpw WARN:
> {"class"=>"ProjectServiceWorker", "args"=>[33, {"object_kind"=>"issue",
> "user"=>{"name"=>"REDACTED", "username"=>"REDACTED", "avatar_url"=>"
> https://secure.gravatar.com/avatar/REDACTED"},
> "repository"=>{"name"=>"REDACTED", "url"=>"git@git.REDACTED",
> "description"=>"REDACTED", "homepage"=>"REDACTED"},
> "object_attributes"=>{"id"=>213, "title"=>"REDACTED", "assignee_id"=>nil,
> "author_id"=>1, "project_id"=>4, "created_at"=>"2016-02-04 11:49:03 UTC",
> "updated_at"=>"2016-02-16 21:24:18 UTC", "position"=>0, "branch_name"=>nil,
> "description"=>"REDACTED", "milestone_id"=>nil, "state"=>"closed",
> "iid"=>207, "updated_by_id"=>nil, "url"=>"https://git.REDACTED";,
> "action"=>"close"}}], "retry"=>true, "queue"=>"project_web_hook",
> "jid"=>"174ef3e92c00664ffe75439a", "created_at"=>1455657858.4603503,
> "enqueued_at"=>1455660884.7553408, "error_message"=>"Cannot allocate memory
> - ps", "error_class"=>"Errno::ENOMEM", "failed_at"=>1455657859.1589346,
> "retry_count"=>7, "retried_at"=>1455660885.8727381}
> 2016-02-16_22:14:45.87616 2016-02-16T22:14:45.876Z 11097 TID-aznpw WARN:
> Errno::ENOMEM: Cannot allocate memory - ps
> 2016-02-16_22:14:45.87626 2016-02-16T22:14:45.876Z 11097 TID-aznpw WARN:
> /opt/gitlab/embedded/lib/ruby/2.1.0/open3.rb:193:in `spawn'
> 2016-02-16_22:14:45.87627
> /opt/gitlab/embedded/lib/ruby/2.1.0/open3.rb:193:in `popen_run'
> 2016-02-16_22:14:45.87627
> /opt/gitlab/embedded/lib/ruby/2.1.0/open3.rb:93:in `popen3'
> 2016-02-16_22:14:45.87627
> /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/popen.rb:23:in `popen'
> 2016-02-16_22:14:45.87628
> /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:47:in
> `get_rss'
> 2016-02-16_22:14:45.87628
> /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:18:in
> `call'
> 2016-02-16_22:14:45.87628
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in
> `block in invoke'
> 2016-02-16_22:14:45.87628
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/active_record.rb:6:in
> `call'
> 2016-02-16_22:14:45.87629
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in
> `block in invoke'
> 2016-02-16_22:14:45.87629
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in
> `call'
> 2016-02-16_22:14:45.87629
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in
> `block in invoke'
> 2016-02-16_22:14:45.87630
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:11:in
> `block in call'
> 2016-02-16_22:14:45.87630
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/logging.rb:30:in
> `with_context'
> 2016-02-16_22:14:45.87630
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:7:in
> `call'
> 2016-02-16_22:14:45.87630
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in
> `block in invoke'
> 2016-02-16_22:14:45.87631
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in
> `call'
> 2016-02-16_22:14:45.87631
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in
> `invoke'
> 2016-02-16_22:14:45.87631
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:127:in
> `block in process'
> 2016-02-16_22:14:45.87633
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:166:in
> `stats'
> 2016-02-16_22:14:45.87634
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:126:in
> `process'
> 2016-02-16_22:14:45.87634
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:79:in
> `process_one'
> 2016-02-16_22:14:45.87634
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:67:in
> `run'
> 2016-02-16_22:14:45.87635
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:16:in
> `watchdog'
> 2016-02-16_22:14:45.87635
> /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:24:in
> `block in safe_thread'
>
>
>
> On Wed, Feb 17, 2016 at 2:03 PM, Stan Hu <sta...@gmail.com> wrote:
>
>> Actually, I don't think that bug fixes this specific problem. It looks to
>> me that the Sidekiq task keeps re-running for some reason, perhaps because
>> it didn't get a 200 response from the endpoint in time. Can you check
>> /var/log/gitlab/sidekiq/current to see if there are errors?
>>
>>
>> On Wed, Feb 17, 2016 at 12:36 AM, Achilleas Pipinellis <
>> markel...@gmail.com> wrote:
>>
>>> On 02/16/2016 11:35 PM, m...@askkodiak.com wrote:
>>> > Rocking GitLab community edition with Slack integration for
>>> > issues/commits/etc. Have been using since last April or so. Recently
>>> > started seeing duplicate updates trickle out of GitLab as shown in the
>>> > attached screenshot. Am I alone? Anyone got any tips for me?
>>> >
>>> > Thanks in advance.
>>> >
>>> > Mike
>>> >
>>> >
>>>
>>> Ti should be fixed by
>>> https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2763 which will
>>> be included in 8.5.
>>>
>>>
>>> --
>>> Blog: http://axilleas.me
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "GitLab" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to gitlabhq+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/gitlabhq/56C43121.6040303%40gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "GitLab" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/gitlabhq/cBKh90iF0fk/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> gitlabhq+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/gitlabhq/CAMBWrQmcueZJKBvx5X%3DhAAEAgAWLQj6XiGNYUEDemR8NeY_L9g%40mail.gmail.com
>> <https://groups.google.com/d/msgid/gitlabhq/CAMBWrQmcueZJKBvx5X%3DhAAEAgAWLQj6XiGNYUEDemR8NeY_L9g%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "GitLab" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gitlabhq+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gitlabhq/CAHVQ%2BkVBYKWneWEs-MiA089-fjycRDfyuU_C_mHspEF7sW8VVw%40mail.gmail.com
> <https://groups.google.com/d/msgid/gitlabhq/CAHVQ%2BkVBYKWneWEs-MiA089-fjycRDfyuU_C_mHspEF7sW8VVw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"GitLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to gitlabhq+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/gitlabhq/CAMBWrQk_9iST6G2UXFJdm86oR2SMDREcx-co2pYS4upWf5hyrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
