Larry, Can you discontinue accessing MAAS over port 80 and start accessing it through port 5240? And see if you have issues.
Apache does a bad job proxying! On Wednesday, September 28, 2016, Larry Michel <larry.mic...@canonical.com> wrote: > Public bug reported: > > We stopped being able to access maas through its web address and this > caused a crash: > > File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line > 137, in get_tagged_nodes > all_nodes_list = self.nodes_list(user) > File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line > 378, in nodes_list > return json.loads(self.cmd(command)) > File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line > 281, in cmd > content = self.client.get(command).read() > File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line > 237, in get > url, method="GET", headers=headers) > File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line > 116, in dispatch_query > res = urllib2.urlopen(req) > File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen > return _opener.open(url, data, timeout) > File "/usr/lib/python2.7/urllib2.py", line 404, in open > response = self._open(req, data) > File "/usr/lib/python2.7/urllib2.py", line 422, in _open > '_open', req) > File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain > result = func(*args) > File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open > return self.do_open(httplib.HTTPConnection, req) > File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open > raise URLError(err) > URLError: <urlopen error [Errno 111] Connection refused> > > On the apache2 error log, I saw: > > ue Sep 27 06:25:12.159028 2016] [mpm_event:notice] [pid 1895:tid > 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming > normal operations > [Tue Sep 27 06:25:12.159257 2016] [core:notice] [pid 1895:tid > 140037883340672] AH00094: Command line: '/usr/sbin/apache2' > [Wed Sep 28 06:25:09.318844 2016] [mpm_event:notice] [pid 1895:tid > 140037883340672] AH00493: SIGUSR1 received. Doing graceful restart > AH00558: apache2: Could not reliably determine the server's fully > qualified domain name, using 127.0.1.1. Set the 'ServerName' directive > globally to suppress this message > [Wed Sep 28 06:25:12.038905 2016] [mpm_event:notice] [pid 1895:tid > 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming > normal operations > [Wed Sep 28 06:25:12.038944 2016] [core:notice] [pid 1895:tid > 140037883340672] AH00094: Command line: '/usr/sbin/apache2' > [Wed Sep 28 06:25:14.040425 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 06:25:15.041505 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 06:25:16.042554 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 06:25:17.043605 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 06:25:18.045062 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 06:25:19.046130 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 06:25:20.047475 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 06:25:21.048295 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > ... <repeats non-stop> > ull, not at MaxRequestWorkers > [Wed Sep 28 12:53:25.662213 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 12:53:26.663278 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 12:53:27.664325 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 12:53:28.665413 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > [Wed Sep 28 12:53:29.666492 2016] [mpm_event:error] [pid 1895:tid > 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers > > > The errors stopped when I restarted the apache2 service with sudo service > apache2 restart. > > I couldn't tell where to find the core. There was only this file from a > week ago: > $ ls -l _usr_sbin_apache2.0.crash > -rw-r----- 1 root root 2526598 Sep 21 22:33 _usr_sbin_apache2.0.crash > > I tried to rerun it again and timestamp changed but core info did not > appear to change. > > I have collected access and error logs of the past couple of days as > well as other logs including maas logs. Tarball is attached to this bug > and it contains the 2 crash files: > > $ ls -l _usr_sbin_apache2.0.crash.* > -rw-r----- 1 ubuntu ubuntu 2698755 Sep 28 13:42 > _usr_sbin_apache2.0.crash.new > -rw-r----- 1 ubuntu ubuntu 2526598 Sep 21 22:33 > _usr_sbin_apache2.0.crash.old > > > These are the package versions: > > ii maas 2.0.0+bzr5189-0 all "Metal as a > Service" is a physical cloud and I > ii apache2 2.4.18-2ubuntu3.1 > amd64 Apache HTTP Server > ii apache2-bin 2.4.18-2ubuntu3.1 > amd64 Apache HTTP Server (modules and other binary files) > ii apache2-data 2.4.18-2ubuntu3.1 > all Apache HTTP Server (common files) > ii apache2-utils 2.4.18-2ubuntu3.1 > amd64 Apache HTTP Server (utility programs for web servers) > ii init 1.29ubuntu2 > amd64 System-V-like init utilities - metapackage > > > $ lsb_release -rd > Description: Ubuntu 16.04.1 LTS > Release: 16.04 > > ** Affects: maas > Importance: Undecided > Status: New > > ** Affects: apache2 (Ubuntu) > Importance: Undecided > Status: New > > > ** Tags: oil > > ** Attachment added: "apache2.logs.tar.gz" > https://bugs.launchpad.net/bugs/1628535/+attachment/ > 4750157/+files/apache2.logs.tar.gz > > ** Summary changed: > > - AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes > + AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - > Can't access maas server URL > > ** Also affects: maas > Importance: Undecided > Status: New > > -- > You received this bug notification because you are subscribed to MAAS. > https://bugs.launchpad.net/bugs/1628535 > > Title: > AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - > Can't access maas server URL > > To manage notifications about this bug go to: > https://bugs.launchpad.net/maas/+bug/1628535/+subscriptions > -- Andres Rodriguez (RoAkSoAx) Ubuntu Server Developer MSc. Telecom & Networking Systems Engineer -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1628535 Title: AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - Can't access maas server URL To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1628535/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
