I'm experiencing an annoying problem with a couple of django apps. Both apps are games built on Django that make regular calls to the db using python-db. We have error emails turned on, and every so often (the interval doesn't seem to follow a pattern but it's usually no more than once in a half-hour) we get an email with the following info (I've obscured/changed some identifying info, for the record):
OperationalError at /analytics/ (2003, "Can't connect to MySQL server on 'x.x.x.x' (111)") Request Method: POST Request URL: http://hostname.domain/analytics<http://rubyskies.eastsidegamestudio.com/analytics/> Django Version: 1.3 Exception Type: OperationalError Exception Value: (2003, "Can't connect to MySQL server on 'x.x.x.x' (111)") Exception Location: /usr/lib/pymodules/python2.6/MySQLdb/connections.py in __init__, line 170 Python Executable: /usr/sbin/uwsgi-1.2.3 Python Version: 2.6.5 Python Path: ['app_path', '.', '', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/lib/python2.6/dist-packages', '/usr/lib/python2.6/dist-packages/PIL', '/usr/lib/pymodules/python2.6', '/usr/local/lib/python2.6/dist-packages', '/usr/local/lib/python2.6/dist-packages/newrelic-1.5.0.103', 'app_path/src', 'app_path/src/expedition', 'app_path/src/expedition/settings'] Server time: Thu, 25 Oct 2012 14:14:08 -0700 Traceback - /usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py in get_response 1. response = callback(request, *callback_args, **callback_kwargs) Local Vars Variable Value exceptions <module 'django.core.exceptions' from '/usr/local/lib/python2.6/dist-packages/django/core/exceptions.pyc'> e OperationalError(2003, "Can't connect to MySQL server on 'x.x.x.x' (111)") callback_args () receivers [(<function _rollback_on_exception at 0x1f9a578>, None)] middleware_method <newrelic.api.object_wrapper.ObjectWrapper object at 0x1fe0350> self <django.core.handlers.wsgi.WSGIHandler object at 0x1d6c810> settings <django.conf.LazySettings object at 0x1abded0> request <WSGIRequest GET:<QueryDict: {}>, POST:<could not parse>, COOKIES:{}, META:{'CONTENT_LENGTH': '275', 'CONTENT_TYPE': 'application/x-www-form-urlencoded', 'DOCUMENT_ROOT': '/opt/nginx-1.2.1/html', 'HTTP_ACCEPT': '*/*', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us', 'HTTP_CONTENT_LENGTH': '275', 'HTTP_CONTENT_TYPE': 'application/x-www-form-urlencoded', 'HTTP_HOST': 'hostname.domain', 'HTTP_USER_AGENT': 'HTTPagent/1.2 CFNetwork/609 Darwin/13.0.0', 'HTTP_VIA': 'HTTP/1.1 akrmspsrvz4ts212.wnsnet.attws.com', 'HTTP_X_FORWARDED_FOR': 'x.x.x.x, x.x.x.x', 'HTTP_X_REAL_IP': 'x.x.x.x', 'PATH_INFO': u'/analytics/', 'QUERY_STRING': '', 'REMOTE_ADDR': 'x.x.x.x', 'REMOTE_PORT': '49328', 'REQUEST_METHOD': 'POST', 'REQUEST_URI': '/analytics/', 'SCRIPT_NAME': u'', 'SERVER_NAME': 'hostname.domain', 'SERVER_PORT': '80', 'SERVER_PROTOCOL': 'HTTP/1.0', 'uwsgi.core': 15, 'uwsgi.node': 'hostname', 'uwsgi.version': '1.2.3', 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x1b9e420>, 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>, 'wsgi.input': <newrelic.api.web_transaction.WSGIInputWrapper object at 0x309b5d0>, 'wsgi.multiprocess': True, 'wsgi.multithread': True, 'wsgi.run_once': False, 'wsgi.url_scheme': 'http', 'wsgi.version': (1, 0)}> callback <newrelic.api.object_wrapper.ObjectWrapper object at 0x7fc0c825ed10> resolver <RegexURLResolver expedition.urls (None:None) ^/> urlresolvers <module 'django.core.urlresolvers' from '/usr/local/lib/python2.6/dist-packages/django/core/urlresolvers.pyc'> callback_kwargs {} response None urlconf 'expedition.urls' - /usr/local/lib/python2.6/dist-packages/newrelic-1.5.0.103/newrelic/api/object_wrapper.pyin __call__ 1. self._nr_instance, args, kwargs) The db connection works fine most of the time, it just seems like intermittently it can't connect. I've done a couple of things, like enabled tcp_tw_reuse (it seems I usually have about 150-200 connections to mysql in TIME_WAIT on any one web server), checked the mysql server and web servers message logs for too many connections errors, and looked at the mysql log to see if there's any info there. I also wrote a script that connects to the db and sends an email to me if it doesn't get expected info, initially I ran it with a cron job and didn't get any emails from it, but obviously that was only every minute. I'm now running it every second using the watch command but haven't so far had any emails. It's very strange, and I'm not sure why it's happening. As I said, we have another django app (on completely separate infrastructure) that has the same issue. I don't believe there's a network issue but I wouldn't rule it out, but I'm curious if anyone else on here has seen this behaviour before, and if so, I'd like to know how you fixed it. Any help would be appreciated. Thanks, Guy -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/QRBTXR2Kc5cJ. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
