Hello everyone,
We are using latest reviewboard and we've recently migrated to Docker
setup. Problem is, reviewboard randomly spits error 500 while browsing the
site. Looking into `docker compose logs -f`, we can see that sometimes (not
always) it tries to connect to memcached server on localhost:11211:
reviewboard-1 | 2024-03-14 10:36:23,523 - ERROR - None - AnonymousUser -
/health/ - djblets.util.views - Health check: unable to connect to cache
"default": All servers seem to be down right now
reviewboard-1 | 2024-03-14 10:36:23,524 - ERROR - None - AnonymousUser -
/health/ - django.request - Service Unavailable: /health/
reviewboard-1 | 2024-03-14 10:36:23,525 - DEBUG - -
pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
reviewboard-1 | [2024-03-14 10:36:23 +0000] [17] [DEBUG] Ignoring
connection reset
reviewboard-1 | [2024-03-14 10:36:27 +0000] [20] [DEBUG] GET /
reviewboard-1 | 2024-03-14 10:36:27,126 - DEBUG - -
pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
reviewboard-1 | 2024-03-14 10:36:27,127 - ERROR - -
djblets.cache.synchronizer - Unexpected error checking for expiration in
cached synchronization state key
"{OUR_DOMAIN}:/:{OUR_DOMAIN}:siteconfig:1:generation". Is the cache server
down? Error = [Errno 111] Connection refused
reviewboard-1 | Traceback (most recent call last):
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
82, in is_expired
reviewboard-1 | sync_gen = self._get_latest_sync_gen()
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
162, in _get_latest_sync_gen
reviewboard-1 | return cache.get(self.cache_key)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
line 77, in get
reviewboard-1 | return self._cache.get(key, default)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
in get
reviewboard-1 | return self._run_cmd("get", key, default,
default=default, **kwargs)
reviewboard-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322,
in _run_cmd
reviewboard-1 | return self._safely_run_func(client, func,
default_val, *args, **kwargs)
reviewboard-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 199,
in _safely_run_func
reviewboard-1 | result = func(*args, **kwargs)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687,
in get
reviewboard-1 | return self._fetch_cmd(b"get", [key], False,
key_prefix=self.key_prefix).get(
reviewboard-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133,
in _fetch_cmd
reviewboard-1 | self._connect()
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424,
in _connect
reviewboard-1 | sock.connect(sockaddr)
reviewboard-1 | ConnectionRefusedError: [Errno 111] Connection refused
reviewboard-1 | 2024-03-14 10:36:27,128 - DEBUG - -
pymemcache.client.hash - marking server as dead: ('127.0.0.1', 11211)
reviewboard-1 | 2024-03-14 10:36:27,129 - ERROR - -
djblets.cache.synchronizer - Unexpected error checking for expiration in
cached synchronization state key
"{OUR_DOMAIN}:/:extensionmgr:reviewboard.extensions:gen". Is the cache
server down? Error = [Errno 111] Connection refused
reviewboard-1 | Traceback (most recent call last):
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
82, in is_expired
reviewboard-1 | sync_gen = self._get_latest_sync_gen()
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
162, in _get_latest_sync_gen
reviewboard-1 | return cache.get(self.cache_key)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
line 77, in get
reviewboard-1 | return self._cache.get(key, default)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
in get
reviewboard-1 | return self._run_cmd("get", key, default,
default=default, **kwargs)
reviewboard-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322,
in _run_cmd
reviewboard-1 | return self._safely_run_func(client, func,
default_val, *args, **kwargs)
reviewboard-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 211,
in _safely_run_func
reviewboard-1 | result = func(*args, **kwargs)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687,
in get
reviewboard-1 | return self._fetch_cmd(b"get", [key], False,
key_prefix=self.key_prefix).get(
reviewboard-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133,
in _fetch_cmd
reviewboard-1 | self._connect()
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424,
in _connect
reviewboard-1 | sock.connect(sockaddr)
reviewboard-1 | ConnectionRefusedError: [Errno 111] Connection refused
reviewboard-1 | 2024-03-14 10:36:27,131 - ERROR - -
djblets.cache.synchronizer - Unexpected error checking for expiration in
cached synchronization state key
"{OUR_DOMAIN}:/:integrationmgr:djblets.integrations.manager.IntegrationManager:gen".
Is the cache server down? Error = All servers seem to be down right now
reviewboard-1 | Traceback (most recent call last):
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
82, in is_expired
reviewboard-1 | sync_gen = self._get_latest_sync_gen()
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line
162, in _get_latest_sync_gen
reviewboard-1 | return cache.get(self.cache_key)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py",
line 77, in get
reviewboard-1 | return self._cache.get(key, default)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347,
in get
reviewboard-1 | return self._run_cmd("get", key, default,
default=default, **kwargs)
reviewboard-1 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 314,
in _run_cmd
reviewboard-1 | client = self._get_client(key)
reviewboard-1 | ^^^^^^^^^^^^^^^^^^^^^
reviewboard-1 | File
"/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 182,
in _get_client
reviewboard-1 | raise MemcacheError("All servers seem to be down right
now")'
This won't work, because memcached is running in separate container in
compose setup. But in settings-local.py the address is correct:
# cat sitedir/conf/settings_local.py
...
# Cache backend settings.
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache",
"LOCATION": "memcached:11211"
}
}
...
And there is no 127.0.0.1:11211 anywhere in settings, so I assume this is
default value that somehow ended up in the cache server pool.
How to fix this?
The problem is frequent, basically 50% page loads fail due to this.
Thanks in advance,
Best regards,
Szymon
--
Supercharge your Review Board with Power Pack:
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons:
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to reviewboard+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/reviewboard/f083ab4c-cea6-4eda-aa17-0aa9e28b6239n%40googlegroups.com.
