Brian - You didn't describe the exact symptoms you're seeing. "Deadlock" has a particular technical meaning around multiple processes requesting locks mutually off each other, and normally ends in one process being terminated. But I presume you're experiencing more of a "standstill" and future requests no longer work?
Additionally you don't describe your Python version. Since logging is part of the Python standard library (and you're not using any extensions) any logging issues you're seeing are part of Python, and may be fixed on later versions. Minor versions contain lots of such fixes and Django only officially supports the latest in each series. Steven has managed to test with 1000's of requests and I think that's an indicator it could be a Python bug. Also Steven is right, writing to disk is expensive and something to avoid in production (at scale). In production I've previously used two high quality setups: 1. Logging to stdout/stderr, running the server under systemd, and letting it capture stdout logs into journald 2. Logging directly into a running fluentd process Both logging agents - journald and fluentd - provide in-memory buffering before disk, discarding of extreme bursts, etc. Journald is easier to get started with since it's included in most linuxes, fluentd is more flexible and easy to configure. There are many others out there too! And if you use a PaaS like Heroku, they normally capture your stdout/stderr into such a logging agent already. Hope that helps, Adam On Wed, 22 Apr 2020 at 19:47, Steven Mapes <[email protected]> wrote: > This is more an issue at the file system level and the hardware not being > able to keep up rather than anything Django is doing. It's the same > fundamental principal for why you may turn off webserver logging to > increase performance, writing to disk is expensive and so when dealing with > high throughout you probably want to question why. > > I've just run that on my laptop and hit your 1000 requests fine. I > actually them bumped it up to run 6 requests per iteration and still was > fine writing to disk. I started to have network resource issues when I > bumped it up further but that was more limitations within chrome > > In the real world if you were writing logs like this for information or > debugging you would be best served to defer them by writing to a message > queue of sort some, probably FIFO based as its logging, and then have as > many subscribers as you want dealing with reading and processing those > queued messages. This could be as few as one so that you never hit a > deadlock or it could be a few, again it depends on the performance of the > underlying hardware, what you are writing and where (database, disk, > memory). > > > On Wednesday, 22 April 2020 18:23:31 UTC+1, Brian Tiemann wrote: >> >> Hi all, >> >> I was directed here after getting corroboration on #django and elsewhere. >> >> I have what appears to be a bug in which a Django app can deadlock if you >> hit it with a lot (>3) of AJAX requests within a short time (i.e. all >> triggered in parallel from a single HTML page). I have a reproducible case >> here: >> >> https://github.com/data-graham/wedge >> >> I'd really appreciate it if someone could take a look and let me know >> whether I'm doing something wrong, or if there's something systemic going >> on. Thanks much! >> >> -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/1445d05d-eefc-43b1-8913-79d6e55de216%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/1445d05d-eefc-43b1-8913-79d6e55de216%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Adam -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM3dM2B%2BhKTVEgUA458G0FO%2BWhU1%3Dy_VDY5P7e2OTis1mw%40mail.gmail.com.
