Henrik Rasmussen via Mailman-users writes: > I have been chatting with Copilot AI
Be careful with that. I can't speak to Copilot, but I assure you that Gemini and Claude both gave a lot of crappy advice about Mailman 3 over the last couple of years (the latest most expensive versions may be better). There was a lot of confusion among Mailman 2 and Mailman 3. > for some days to get help investigating some logging-related issues > in a Mailman 3.3.4 Docker installation > We are working on a long needed upgrade, I assume you intend to continue with a containerized Mailman 3. In that case, my understanding is that you want to upgrade Python to 3.13 right now. But once a couple of dependencies that don't work with Python 3.14 are upgraded upstream, 3.14 is probably the way to go (you may be more conservative than that though). I don't know how likely 3.3.11 is to support 3.14. Experience suggests we surely won't be able to support 3.15 for a couple of releases. > but for now I need the logs in Elasticsearch, You mean dumping the Mailman logs into Elasticsearch, not using Elasticsearch for the archives, right? > and they are aweful. I don't know any logs produced by concurrent server systems that aren't, to be honest. Even Postfix, which seems to be a pretty well- designed logging system, is a real pain to analyze, especially if you enable debug output. I sympathize with your desire for a consistent date format, but to get that you probably need to conform Mailman's logging to gunicorn's. Gunicorn (the WSGI server for the REST API) allows much flexibility in selecting different information to add to each log message, but I don't recall any in formatting the individual fields. > Environment > > * GNU Mailman 3.3.4 (Tom Sawyer) > * Python 3.8.10 > * Docker-based installation > * Test environment with a single mailing list: > [email protected]<mailto:[email protected]> > > Logging configuration > > I've configured dedicated log files for most logging channels. For > example, [logging.http] is configured with format: %(asctime)s > (%(process)d) %(message)s, datefmt: %Y-%m-%d %H:%M:%S.%3N, > propagate: no, level: debug, and path: http.log. You can configure all configurable logging to defaults, then override only the non-default attributes (I didn't look carefully, but it seems like all of your settings are identical except for path and level). Using the [logging.template] section described below (mailman/config/schema.cfg has more details) may give you a little more consistency in date formatting. From schema.cfg: ------------------------------------------------------------------------ [logging.template] # This defines various log settings. The options available are: # # - level -- Overrides the default level; this may be any of the # standard Python logging levels, case insensitive. # - format -- Overrides the default format string # - datefmt -- Overrides the default date format string # - path -- Overrides the default logger path. This may be a relative # path name, in which case it is relative to Mailman's LOG_DIR, # or it may be an absolute path name. You cannot change the # handler class that will be used. # - propagate -- Boolean specifying whether to propagate log message from this # logger to the root "mailman" logger. You cannot override # settings for the root logger. # # In this section, you can define defaults for all loggers, which will be # prefixed by 'mailman.'. ------------------------------------------------------------------------ I guess your logging.template would look like [logging.template] format: %(asctime)s (%(process)d) %(message)s datefmt: %Y-%m-%d %H:%M:%S.%3N propagate: no level: debug # some sections should override to "info" Then use subsections as in your current configuration to override the path, and occasionally level, settings for specific loggers. Somebody added a [logging.gunicorn] section, but did not add it to that list. The date format is gunicorn-specific. I don't know if there's a way to change it, there's no documentation beyond the default setting itself in schema.cfg for logging.gunicorn. (Gunicorn's own documentation is quite good, though.) The log message format (the syntax is also gunicorn-specific, and not generic logging format syntax) is [logging.gunicorn] format: %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s" That won't help in 3.3.4, though. > Issue 1: Mixed log formats > This suggests that multiple logging systems may be writing to the > same log file. Pretty sure all Mailman modules use the stdlib logging module, but I'm not sure about gunicorn. Four possibilities for inconsistency: 1. You missed configuring some particular logger (unlikely but possible) 2. Date format is hard-coded in some module 3. Bug in parsing the config file or updating the config object 4. REST API logging comes from gunicorn. You may need to configure this in gunicorn.conf (in the same place as mailman.cfg). > Issue 2: Runner debug noise > > With logging level set to debug, mailman.log contains a large > number of entries such as: [omitted] > Is there a recommended way to suppress these internal runner loop > messages while retaining useful operational logging? "level: info" is the only standard way. If that is not detailed enough, you'll need to customize the code or filter using "grep -v" or similar. > Issue 3: HTTP request logging > > I intentionally keep [logging.http] at level: debug because I want > to see which REST endpoints are called. > My goal is to keep these request entries. I don't understand where the problem for your current 3.3.4 lies. Just keep "level: debug", no? For current 3.3.10, [logging.http] has "level: info" and generates all API requests like [24/Sep/2025:00:00:10 +0000] "GET /3.1/lists/[email protected] HTTP/1.1" 200 368 "-" "GNU Mailman REST client v3.3.5" Pretty sure the slashes and the brackets come from gunicorn's "%(t)s" format. > Issue 4: Falcon tracebacks in http.log > > The same http.log also contains Falcon tracebacks. One example is: > Is it possible to: > > 1. Continue logging HTTP GET requests to http.log > 2. Send Falcon tracebacks and REST API errors to a separate log file For 3.3.4, you'll have to customize the relevant loggers, but I don't think there is provision for this in the config module. You can reconfigure Falcon once and for all using standard logging APIs. Splitting GET requests from REST API debugging would need to be done at the call sites. For 3.3.10, you can't split, but you can inhibit debug output. I'm not sure what happens with "level: info" when an exception is raised. Probably that goes to the [logging.http] file, though, as exceptions are high priority events. > At the moment, both request logging and tracebacks appear in the > same file. Are you sure you want to split them? Since Mailman usually works OK, generally we want to know what requests cause exceptions and tracebacks. > Issue 5: Worker timeout > > After the traceback above, I consistently see: > > * [2026-08-03 12:21:29 +0000] [34] [CRITICAL] WORKER TIMEOUT (pid:52) > * [2026-08-03 12:21:30 +0000] [53] [INFO] Booting worker with pid: 53 > > This appears to indicate that the worker handling GET > /3.1/lists/[email protected]/config<mailto:/3.1/lists/[email protected]/config> > times out and is restarted. Yes. That's by design, since the workers are a different process from where the exception is raised, and the IPC socket is supposed to be persistent for the whole session, but it goes idle without being closed. > However, the Falcon/REST traceback shown above does not appear in error.log. > > This suggests that: > > * error.log receives Mailman Core exceptions. > * http.log receives REST API/Falcon exceptions. > * The two do not appear to be linked. That is correct. The REST API runs in a gunicorn process, separate from the other Mailman core runners. > Questions > > 1. Is the mixed timestamp formatting expected in Mailman 3.3.4 > Docker deployments? Yes. This is a bug, but it's not high priority. > 2. Which component produces the starting oneloop / ending > oneloop runner messages? mailman/core/runner.py > 3. Can HTTP request logging and Falcon tracebacks be directed to > separate log files? See above. > 4. Is ResourceClosedError when accessing GET > > /3.1/lists/[email protected]/config<mailto:/3.1/lists/[email protected]/config> > a known Mailman 3.3.4 issue? That error is not raised by Mailman, but by some other package. Mailman 3.3.4 is over 5 years old now. It is quite likely it has been fixed either by that package's maintainer, or "en passant" in some Mailman fix. > 5. Is there a recommended logging configuration that preserves > REST API access logging while reducing runner/debug noise? "level: info" where "level: debug" is considered too noisy. If you don't like that, you can submit a specific RFE, but as you can see from where I pushback on your requests, opinions vary on logging. I rather doubt that Mailman admins will fracture cleanly into two factions, one agreeing with you and the other with me. ;-) So it could be a long messy discussion about how to factor literally hundreds of individual logging calls. Regards, Steve -- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan _______________________________________________ Mailman-users mailing list -- [email protected] To unsubscribe send an email to [email protected] https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/[email protected]/message/S3ZVKZBCUPJFIC3ZMGIM7GK47DTWY3SM/ This message sent to [email protected]
