Re: v2.3.19.1 released

Mon, 20 Jun 2022 00:05:20 -0700 

Hi Aki,

Am 14.06.22 um 12:24 schrieb Aki Tuomi:

Hi everyone!

Due to a severe bug in doveadm deduplicate, we are releasing patch release 
2.3.19.1. Please find it at locations below:

https://dovecot.org/releases/2.3/dovecot-2.3.19.1.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.19.1.tar.gz.sig
Binary packages in https://repo.dovecot.org/
Docker images in https://hub.docker.com/r/dovecot/dovecot

Aki Tuomi
Open-Xchange oy

---

- doveadm deduplicate: Non-duplicate mails were deleted. v2.3.19 regression.
- auth: Crash would occur when iterating multiple backends.
   Fixes: Panic: file userdb-blocking.c: line 125 (userdb_blocking_iter_next): 
assertion failed: (ctx->conn != NULL)
As the above Panic is fixed I tried again (see my attached mail to the 2.3.19 release) and I can confirm to no longer get the Panic, BUT authentication is NOT working either :(
Reverting back to a container with Dovecot 2.3.16, get's everything working again.
We use a hourly updated local SQLight database and a dict for user- and passdb.
Is the usage of multiple backends no longer supported, or did something in that regard changed between 2.3.16 and 2.3.19.1? 

Here's the relevant part of my config (full doveadm config -n is attached):

userdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
userdb {
  args = /etc/dovecot/dovecot-dict-auth.conf
  driver = dict
}
passdb {
  args = /etc/dovecot/dovecot-dict-master-auth.conf
  driver = dict
  master = yes
}
passdb {
  args = /etc/dovecot/dovecot-dict-auth.conf
  driver = dict
}

The SQLight DB is used for listing all users and to keep the replication 
running, even if the dict is unavailable.

Any ideas what might be the cause or how to narrow the problem down?

Ralf

--
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Geschäftsführer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0
--- Begin Message --- 
After updating to 2.3.19 (from 2.3.16) passdb and userdb lookups fail:

root@backup:~# doveadm user r...@egroupware.org; doveadm log errors

userdb lookup: user r...@egroupware.org doesn't exist
field    value
May 15 07:22:18 Panic: auth: file userdb-blocking.c: line 124 (userdb_blocking_iter_next): assertion failed: (ctx->conn != NULL) May 15 07:22:18 Error: auth: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x41) [0x7f019a651c91] -> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x22) [0x7f019a651db2] -> /usr/lib/dovecot/libdovecot.so.0(+0x10b0bb) [0x7f019a65f0bb] -> /usr/lib/dovecot/libdovecot.so.0(+0x10b157) [0x7f019a65f157] -> /usr/lib/dovecot/libdovecot.so.0(+0x5d375) [0x7f019a5b1375] -> dovecot/auth [0 wait, 0 passdb, 0 userdb](+0x157a7) [0x55e256d287a7] -> dovecot/auth [0 wait, 0 passdb, 0 userdb](+0x1954b) [0x55e256d2c54b] -> dovecot/auth [0 wait, 0 passdb, 0 userdb](+0x36ca7) [0x55e256d49ca7] -> dovecot/auth [0 wait, 0 passdb, 0 userdb](+0x2ab86) [0x55e256d3db86] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0x15f) [0x7f019a67576f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xcf) [0x7f019a67702f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x54) [0x7f019a675a54] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7f019a675bc0] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x17) [0x7f019a5e7207] -> dovecot/auth [0 wait, 0 passdb, 0 userdb](main+0x3c8) [0x55e256d29588] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f019a2de0b3] -> dovecot/auth [0 wait, 0 passdb, 0 userdb](_start+0x2e) [0x55e256d2976e] May 15 07:22:19 Fatal: auth: master: service(auth): child 19 killed with signal 6 (core dumped) May 15 07:22:19 Error: replicator: auth-master: userdb list: Disconnected unexpectedly May 15 07:22:19 Error: replicator: listing users failed, can't replicate existing data May 15 07:22:19 Error: doveadm(arash 2stud...@bb-trunk.egroupware.de): User doesn't exist May 15 07:22:19 Error: doveadm(arash teac...@bb-trunk.egroupware.de): User doesn't exist May 15 07:22:20 Error: doveadm(christoph thys...@bb-trunk.egroupware.de): User doesn't exist May 15 07:23:21 Error: doveadm(arash stud...@bb-trunk.egroupware.de): User doesn't exist May 15 07:24:02 Error: doveadm(schie...@uni-kl.de@bb-trunk.egroupware.de): User doesn't exist May 15 07:24:07 Error: doveadm(sab...@uni-kl.de@bb-trunk.egroupware.de): User doesn't exist May 15 07:24:24 Error: doveadm(ralf.imapt...@outdoor-training.de@bb-trunk.egroupware.de): User doesn't exist May 15 07:24:31 Error: doveadm(arash to...@bb-trunk.egroupware.de): User doesn't exist May 15 07:24:31 Error: doveadm(becke...@uni-kl.de@bb-trunk.egroupware.de): User doesn't exist May 15 07:24:49 Error: doveadm(olat.vcrp.de:2723414...@bb-trunk.egroupware.de): User doesn't exist May 15 07:24:56 Error: doveadm(olat.vcrp.de:1167852...@bb-trunk.egroupware.de): User doesn't exist 

Reverting back to 2.3.16 fixes the problem for now.
My doveadm config -n is attached. We use a hourly updated local sqlight database and a dict for userdb. 

Any ideas?

Ralf


Am 10.05.22 um 08:33 schrieb Aki Tuomi:

Hi all!

We are pleased to release v2.3.19 of Dovecot.

The docker images have been upgraded to use bullseye as base image.

https://dovecot.org/releases/2.3/dovecot-2.3.19.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.19.tar.gz.sig
Binary packages in https://repo.dovecot.org/
Docker images in https://hub.docker.com/r/dovecot/dovecot

Regards,
Aki Tuomi
Open-Xchange oy

--

+ Added mail_user_session_finished event, which is emitted when the mail
   user session is finished (e.g. imap, pop3, lmtp). It also includes
   fields with some process statistics information.
   See https://doc.dovecot.org/admin_manual/list_of_events/ for more
   information.
+ Added process_shutdown_filter setting. When an event matches the filter,
   the process will be shutdown after the current connection(s) have
   finished. This is intended to reduce memory usage of long-running imap
   processes that keep a lot of memory allocated instead of freeing it to
   the OS.
+ auth: Add cache hit indicator to auth passdb/userdb finished events.
   See https://doc.dovecot.org/admin_manual/list_of_events/ for more
   information.
+ doveadm deduplicate: Performance is improved significantly.
+ imapc: COPY commands were sent one mail at a time to the remote IMAP
   server. Now the copying is buffered, so multiple mails can be copied
   with a single COPY command.
+ lib-lua: Add a Lua interface to Dovecot's HTTP client library. See
   https://doc.dovecot.org/admin_manual/lua/ for more information.
- auth: Cache lookup would use incorrect cache key after username change.
- auth: Improve handling unexpected LDAP connection errors/hangs.
   Try to fix up these cases by reconnecting to the LDAP server and
   aborting LDAP requests earlier.
- auth: Process crashed if userdb iteration was attempted while auth-workers
   were already full handling auth requests.
- auth: db-oauth2: Using %{oauth2:name} variables caused unnecessary
   introspection requests.
- dict: Timeouts may have been leaked at deinit.
- director: Ring may have become unstable if a backend's tag was changed.
   It could also have caused director process to crash.
- doveadm kick: Numeric parameter was treated as IP address.
- doveadm: Proxying can panic when flushing print output. Fixes
   Panic: file ioloop.c: line 865 (io_loop_destroy): assertion failed:
   (ioloop == current_ioloop).
- doveadm sync: BROKENCHAR was wrongly changed to '_' character when
   migrating mailboxes. This was set by default to %, so any mailbox
   names containing % characters were modified to "_25".
- imapc: Copying or moving mails with doveadm to an imapc mailbox could
   have produced "Error: Syncing mailbox '[...]' failed" Errors. The
   operation itself succeeded but attempting to sync the destination
   mailbox failed.
- imapc: Prevent index log synchronization errors when two or more imapc
   sessions are adding messages to the same mailbox index files, i.e.
   INDEX=MEMORY is not used.
- indexer: Process was slowly leaking memory for each indexing request.
- lib-fts: fts header filters caused binary content to be sent to the
   indexer with non-default configuration.
- doveadm-server: Process could hang in some situations when printing
   output to TCP client, e.g. when printing doveadm sync state.
- lib-index: dovecot.index.log files were often read and parsed entirely,
   rather than only the parts that were actually necessary. This mainly
   increased CPU usage.
- lmtp-proxy: Session ID forwarding would cause same session IDs being
   used when delivering same mail to multiple backends.
- log: Log prefix update may have been lost if log process was busy.
   This could have caused log prefixes to be empty or in some cases
   reused between sessions, i.e. log lines could have been logged for the
   wrong user/session.
- mail_crypt: Plugin crashes if it's loaded only for some users. Fixes
   Panic: Module context mail_crypt_user_module missing.
- mail_crypt: When LMTP was delivering mails to both recipients with mail
   encryption enabled and not enabled, the non-encrypted recipients may
   have gotten mails encrypted anyway. This happened when the first
   recipient was encrypted (mail_crypt_save_version=2) and the 2nd
   recipient was not encrypted (mail_crypt_save_version=0).
- pop3: Session would crash if empty line was sent.
- stats: HTTP server leaked memory.
- submission-login: Long credentials, such as OAUTH2 tokens, were refused
   during SASL interactive due to submission server applying line length
   limits.
- submission-login: When proxying to remote host, authentication was not
   using interactive SASL when logging in using long credentials such as
   OAUTH2 tokens. This caused authentication to fail due to line length
   constraints in SMTP protocol.
- submission: Terminating the client connection with QUIT command after
   mail transaction is started with MAIL command and before it is
   finished with DATA/BDAT can cause a segfault crash.
- virtual: doveadm search queries with mailbox-guid as the only parameter
   crashes: Panic: file virtual-search.c: line 77 (virtual_search_get_records):
   assertion failed: (result != 0)



--
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Geschäftsführer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0

# 2.3.19 (b3ad6004dc): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.19 (4eae2f79)
# OS: Linux 4.15.0-176-generic x86_64 Ubuntu 20.04.4 LTS 
# Hostname: f7cd89ea62ff
auth_cache_negative_ttl = 2 mins
auth_cache_size = 10 M
auth_cache_ttl = 5 mins
auth_master_user_separator = *
auth_mechanisms = plain login
auth_username_chars = 
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
default_client_limit = 3500
default_process_limit = 512
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
first_valid_uid = 90
listen = *
log_path = /dev/stderr
login_greeting = Dovecot KA.nfs ready
mail_access_groups = dovecot
mail_attribute_dict = file:%h/dovecot-metadata
mail_gid = dovecot
mail_location = mdbox:~/mdbox
mail_log_prefix = "%s(%u %p): "
mail_max_userip_connections = 200
mail_plugins = acl quota notify replication mail_log mail_lua notify 
push_notification push_notification_lua
mail_uid = dovecot
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date ihave vnd.dovecot.debug
mbox_min_index_size = 1000 B
mbox_write_locks = fcntl
mdbox_rotate_size = 50 M
namespace inboxes {
  inbox = yes
  location = 
  mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Junk {
    auto = subscribe
    special_use = \Junk
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox Templates {
    auto = subscribe
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  subscriptions = no
}
namespace subs {
  hidden = yes
  list = no
  location = 
  prefix = 
  separator = /
}
namespace users {
  location = mdbox:%%h/mdbox
  prefix = user/%%n/
  separator = /
  subscriptions = no
  type = shared
}
passdb {
  args = /etc/dovecot/dovecot-dict-master-auth.conf
  driver = dict
  master = yes
}
passdb {
  args = /etc/dovecot/dovecot-dict-auth.conf
  driver = dict
}
plugin {
  acl = vfile
  acl_shared_dict = file:/var/dovecot/imap/%d/shared-mailboxes.db
  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
  mail_log_fields = uid box msgid size
  mail_replica = tcp:10.44.88.5
  push_lua_url = http://push-proxy/
  push_notification_driver = lua:file=/etc/dovecot/dovecot-push.lua
  quota = dict:User quota::ns=INBOX/:file:%h/dovecot-quota
  quota_rule = *:storage=200GB
  sieve = ~/sieve/dovecot.sieve
  sieve_after = /var/dovecot/sieve/after.d/
  sieve_before = /var/dovecot/sieve/before.d/
  sieve_dir = ~/sieve
  sieve_extensions = +editheader
  sieve_user_log = ~/.sieve.log
}
postmaster_address = adm...@egroupware.org
protocols = imap pop3 lmtp sieve
quota_full_tempfail = yes
replication_dsync_parameters = -d -n INBOX -l 30 -U
service aggregator {
  fifo_listener replication-notify-fifo {
    user = dovecot
  }
  unix_listener replication-notify {
    user = dovecot
  }
}
service auth-worker {
  user = $default_internal_user
}
service auth {
  drop_priv_before_exec = no
  inet_listener {
    port = 113
  }
}
service doveadm {
  inet_listener {
    port = 12345
  }
  inet_listener {
    port = 26
  }
  vsz_limit = 640 M
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 5
  service_count = 1
  vsz_limit = 64 M
}
service imap {
  executable = imap
  process_limit = 2048
  vsz_limit = 640 M
}
service lmtp {
  inet_listener lmtp {
    port = 24
  }
  unix_listener lmtp {
    mode = 0666
  }
  vsz_limit = 512 M
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  inet_listener sieve_deprecated {
    port = 2000
  }
}
service pop3-login {
  inet_listener pop3 {
    port = 110
  }
  inet_listener pop3s {
    port = 995
    ssl = yes
  }
}
service pop3 {
  executable = pop3
}
service postlogin {
  executable = script-login -d rawlog -b -t
}
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    group = dovecot
    mode = 0660
    user = dovecot
  }
}
ssl_cert = </etc/certs/mail.egroupware.org.pem
ssl_cipher_list = 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
userdb {
  args = /etc/dovecot/dovecot-dict-auth.conf
  driver = dict
}
verbose_proctitle = yes
protocol lda {
  mail_plugins = acl quota notify replication mail_log mail_lua notify 
push_notification push_notification_lua acl sieve quota
}
protocol imap {
  imap_metadata = yes
  mail_max_userip_connections = 200
  mail_plugins = acl quota notify replication mail_log mail_lua notify 
push_notification push_notification_lua acl imap_acl quota imap_quota
}
protocol lmtp {
  mail_max_lock_timeout = 25 secs
  mail_plugins = acl quota notify replication mail_log mail_lua notify 
push_notification push_notification_lua acl sieve quota notify push_notification
}

--- End Message ---

Attachment: doveadm-config-n
Description: application/unknown-content-type

Reply via email to