Re: mdbox / selecting folder accesses hundrets of others
> On May 8, 2021, at 7:08 AM, Florian Lohoff wrote: > > On Fri, May 07, 2021 at 06:15:55PM +0200, Florian Lohoff wrote: >> >> Dovecot: 2.3.4 >> >> Hi, >> i am debugging a problem where after a force-resync removed all cache >> files (All files are 44 byte). Trying to open a folder takes literally >> minutes >> (Huge mailbox - 220K folder, 45GByte compressed storage) I don’t have an answer for you, unfortunately, as you all exercise your mail system far more thanI do. Taking a wild guess, I’d think that having a large number of folders has potential to be more of an issue than having a large amount of mail in fewer folders. Operating under that assumption, is it possible to bound the problem? (Hypothetically say 128k folders were good, more than that and something went out to lunch or something similar.) I am, however, watching this with interest. -- Coy Hile coy.h...@coyhile.com signature.asc Description: Message signed with OpenPGP
Re: Current thinking on backups ?
This all gives me an interesting idea. Currently, I only have one box running dovecot doing IMAP duties. For other systems where I want availability (specifically, postgres databases), I use manatee (https://github.com/joyent/manatee). I wonder if something similar would work to have dovecot instances bootstrap themselves: - discover existing instances via whatever method. - zfs send / zfs recv a snapshot from one of the existing primaries - setup replication from one of the existing boxes. It’s perhaps not quite as straightforward as a database that already has native definitions of “primary”, “synchronous peer”, and “asynchronous peer” Is this something that people see as among the land of the possible or desired? -c > On May 29, 2020, at 10:34 PM, deano-dove...@areyes.com wrote: > > I run a pair of dovecot servers for personal small domains with several > layers of backup in place ... > > - The two dovecot servers replicate to each via a Tinc vpn mesh. That gives > email resiliency. > - All mail is replicated via offlineimap to a 3rd server over that Tinc vpn. > It's on the mesh, it has space, so why not ? > - All mail is replicated as well as via mbsync to a zfs dataset on my main > media server at home once an hour. > - That zfs dataset (and others) is snapshot'd hourly, and zfs send/recv to a > backup box nightly. > > Outside of dovecot procedures, I find mbsync to work extremely well. It was > easy enough to set up a systemd timer and service to pull the mail down. > > > mysync.timer > > # Run the mbsync process to sync mail down to local mediabox > > [Unit] > Description=mbsync timer > ConditionPathExists=%h/.mbsyncrc > ConditionPathIsDirectory=/stuff/Backups/Mailsystems/mbsync-backups > > [Timer] > OnBootSec=15m > OnCalendar=hourly > Persistent=true > > [Install] > WantedBy=timers.target > > > mysync.service > == > # mbsync service > > [Unit] > Description=mbsync backup from mailsystems > ConditionPathExists=%h/.mbsyncrc > ConditionPathIsDirectory=/stuff/Backups/Mailsystems/mbsync-backups > > [Service] > Type=oneshot > ExecStart=/usr/local/bin/mbsync backup > > [Install] > WantedBy=default.target > > > "backup" is the mbsync group that includes all the defined channels that > determine what should be backed up. Transparent. In the background. Don't > have to think about it, it's just there. > > I've done test restores to test environments via mbsync, and it all works > flawlessly. > > > On 2020-05-26 12:31 am, Germain Le Chapelain wrote: >>> Le 24 mai 2020 à 14:42, Laura Smith a >>> écrit : >>> Hi, >>> What are people doing for backups ? >>> My current process is LVM snapshot and backup from that to NFS share. >>> But there seems to be hints around the internet that people use/abuse >>> "doveadm backup" for backup purposes even though it seems its original >>> intention was for transferring mailboxes between dovecot instances. >>> Assuming its ok to "doveadm backup" to an NFS share, is it ok to use >>> "doveadm backup" when dovecot has replication setup (replication-notify >>> etc.) ? Or will it interfere ? >>> Thanks! >>> Laura >> This has came up in the past: >> https://dovecot.org/pipermail/dovecot/2020-February/thread.html#118206 >> I ended up developing my own system based on forwarding all emails to >> a program (from which I back-up as they come in.) >> I am hoping if disaster and/or misfortune were to strike my server, I >> could simply cat >> back all those files in order (or not come to >> think of it) in the /var/mail/ (or somewhere even better fit >> in Postfix.) >> I am not interested in saving the state of the mailbox as much as all >> the mails that ever come in (or go out.) > > -- > Dean Carpenter > deano is at areyes dot com > 203 six oh four 6644 -- Coy Hile coy.h...@coyhile.com
Re: NFS vs Replication
> On Jul 15, 2020, at 12:33 PM, Andrea Gabellini > wrote: > > Hello list, > > I built an email system using a proxy / director pair (IMAP, POP3, LMTP) > and a backend pair. > > To have an HA system, I would like to understand if it is better to use > an NFS export or replication to save emails and index files > > NFS is provided by a NAS (in HA), while for replication I would use the > local backend disks > > Which of the two systems is more reliable? Are there any drawbacks for > one or the other? > The biggest problem with using NFS is that you’re using NFS and bringing along all the baggage that comes with it. Writes over the network are inherently slower than writes to local storage, plus locking gets interesting, to say the least. I posted a while back about using something similar to Joyent's manatee to bootstrap replication. (If IMAP replication works anything like databases, a system could join the cluster, get a base state by streaming a ZFS snapshot of an existing peer to the new peer, and from there it catches up via the normal replication mechanisms.) I don’t know if that would be feasible, but it’s certainly something I might try to make work. I also don’t know whether it gets more dicey in a multiple-primary situation.) But long and short of it. Avoid NFS if you can. The last time I used NFS for mail was last century, and even with everybody using native *nix MUAs like pine and elm, we could run into fun locking issues. -- Coy Hile coy.h...@coyhile.com
LDAP virtual user username being rewritten:
sl_authz_id =
# Use TLS to connect to the LDAP server.
#tls = no
# TLS options, currently supported only with OpenLDAP:
#tls_ca_cert_file =
#tls_ca_cert_dir =
#tls_cipher_suite =
# TLS cert/key is used only if LDAP server requires a client certificate.
#tls_cert_file =
#tls_key_file =
# Valid values: never, hard, demand, allow, try
#tls_require_cert =
# Use the given ldaprc path.
#ldaprc_path =
# LDAP library debug level as specified by LDAP_DEBUG_* in ldap_log.h.
# -1 = everything. You may need to recompile OpenLDAP with debugging enabled
# to get enough output.
#debug_level = 0
# Use authentication binding for verifying password's validity. This works by
# logging into LDAP server using the username and password given by client.
# The pass_filter is used to find the DN for the user. Note that the pass_attrs
# is still used, only the password field is ignored in it. Before doing any
# search, the binding is switched back to the default DN.
#auth_bind = no
# If authentication binding is used, you can save one LDAP request per login
# if users' DN can be specified with a common template. The template can use
# the standard %variables (see user_filter). Note that you can't
# use any pass_attrs if you use this setting.
#
# If you use this setting, it's a good idea to use a different
# dovecot-ldap.conf.ext for userdb (it can even be a symlink, just as long as
# the filename is different in userdb's args). That way one connection is used
# only for LDAP binds and another connection is used for user lookups.
# Otherwise the binding is changed to the default DN before each user lookup.
#
# For example:
# auth_bind_userdn = cn=%u,ou=people,o=org
#
#auth_bind_userdn =
# LDAP protocol version to use. Likely 2 or 3.
#ldap_version = 3
# LDAP base. %variables can be used here.
# For example: dc=mail, dc=example, dc=org
base = ou=People,dc=coyhile,dc=com
# Dereference: never, searching, finding, always
#deref = never
# Search scope: base, onelevel, subtree
#scope = subtree
scope = subtree
# User attributes are given in LDAP-name=dovecot-internal-name list. The
# internal names are:
# uid - System UID
# gid - System GID
# home - Home directory
# mail - Mail location
#
# There are also other special fields which can be returned, see
# http://wiki2.dovecot.org/UserDatabase/ExtraFields
#user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid
# Filter for user lookup. Some variables can be used (see
# http://wiki2.dovecot.org/Variables for full list):
# %u - username
# %n - user part in user@domain, same as %u if there's no domain
# %d - domain part in user@domain, empty if user there's no domain
#user_filter = (&(objectClass=posixAccount)(uid=%u))
user_filter = (&(objectClass=posixAccount)(uid=%u))
# Password checking attributes:
# user: Virtual user name (user@domain), if you wish to change the
#user-given username to something else
# password: Password, may optionally start with {type}, eg. {crypt}
# There are also other special fields which can be returned, see
# http://wiki2.dovecot.org/PasswordDatabase/ExtraFields
#pass_attrs = uid=user,userPassword=password
# If you wish to avoid two LDAP lookups (passdb + userdb), you can use
# userdb prefetch instead of userdb ldap in dovecot.conf. In that case you'll
# also have to include user_attrs in pass_attrs field prefixed with "userdb_"
# string. For example:
#pass_attrs = uid=user,userPassword=password,\
# homeDirectory=userdb_home,uidNumber=userdb_uid,gidNumber=userdb_gid
# Filter for password lookups
#pass_filter = (&(objectClass=posixAccount)(uid=%u))
pass_filter = (&(objectClass=posixAccount)(uid=%n))
# Attributes and filter to get a list of all users
#iterate_attrs = uid=user
iterate_attrs = maildrop=user
iterate_filter = (objectClass=posixAccount)
# Default password scheme. "{scheme}" before password overrides this.
# List of supported schemes is in: http://wiki2.dovecot.org/Authentication
#default_pass_scheme = CRYPT
# By default all LDAP lookups are performed by the auth master process.
# If blocking=yes, auth worker processes are used to perform the lookups.
# Each auth worker process creates its own LDAP connection so this can
# increase parallelism. With blocking=no the auth master process can
# keep 8 requests pipelined for the LDAP connection, while with blocking=yes
# each connection has a maximum of 1 request running. For small systems the
# blocking=no is sufficient and uses less resources.
#blocking = no
root@basement-imap01:/etc/dovecot#
--
Coy Hile
coy.h...@coyhile.com
signature.asc
Description: Message signed with OpenPGP
Re: LDAP virtual user username being rewritten:
Replying to myself…
Empirically, changing the following two lines in the LDAP configuration seems
to have worked:
user_filter = (&(objectClass=posixAccount)(uid=%n))
and
pass_filter = (&(objectClass=posixAccount)(uid=%n))
My question remains, though, why was the original username getting rewritten by
default?
-c
> On May 21, 2022, at 3:18 PM, Coy Hile wrote:
>
> Hi all,
>
> I’m working on migrating a dovecot instance that formerly had its only user
> (me) stored in a SQL database to a new instance that uses LDAP so I don’t
> have to maintain a handful of different passwords. However, I’m ending up
> getting sent to an incorrect mailbox because the %d part of the username I
> enter is getting stripped somewhere along the lines. Dovecot debug logs show
> this happening here:
>
> May 21 18:36:53 auth: Debug: client in: AUTH1 PLAIN service=imap
> secured session=itEO5InfPJusEgKDlip=172.18.2.131
> rip=172.18.2.131lport=143 rport=39740 resp=
> May 21 18:36:53 auth: Debug:
> ldap(h...@coyhile.com,172.18.2.131,): Performing passdb
> lookup
> May 21 18:36:53 auth: Debug:
> ldap(h...@coyhile.com,172.18.2.131,): pass search:
> base=ou=People,dc=coyhile,dc=com scope=subtree
> filter=(&(objectClass=posixAccount)(uid=hile)) fields=uid,userPassword
> May 21 18:36:53 auth: Debug:
> ldap(h...@coyhile.com,172.18.2.131,): result: uid=hile
> userPassword=; uid,userPassword unused
> May 21 18:36:53 auth: Debug:
> ldap(h...@coyhile.com,172.18.2.131,): username changed
> h...@coyhile.com -> hile
> May 21 18:36:53 auth: Debug: ldap(hile,172.18.2.131,):
> Finished passdb lookup
> May 21 18:36:53 auth: Debug: auth(hile,172.18.2.131,): Auth
> request finished
> May 21 18:36:53 auth: Debug: client passdb out: OK 1 user=hile
> original_user=h...@coyhile.com
> May 21 18:36:53 auth: Debug: master in: REQUEST 473956353 94091
> 2c1f94ded4de2c343425c908d094session_pid=9412
> request_auth_token
> May 21 18:36:53 auth: Debug: ldap(hile,172.18.2.131,):
> Performing userdb lookup
> May 21 18:36:53 auth: Debug: ldap(hile,172.18.2.131,): user
> search: base=ou=People,dc=coyhile,dc=com scope=subtree
> filter=(&(objectClass=posixAccount)(uid=hile))
> fields=homeDirectory,uidNumber,gidNumber
> May 21 18:36:53 auth: Debug: ldap(hile,172.18.2.131,):
> result: homeDirectory=/home/hile uidNumber=1 gidNumber=1;
> homeDirectory,uidNumber,gidNumber unused
> May 21 18:36:53 auth: Debug: ldap(hile,172.18.2.131,):
> Finished userdb lookup
> May 21 18:36:53 auth: Debug: master userdb out: USER473956353 hile
> home=/data/mail/vmail//hile uid=998 gid=998
> auth_token=3a262da408d33ce2c51ecb1ddd943203fdbb17a3
> auth_user=h...@coyhile.com
>
>
> Note the “username changed” line above. Clearly I’ve misconfigured something,
> butI am unsure what. My configuration is as follows:
>
> root@basement-imap01:/var/log# doveconf -n
> # 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.7.2 ()
> # OS: Linux 5.4.0-110-generic x86_64 Ubuntu 20.04.4 LTS
> # Hostname: basement-imap01.coyhile.com
> auth_debug = yes
> debug_log_path = /var/log/dovecot-debug.log
> doveadm_password = # hidden, use -P to show it
> mail_location = maildir:~/Maildir
> mail_privileged_group = mail
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
>special_use = \Drafts
> }
> mailbox Junk {
>special_use = \Junk
> }
> mailbox Sent {
>special_use = \Sent
> }
> mailbox "Sent Messages" {
>special_use = \Sent
> }
> mailbox Trash {
>special_use = \Trash
> }
> prefix =
> }
> passdb {
> args = /etc/dovecot/dovecot-ldap.conf.ext
> driver = ldap
> }
> protocols = " imap lmtp"
> service aggregator {
> fifo_listener replication-notify-fifo {
>mode = 0666
>user = vmail
> }
> unix_listener replication-notify {
>mode = 0666
>user = vmail
> }
> }
> service doveadm {
> inet_listener {
>port = 12345
> }
> user = vmail
> }
> service replicator {
> unix_listener replicator-doveadm {
>mode = 0666
> }
> }
> ssl_cert = ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
> args = /etc/dovecot/dovecot-ldap.conf.ext
> driver = ldap
> override_fields = uid=vmail gid=vmail home=/data/mail/vmail/%d/%n
> }
> root@basement-imap01:/var/log#
>
> with the dovecot-ldap.conf.ext as follows:
>
> root@basement-imap01:/etc/dovecot# cat dovecot-ldap.conf.ext
> # This
mail_replica in userdb?
I seem to have got replication working from an older instance to a newer one
(insofar as the data are present on the new box) that I’m looking to stand up
to replace a newer box. In this case, nothing is configured to talk to the new
machine yet, so the replication is unidirectional (old -> new).
However, I’m seeing an error like like this repeatedly in syslog:
2022-05-21T23:23:15.295252+00:00 basement-imap01 dovecot:
doveadm(h...@coyhile.com): Error: sync: User has no mail_replica in userdb
What setting did I miss?
kdoveconf -n
# 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7.2 ()
# OS: Linux 5.4.0-110-generic x86_64 Ubuntu 20.04.4 LTS
# Hostname: basement-imap01.coyhile.com
doveadm_password = # hidden, use -P to show it
mail_location = maildir:~/Maildir
mail_privileged_group = mail
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
protocols = " imap lmtp"
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
user = vmail
}
unix_listener replication-notify {
mode = 0666
user = vmail
}
}
service doveadm {
inet_listener {
port = 12345
}
user = vmail
}
service replicator {
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl_cert =
signature.asc
Description: Message signed with OpenPGP
Re: mail_replica in userdb?
> On May 23, 2022, at 3:20 AM, Aki Tuomi wrote:
>
>>
>> On 22/05/2022 02:32 Coy Hile wrote:
>>
>>
>> I seem to have got replication working from an older instance to a newer one
>> (insofar as the data are present on the new box) that I’m looking to stand
>> up to replace a newer box. In this case, nothing is configured to talk to
>> the new machine yet, so the replication is unidirectional (old -> new).
>>
>> However, I’m seeing an error like like this repeatedly in syslog:
>>
>> 2022-05-21T23:23:15.295252+00:00 basement-imap01 dovecot:
>> doveadm(h...@coyhile.com): Error: sync: User has no mail_replica in userdb
>>
>> What setting did I miss?
>>
>
> plugin {
> mail_replica =
> }
>
> https://doc.dovecot.org/settings/plugin/replication-plugin/
>
> Aki
The way I read that, it configures a location to where a machine will replicate
data. At present, I’m trying to turn up the new box slowly (haven’t yet
configured LDA or appropriate TLS settings, for example), so my intent was that
this host not replicate any of its data anywhere, yet. Eventually, I’ll have
data moving hostA <—> hostB bidirectionally.
Is the phased approach I was trying to take not something supported?
--
Coy Hile
coy.h...@coyhile.com
signature.asc
Description: Message signed with OpenPGP
Re: mail_replica in userdb?
> On May 23, 2022, at 11:00 AM, Paul Kudla (SCOM.CA Internet Services Inc.) > wrote: > > > ok there is more to replica setup > > user-db error seems to indicate the sql iterate it not setup or not returning > stuff properly > > can you specify what you are using for a sql setup (if any) > > can help from there? > Ah, on the new side, I am using LDAP, rather than SQL, setup thusly in dovecot-ldap.conf.ext (once I removed all the comments and blank lines): '/^$/d' hosts = ldap.coyhile.com dn = uid=dovecotquery,ou=people,dc=coyhile,dc=com dnpass = [redacted] base = ou=People,dc=coyhile,dc=com scope = subtree user_filter = (&(objectClass=posixAccount)(uid=%n)) pass_attrs = uid=%n,userPassword=password pass_filter = (&(objectClass=posixAccount)(uid=%n)) iterate_attrs = maildrop=user iterate_filter = (objectClass=posixAccount) -- Coy Hile coy.h...@coyhile.com signature.asc Description: Message signed with OpenPGP
Re: mail_replica in userdb?
> On May 23, 2022, at 11:42 AM, Paul Kudla (SCOM.CA Internet Services Inc.) > wrote: > > > So are you ok now I don’t know that. I don’t know if what I’m trying to do is actually supported as yet. I’m going to setup the downstream replica for this box after I finish work today and try to cut-over to this new setup completely. If the error message goes away once I’ve configured bidirectional replication, I’d say I’m good, but we’ll find that out empirically. Appreciate your help, Paul and Aki. The replication seems to work, but I still see that particular error. Need to test that one more thing is working on the new box. -- Coy Hile coy.h...@coyhile.com signature.asc Description: Message signed with OpenPGP
Question about replication
Good morning, I’m looking at dovecot replication for the first time for a particular use case: The VM running dovecot is currently in a public cloud, and I want to move it on-premises, so I want to replicate all mail data from the older instance to the new (then cutover internal DNS) (and in the process update to newer features on the internal instance such as mdbox vs Maildir). However, the new instance I’m building is internal, would have access to the internet only via NAT, so the replication would have to be one way. Is there anything in particular that I need to keep in mind when I’m trying to setup this sort of setup? The documentation that I’ve read seems to imply (or assume) synchronous, bidirectional replication. Thanks, and apologies if this is an obvious question; this is my first foray into doing something not utterly trivial with Dovecot.
Re: Question about replication
> On Aug 2, 2019, at 7:57 AM, Sami Ketola via dovecot > wrote: > > > >> On 2 Aug 2019, at 14.52, Coy Hile via dovecot wrote: >> >> Good morning, >> >> >> I’m looking at dovecot replication for the first time for a particular use >> case: The VM running dovecot is currently in a public cloud, and I want to >> move it on-premises, so I want to replicate all mail data from the older >> instance to the new (then cutover internal DNS) (and in the process update >> to newer features on the internal instance such as mdbox vs Maildir). >> >> However, the new instance I’m building is internal, would have access to the >> internet only via NAT, so the replication would have to be one way. Is there >> anything in particular that I need to keep in mind when I’m trying to setup >> this sort of setup? The documentation that I’ve read seems to imply (or >> assume) synchronous, bidirectional replication. >> >> Thanks, and apologies if this is an obvious question; this is my first foray >> into doing something not utterly trivial with Dovecot. > > Sounds like you do not want to do replication but migration. > > In this case migration can be easily done by using doveadm dsync to pull the > mails from old server to new server. > > Something like: > > doveadm backup -u userid -R -f ssh syncuser@remote "sudo /usr/bin/doveadm > dsync-server -u userid” > That may be the easier solution, you’re right. I was hoping to run both in parallel for a bit, but cutting over probably makes the most sense. Thanks, -Coy
What does `iterate_query` for SQL want as output?
Hi all, In an earlier thread, https://dovecot.org/pipermail/dovecot/2019-August/116694.html I got a lot of useful help about migration. On my older host, everything was static; on the newer host, I’m storing user information in Postgres. usernames are of the form , say ‘h...@coyhile.com’ as basically a Kerberos principal, and authentication and individual lookups work. My `users` table looks thus: mail=> \d users Table "public.users" Column | Type | Modifiers --+--+--- username | text | not null domain | text | not null password | text | not null mail=> and contains, as an example: username | domain| password --+-+--- h...@coyhile.com | coyhile.com | [REDACTED] (1 row) Naively, I’d expect something this to work for the iteration query: iterate_query = SELECT username, domain FROM users But, when I do that, I end up doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com /opt/local/bin/doveadm dsync-server -A doveadm(h...@coyhile.com@coyhile.com): Info: User no longer exists, skipping [root@81716ec5-bca4-6d53-ed81-bd1a55d46b4f /tmp]# Note the extra “@coyhile.com” in there. Thanks, — Coy Hile coy.h...@coyhile.com
Re: What does `iterate_query` for SQL want as output?
> On Aug 9, 2019, at 3:45 PM, Aki Tuomi wrote: > > >> On 09/08/2019 22:16 Coy Hile via dovecot wrote: >> >> >> Hi all, >> >> In an earlier thread, >> https://dovecot.org/pipermail/dovecot/2019-August/116694.html I got a lot of >> useful help about migration. On my older host, everything was static; on the >> newer host, I’m storing user information in Postgres. usernames are of the >> form , say ‘h...@coyhile.com’ as basically a Kerberos >> principal, and authentication and individual lookups work. >> >> My `users` table looks thus: >> >> mail=> \d users >>Table "public.users" >> Column | Type | Modifiers >> --+--+--- >> username | text | not null >> domain | text | not null >> password | text | not null >> >> mail=> >> >> and contains, as an example: >> >> username | domain| >> password >> --+-+--- >> h...@coyhile.com | coyhile.com | [REDACTED] >> (1 row) >> >> Naively, I’d expect something this to work for the iteration query: >> >> iterate_query = SELECT username, domain FROM users >> >> >> But, when I do that, I end up >> >> doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com >> /opt/local/bin/doveadm dsync-server -A >> doveadm(h...@coyhile.com@coyhile.com): Info: User no longer exists, skipping >> [root@81716ec5-bca4-6d53-ed81-bd1a55d46b4f /tmp]# >> >> Note the extra “@coyhile.com” in there. >> >> Thanks, >> >> — >> Coy Hile >> coy.h...@coyhile.com > > If your username field already contains domain, you do not need to return > domain field separately. It is only needed if your username field only > contains local part. That’s what I thought, and a simpler query returns the data I expect: mail=> select username from users; username -- h...@coyhile.com (1 row) mail=> Or SELECT username AS user FROM users; (if the iterate query is the column to be named `user`?) When I configure the iterate_query to be SELET username AS user FROM users; I get this: doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com /opt/local/bin/doveadm dsync-server -A Error: User listing returned failure doveadm: Error: Failed to iterate through some users dsync-local(h...@coyhile.com): Error: read(remote) failed: EOF (version not received) Which brings up two questions: (1) Is there a way to get more useful debugging information than “failed to iterate through some users”? (FWIW there’s nothing relevant in syslog.) (2) Is there a way to isolate and exercise just that particular bit so that I know I’m giving it what it expects? The SQL documentation https://wiki.dovecot.org/AuthDatabase/SQL indicates that iterate_query = SELECT username AS user FROM users should return what it wants. — Coy Hile coy.h...@coyhile.com
Re: What does `iterate_query` for SQL want as output?
> > Did you check your logs? > that’s just it. There’s nothing in syslog (which is logging at mail.debug). Not just nothing useful, absolutely _nothing_ logged when I run that. All I see is this (printed, I presume to STDERR): doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com /opt/local/bin/doveadm dsync-server -A Error: User listing returned failure doveadm: Error: Failed to iterate through some users dsync-local(h...@coyhile.com): Error: read(remote) failed: EOF (version not received) The last line makes sense given the earlier failure. Should this log somewhere else besides syslog? And is there some way I can isolate and test _just_ the user iteration and enumeration code to prove out that the query is correct? Clearly, from output above, it should be. Thanks, -c
Re: What does `iterate_query` for SQL want as output?
> On Aug 10, 2019, at 10:55 AM, Aki Tuomi wrote: > > >> On 10/08/2019 15:51 Coy Hile via dovecot < dovecot@dovecot.org> wrote: >> >> >>> Did you check your logs? >>> >> that’s just it. There’s nothing in syslog (which is logging at mail.debug). >> Not just nothing useful, absolutely _nothing_ logged when I run that. All I >> see is this (printed, I presume to STDERR): >> >> doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com >> /opt/local/bin/doveadm dsync-server -A >> Error: User listing returned failure >> doveadm: Error: Failed to iterate through some users >> dsync-local( h...@coyhile.com): Error: read(remote) >> failed: EOF (version not received) >> >> The last line makes sense given the earlier failure. >> >> Should this log somewhere else besides syslog? And is there some way I can >> isolate and test _just_ the user iteration and enumeration code to prove out >> that the query is correct? Clearly, from output above, it should be. >> >> Thanks, >> -c > > Can you try doveadm -Dv backup ... > > note the -Dv goes before backup > Ok, that explains why I didn’t get anything useful as debug output. Right now, there’s only one user on either side (namely, me), so all of this seems to look sane to my (admittedly untrained) eye: doveadm(h...@coyhile.com)<63392><>: Debug: auth USER input: h...@coyhile.com password= home=/var/mail/vmail/coyhile.com/hile uid=1000 gid=1000 doveadm(h...@coyhile.com)<63392><>: Debug: Added userdb setting: plugin/password= doveadm(h...@coyhile.com): Debug: Effective uid=1000, gid=1000, home=/var/mail/vmail/coyhile.com/hile doveadm(h...@coyhile.com): Debug: acl: No acl setting - ACLs are disabled doveadm(h...@coyhile.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox doveadm(h...@coyhile.com): Debug: fs: root=/var/mail/vmail/coyhile.com/hile/mdbox, index=, indexpvt=, control=, inbox=, alt= doveadm(h...@coyhile.com): Debug: brain M: Namespace has location mdbox:~/mdbox doveadm(h...@coyhile.com): Debug: Namespace : Using permissions from /var/mail/vmail/coyhile.com/hile/mdbox: mode=0700 gid=default Error: User listing returned failure doveadm: Error: Failed to iterate through some users
dovecot-lmtp and postfix
Hi all,
I’ve got postfix setup to use dovecot-lmtp for (virtual) user delivery, and
things to users or aliases that Dovecot knows about now get delivered
correctly. However, I’ve done something wrong with postfix and/or the LMTP
configuration because upon RCPT TO, instead of forwarding the mail off to the
configured smartest, somehow dovecot-lmtp is ending up in the mix. (Perhaps
this is more a postfix question than a dovecot one, but, I’ll start here.)
Relevant postfix config:
[root@81716ec5-bca4-6d53-ed81-bd1a55d46b4f /opt/local/etc/postfix]# postconf -n
canonical_maps = hash:/opt/local/etc/postfix/canonical
command_directory = /opt/local/sbin
compatibility_level = 2
daemon_directory = /opt/local/libexec/postfix
data_directory = /var/db/postfix
debug_peer_level = 2
debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin ddd
$daemon_directory/$process_name $process_id & sleep 5
default_database_type = hash
home_mailbox = Maildir/
html_directory = no
inet_interfaces = all
inet_protocols = all
mail_owner = postfix
mailbox_size_limit = 10240
mailq_path = /opt/local/bin/mailq
manpage_directory = /opt/local/man
message_size_limit = 5120
meta_directory = /opt/local/libexec/postfix
mynetworks = $myhostname, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12 192.168.0.0/16
newaliases_path = /opt/local/bin/newaliases
owner_request_special = no
queue_directory = /var/spool/postfix
queue_minfree = 7680
readme_directory = /opt/local/share/doc/postfix
recipient_canonical_maps = hash:/opt/local/etc/postfix/recipient_canonical
recipient_delimiter = +
relayhost =
relay.svc.a4b01392-e48d-4cdf-cacf-f850d1800d45.basement.cns.coyhile.com
sample_directory = /opt/local/share/examples/postfix
sender_canonical_maps = hash:/opt/local/etc/postfix/sender_canonical
sendmail_path = /opt/local/sbin/sendmail
setgid_group = maildrop
shlib_directory = /opt/local/lib/postfix
smtpd_banner = $myhostname ESMTP
smtpd_client_restrictions = permit_mynetworks, permit_sasl_authenticated,
reject_unknown_reverse_client_hostname
smtpd_data_restrictions = permit_mynetworks, reject_unauth_pipelining,
permit_sasl_authenticated
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks, permit_sasl_authenticated,
reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated,
reject_unlisted_recipient, reject_non_fqdn_recipient,
reject_unauth_destination, reject_unknown_recipient_domain,
check_recipient_access hash:/opt/local/etc/postfix/filtered_domains
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = private/auth
smtpd_sasl_type = dovecot
smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated,
reject_unlisted_sender, reject_non_fqdn_sender, reject_unknown_sender_domain
smtputf8_enable = no
strict_rfc821_envelopes = yes
unknown_local_recipient_reject_code = 550
virtual_alias_maps = pgsql:/opt/local/etc/postfix/postfix_virtual_aliases.cf
virtual_mailbox_domains =
pgsql:/opt/local/etc/postfix/postfix_virtual_domains.cf
virtual_transport = lmtp:unix:private/dovecot-lmtp
[root@81716ec5-bca4-6d53-ed81-bd1a55d46b4f /opt/local/etc/postfix]#
And the dovecot config:
doveconf -n
# 2.3.6 (7eab80676): /opt/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.6 (92dc263a)
doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn
> dovecot-new.conf
doveconf: Warning: Obsolete setting in
/opt/local/etc/dovecot/conf.d/10-ssl.conf:80: ssl_dh_parameters_length is no
longer needed
doveconf: Warning: Obsolete setting in
/opt/local/etc/dovecot/conf.d/10-ssl.conf:81: ssl_protocols has been replaced
by ssl_min_protocol
doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn
> dovecot-new.conf
doveconf: Warning: Obsolete setting in
/opt/local/etc/dovecot/conf.d/10-ssl.conf:80: ssl_dh_parameters_length is no
longer needed
doveconf: Warning: Obsolete setting in
/opt/local/etc/dovecot/conf.d/10-ssl.conf:81: ssl_protocols has been replaced
by ssl_min_protocol
doveconf: Warning: service auth { client_limit=840 } is lower than required
under max. load (1524)
# OS: SunOS 5.11 i86pc
# Hostname: 81716ec5-bca4-6d53-ed81-bd1a55d46b4f.coyhile.com
auth_mechanisms = plain login
auth_socket_path = /var/run/dovecot/auth-userdb
imap_idle_notify_interval = 29 mins
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
mail_attachment_dir = /var/mail/attachments
mail_fsync = never
mail_location = mdbox:~/mdbox
mail_plugins = " acl"
mail_privileged_group = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
Re: dovecot-lmtp and postfix
On 2019-08-11 18:40, Alexander Dalloz via dovecot wrote: Am 11.08.2019 um 18:06 schrieb Coy Hile via dovecot: Your Postfix configuration makes Postfix think gmail.com to be a virtual(8) target domain. Thus it uses virtual_transport. So validate your virtual_alias_maps = pgsql:/opt/local/etc/postfix/postfix_virtual_aliases.cf virtual_mailbox_domains = pgsql:/opt/local/etc/postfix/postfix_virtual_domains.cf definitions. That was exactly it! I was missing a WHERE domain='%s' in that query, so it returned something (which is effectively "I am handled here" apparently) regardless of the domain. Thank you, -- Coy Hile coy.h...@coyhile.com
Dovecot and Apple's Mail.app not playing nicely?
Hi all, Is there anything cute one has to take into account when using Dovecot with users of Apple’s Mail.app? Behavior I’m seeing is that if I delete or move messages via Webmail (Roundcube, Horde, or even ActiveSync via Mail.app on my phone), they do get moved or deleted. However, if I take the same actions in the desktop mail client, when logging in to the Webmail (or phone) app, I see the messages still seeming to be in the Inbox. Is this known behavior? A peculiarity in Apple Mail? -- Coy Hile coy.h...@coyhile.com
Re: Dovecot and Apple's Mail.app not playing nicely?
On 2019-09-03 09:22, Sami Ketola via dovecot wrote: On 3 Sep 2019, at 15.30, Coy Hile via dovecot wrote: Hi all, Is there anything cute one has to take into account when using Dovecot with users of Apple’s Mail.app? Behavior I’m seeing is that if I delete or move messages via Webmail (Roundcube, Horde, or even ActiveSync via Mail.app on my phone), they do get moved or deleted. However, if I take the same actions in the desktop mail client, when logging in to the Webmail (or phone) app, I see the messages still seeming to be in the Inbox. Is this known behavior? A peculiarity in Apple Mail? I am using Apple Mail.App in Macbook, iPhone and iPad. And in fact quite many of us internally are doing the same and I can't see that behaviour. Mail.App correctly obeys \Deleted flag and does not show the mails in folders. Sami That's exactly the converse of what I'm seeing. Mail.app sets the \Deleted flag, or flags a message as Junk and moves it to the Junk folder. But when I login via, say, Roundcube, it still shows in the inbox, though greyed out with a little (/) icon (which I assume is the deleted flag.) If I move or delete the message via the webmail client, it actually gets moved to Junk or Trash. (Or wherever I moved it.) FWIW, I think this applies only to deleted messages (where Mail.app may just set a flag rather than actually moving the messages to Trash) and to Mail.app's own Junk processing. (Things flagged as Spam and moved to Junk via Sieve do end up in the Junk folder.) -- Coy Hile coy.h...@coyhile.com
