Question about replication
Good morning, I’m looking at dovecot replication for the first time for a particular use case: The VM running dovecot is currently in a public cloud, and I want to move it on-premises, so I want to replicate all mail data from the older instance to the new (then cutover internal DNS) (and in the process update to newer features on the internal instance such as mdbox vs Maildir). However, the new instance I’m building is internal, would have access to the internet only via NAT, so the replication would have to be one way. Is there anything in particular that I need to keep in mind when I’m trying to setup this sort of setup? The documentation that I’ve read seems to imply (or assume) synchronous, bidirectional replication. Thanks, and apologies if this is an obvious question; this is my first foray into doing something not utterly trivial with Dovecot.
Re: Question about replication
> On Aug 2, 2019, at 7:57 AM, Sami Ketola via dovecot > wrote: > > > >> On 2 Aug 2019, at 14.52, Coy Hile via dovecot wrote: >> >> Good morning, >> >> >> I’m looking at dovecot replication for the first time for a particular use >> case: The VM running dovecot is currently in a public cloud, and I want to >> move it on-premises, so I want to replicate all mail data from the older >> instance to the new (then cutover internal DNS) (and in the process update >> to newer features on the internal instance such as mdbox vs Maildir). >> >> However, the new instance I’m building is internal, would have access to the >> internet only via NAT, so the replication would have to be one way. Is there >> anything in particular that I need to keep in mind when I’m trying to setup >> this sort of setup? The documentation that I’ve read seems to imply (or >> assume) synchronous, bidirectional replication. >> >> Thanks, and apologies if this is an obvious question; this is my first foray >> into doing something not utterly trivial with Dovecot. > > Sounds like you do not want to do replication but migration. > > In this case migration can be easily done by using doveadm dsync to pull the > mails from old server to new server. > > Something like: > > doveadm backup -u userid -R -f ssh syncuser@remote "sudo /usr/bin/doveadm > dsync-server -u userid” > That may be the easier solution, you’re right. I was hoping to run both in parallel for a bit, but cutting over probably makes the most sense. Thanks, -Coy
What does `iterate_query` for SQL want as output?
Hi all, In an earlier thread, https://dovecot.org/pipermail/dovecot/2019-August/116694.html I got a lot of useful help about migration. On my older host, everything was static; on the newer host, I’m storing user information in Postgres. usernames are of the form , say ‘h...@coyhile.com’ as basically a Kerberos principal, and authentication and individual lookups work. My `users` table looks thus: mail=> \d users Table "public.users" Column | Type | Modifiers --+--+--- username | text | not null domain | text | not null password | text | not null mail=> and contains, as an example: username | domain| password --+-+--- h...@coyhile.com | coyhile.com | [REDACTED] (1 row) Naively, I’d expect something this to work for the iteration query: iterate_query = SELECT username, domain FROM users But, when I do that, I end up doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com /opt/local/bin/doveadm dsync-server -A doveadm(h...@coyhile.com@coyhile.com): Info: User no longer exists, skipping [root@81716ec5-bca4-6d53-ed81-bd1a55d46b4f /tmp]# Note the extra “@coyhile.com” in there. Thanks, — Coy Hile coy.h...@coyhile.com
Re: What does `iterate_query` for SQL want as output?
> On Aug 9, 2019, at 3:45 PM, Aki Tuomi wrote: > > >> On 09/08/2019 22:16 Coy Hile via dovecot wrote: >> >> >> Hi all, >> >> In an earlier thread, >> https://dovecot.org/pipermail/dovecot/2019-August/116694.html I got a lot of >> useful help about migration. On my older host, everything was static; on the >> newer host, I’m storing user information in Postgres. usernames are of the >> form , say ‘h...@coyhile.com’ as basically a Kerberos >> principal, and authentication and individual lookups work. >> >> My `users` table looks thus: >> >> mail=> \d users >>Table "public.users" >> Column | Type | Modifiers >> --+--+--- >> username | text | not null >> domain | text | not null >> password | text | not null >> >> mail=> >> >> and contains, as an example: >> >> username | domain| >> password >> --+-+--- >> h...@coyhile.com | coyhile.com | [REDACTED] >> (1 row) >> >> Naively, I’d expect something this to work for the iteration query: >> >> iterate_query = SELECT username, domain FROM users >> >> >> But, when I do that, I end up >> >> doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com >> /opt/local/bin/doveadm dsync-server -A >> doveadm(h...@coyhile.com@coyhile.com): Info: User no longer exists, skipping >> [root@81716ec5-bca4-6d53-ed81-bd1a55d46b4f /tmp]# >> >> Note the extra “@coyhile.com” in there. >> >> Thanks, >> >> — >> Coy Hile >> coy.h...@coyhile.com > > If your username field already contains domain, you do not need to return > domain field separately. It is only needed if your username field only > contains local part. That’s what I thought, and a simpler query returns the data I expect: mail=> select username from users; username -- h...@coyhile.com (1 row) mail=> Or SELECT username AS user FROM users; (if the iterate query is the column to be named `user`?) When I configure the iterate_query to be SELET username AS user FROM users; I get this: doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com /opt/local/bin/doveadm dsync-server -A Error: User listing returned failure doveadm: Error: Failed to iterate through some users dsync-local(h...@coyhile.com): Error: read(remote) failed: EOF (version not received) Which brings up two questions: (1) Is there a way to get more useful debugging information than “failed to iterate through some users”? (FWIW there’s nothing relevant in syslog.) (2) Is there a way to isolate and exercise just that particular bit so that I know I’m giving it what it expects? The SQL documentation https://wiki.dovecot.org/AuthDatabase/SQL indicates that iterate_query = SELECT username AS user FROM users should return what it wants. — Coy Hile coy.h...@coyhile.com
Re: What does `iterate_query` for SQL want as output?
> > Did you check your logs? > that’s just it. There’s nothing in syslog (which is logging at mail.debug). Not just nothing useful, absolutely _nothing_ logged when I run that. All I see is this (printed, I presume to STDERR): doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com /opt/local/bin/doveadm dsync-server -A Error: User listing returned failure doveadm: Error: Failed to iterate through some users dsync-local(h...@coyhile.com): Error: read(remote) failed: EOF (version not received) The last line makes sense given the earlier failure. Should this log somewhere else besides syslog? And is there some way I can isolate and test _just_ the user iteration and enumeration code to prove out that the query is correct? Clearly, from output above, it should be. Thanks, -c
Re: What does `iterate_query` for SQL want as output?
> On Aug 10, 2019, at 10:55 AM, Aki Tuomi wrote: > > >> On 10/08/2019 15:51 Coy Hile via dovecot < dovecot@dovecot.org> wrote: >> >> >>> Did you check your logs? >>> >> that’s just it. There’s nothing in syslog (which is logging at mail.debug). >> Not just nothing useful, absolutely _nothing_ logged when I run that. All I >> see is this (printed, I presume to STDERR): >> >> doveadm backup -D -A -R -f ssh -i id_rsa.dsync imap01.coyhile.com >> /opt/local/bin/doveadm dsync-server -A >> Error: User listing returned failure >> doveadm: Error: Failed to iterate through some users >> dsync-local( h...@coyhile.com): Error: read(remote) >> failed: EOF (version not received) >> >> The last line makes sense given the earlier failure. >> >> Should this log somewhere else besides syslog? And is there some way I can >> isolate and test _just_ the user iteration and enumeration code to prove out >> that the query is correct? Clearly, from output above, it should be. >> >> Thanks, >> -c > > Can you try doveadm -Dv backup ... > > note the -Dv goes before backup > Ok, that explains why I didn’t get anything useful as debug output. Right now, there’s only one user on either side (namely, me), so all of this seems to look sane to my (admittedly untrained) eye: doveadm(h...@coyhile.com)<63392><>: Debug: auth USER input: h...@coyhile.com password= home=/var/mail/vmail/coyhile.com/hile uid=1000 gid=1000 doveadm(h...@coyhile.com)<63392><>: Debug: Added userdb setting: plugin/password= doveadm(h...@coyhile.com): Debug: Effective uid=1000, gid=1000, home=/var/mail/vmail/coyhile.com/hile doveadm(h...@coyhile.com): Debug: acl: No acl setting - ACLs are disabled doveadm(h...@coyhile.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox doveadm(h...@coyhile.com): Debug: fs: root=/var/mail/vmail/coyhile.com/hile/mdbox, index=, indexpvt=, control=, inbox=, alt= doveadm(h...@coyhile.com): Debug: brain M: Namespace has location mdbox:~/mdbox doveadm(h...@coyhile.com): Debug: Namespace : Using permissions from /var/mail/vmail/coyhile.com/hile/mdbox: mode=0700 gid=default Error: User listing returned failure doveadm: Error: Failed to iterate through some users
dovecot-lmtp and postfix
Hi all,
I’ve got postfix setup to use dovecot-lmtp for (virtual) user delivery, and
things to users or aliases that Dovecot knows about now get delivered
correctly. However, I’ve done something wrong with postfix and/or the LMTP
configuration because upon RCPT TO, instead of forwarding the mail off to the
configured smartest, somehow dovecot-lmtp is ending up in the mix. (Perhaps
this is more a postfix question than a dovecot one, but, I’ll start here.)
Relevant postfix config:
[root@81716ec5-bca4-6d53-ed81-bd1a55d46b4f /opt/local/etc/postfix]# postconf -n
canonical_maps = hash:/opt/local/etc/postfix/canonical
command_directory = /opt/local/sbin
compatibility_level = 2
daemon_directory = /opt/local/libexec/postfix
data_directory = /var/db/postfix
debug_peer_level = 2
debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin ddd
$daemon_directory/$process_name $process_id & sleep 5
default_database_type = hash
home_mailbox = Maildir/
html_directory = no
inet_interfaces = all
inet_protocols = all
mail_owner = postfix
mailbox_size_limit = 10240
mailq_path = /opt/local/bin/mailq
manpage_directory = /opt/local/man
message_size_limit = 5120
meta_directory = /opt/local/libexec/postfix
mynetworks = $myhostname, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12 192.168.0.0/16
newaliases_path = /opt/local/bin/newaliases
owner_request_special = no
queue_directory = /var/spool/postfix
queue_minfree = 7680
readme_directory = /opt/local/share/doc/postfix
recipient_canonical_maps = hash:/opt/local/etc/postfix/recipient_canonical
recipient_delimiter = +
relayhost =
relay.svc.a4b01392-e48d-4cdf-cacf-f850d1800d45.basement.cns.coyhile.com
sample_directory = /opt/local/share/examples/postfix
sender_canonical_maps = hash:/opt/local/etc/postfix/sender_canonical
sendmail_path = /opt/local/sbin/sendmail
setgid_group = maildrop
shlib_directory = /opt/local/lib/postfix
smtpd_banner = $myhostname ESMTP
smtpd_client_restrictions = permit_mynetworks, permit_sasl_authenticated,
reject_unknown_reverse_client_hostname
smtpd_data_restrictions = permit_mynetworks, reject_unauth_pipelining,
permit_sasl_authenticated
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks, permit_sasl_authenticated,
reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated,
reject_unlisted_recipient, reject_non_fqdn_recipient,
reject_unauth_destination, reject_unknown_recipient_domain,
check_recipient_access hash:/opt/local/etc/postfix/filtered_domains
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = private/auth
smtpd_sasl_type = dovecot
smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated,
reject_unlisted_sender, reject_non_fqdn_sender, reject_unknown_sender_domain
smtputf8_enable = no
strict_rfc821_envelopes = yes
unknown_local_recipient_reject_code = 550
virtual_alias_maps = pgsql:/opt/local/etc/postfix/postfix_virtual_aliases.cf
virtual_mailbox_domains =
pgsql:/opt/local/etc/postfix/postfix_virtual_domains.cf
virtual_transport = lmtp:unix:private/dovecot-lmtp
[root@81716ec5-bca4-6d53-ed81-bd1a55d46b4f /opt/local/etc/postfix]#
And the dovecot config:
doveconf -n
# 2.3.6 (7eab80676): /opt/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.6 (92dc263a)
doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn
> dovecot-new.conf
doveconf: Warning: Obsolete setting in
/opt/local/etc/dovecot/conf.d/10-ssl.conf:80: ssl_dh_parameters_length is no
longer needed
doveconf: Warning: Obsolete setting in
/opt/local/etc/dovecot/conf.d/10-ssl.conf:81: ssl_protocols has been replaced
by ssl_min_protocol
doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -Pn
> dovecot-new.conf
doveconf: Warning: Obsolete setting in
/opt/local/etc/dovecot/conf.d/10-ssl.conf:80: ssl_dh_parameters_length is no
longer needed
doveconf: Warning: Obsolete setting in
/opt/local/etc/dovecot/conf.d/10-ssl.conf:81: ssl_protocols has been replaced
by ssl_min_protocol
doveconf: Warning: service auth { client_limit=840 } is lower than required
under max. load (1524)
# OS: SunOS 5.11 i86pc
# Hostname: 81716ec5-bca4-6d53-ed81-bd1a55d46b4f.coyhile.com
auth_mechanisms = plain login
auth_socket_path = /var/run/dovecot/auth-userdb
imap_idle_notify_interval = 29 mins
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
mail_attachment_dir = /var/mail/attachments
mail_fsync = never
mail_location = mdbox:~/mdbox
mail_plugins = " acl"
mail_privileged_group = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
Re: dovecot-lmtp and postfix
On 2019-08-11 18:40, Alexander Dalloz via dovecot wrote: Am 11.08.2019 um 18:06 schrieb Coy Hile via dovecot: Your Postfix configuration makes Postfix think gmail.com to be a virtual(8) target domain. Thus it uses virtual_transport. So validate your virtual_alias_maps = pgsql:/opt/local/etc/postfix/postfix_virtual_aliases.cf virtual_mailbox_domains = pgsql:/opt/local/etc/postfix/postfix_virtual_domains.cf definitions. That was exactly it! I was missing a WHERE domain='%s' in that query, so it returned something (which is effectively "I am handled here" apparently) regardless of the domain. Thank you, -- Coy Hile coy.h...@coyhile.com
Dovecot and Apple's Mail.app not playing nicely?
Hi all, Is there anything cute one has to take into account when using Dovecot with users of Apple’s Mail.app? Behavior I’m seeing is that if I delete or move messages via Webmail (Roundcube, Horde, or even ActiveSync via Mail.app on my phone), they do get moved or deleted. However, if I take the same actions in the desktop mail client, when logging in to the Webmail (or phone) app, I see the messages still seeming to be in the Inbox. Is this known behavior? A peculiarity in Apple Mail? -- Coy Hile coy.h...@coyhile.com
Re: Dovecot and Apple's Mail.app not playing nicely?
On 2019-09-03 09:22, Sami Ketola via dovecot wrote: On 3 Sep 2019, at 15.30, Coy Hile via dovecot wrote: Hi all, Is there anything cute one has to take into account when using Dovecot with users of Apple’s Mail.app? Behavior I’m seeing is that if I delete or move messages via Webmail (Roundcube, Horde, or even ActiveSync via Mail.app on my phone), they do get moved or deleted. However, if I take the same actions in the desktop mail client, when logging in to the Webmail (or phone) app, I see the messages still seeming to be in the Inbox. Is this known behavior? A peculiarity in Apple Mail? I am using Apple Mail.App in Macbook, iPhone and iPad. And in fact quite many of us internally are doing the same and I can't see that behaviour. Mail.App correctly obeys \Deleted flag and does not show the mails in folders. Sami That's exactly the converse of what I'm seeing. Mail.app sets the \Deleted flag, or flags a message as Junk and moves it to the Junk folder. But when I login via, say, Roundcube, it still shows in the inbox, though greyed out with a little (/) icon (which I assume is the deleted flag.) If I move or delete the message via the webmail client, it actually gets moved to Junk or Trash. (Or wherever I moved it.) FWIW, I think this applies only to deleted messages (where Mail.app may just set a flag rather than actually moving the messages to Trash) and to Mail.app's own Junk processing. (Things flagged as Spam and moved to Junk via Sieve do end up in the Junk folder.) -- Coy Hile coy.h...@coyhile.com
