Understanding doveadm rebuild

[Patrick Ben Koetter]

Greetings,

I'm trying to understand if the doveadm rebuild command is suited to fix a
problem we have on a customers platform.

The customer uses sdbox in combination with SIS and for $REASONS dovecot
cannot find the attachements anymore. They are there, but the references to
them in the messages seem to be wrong.

Is that what the rebuild subcommand has been made for? Will it e.g. scan the
messages for attachment references and try to find them in the SIS storage
location and then fix the references in messages?

TIA,

p@rick


-- 
[*] sys4 AG

https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein

Help with migration to Dovecot from raw files

[Stefano Antonelli]

Hello list,

I recently migrated to Dovecot from a very old version of IceWarp 
running on failing hardware running Windows XP.  I had a lot of issues 
downloading mail through imap.  What I have instead are the email files 
directly off the harddrive from IceWarp's mail store.


I rearranged the mail files to look like a Dovecot maildir and tried to 
use these emails directly.  Dovecot did not complain, but I ran into my 
first problem.  The emails were out of order on the client side.  None 
of my imap clients store email.  They all just fetch headers and 
download mail as it's required.  So a resync is not a problem.  I just 
can't figure out how and I don't know if the filenames are a problem.


The files I'm working with have this format:

2020032512105639B0-0100D2D45079615C.imap

The first part is the date and all the the filenames appear to be 
unique.  Is it okay to leave them this way or should I rename them in 
the same fashion as Dovecot?  If so, is there a tool I can use or do I 
need to write a script?  If I have to write a script, could someone help 
me decipher this filename:


1585197800.M440284P10160.hostname,S=2812,W=2875:2,S

I understand the first bit is a unix timestamp.  I couldn't find out 
what M440284P10160 means or how I might create that number myself to use 
in a renaming script.  Additionally, I'd like to customise the hostname 
without changing the machine's hostname and I understand that the S and 
W can also be omitted.  I'm not using quotas.  I couldn't find anywhere 
in the documentation where it talked about any configuration options for 
the filenames.  Are there any or do I have to patch the source?


I was also planning on directly backing up the email using rsync, but 
the changing filenames with flags is obviously an issue.  I understand 
that dovecot can store flags in a file instead, if the mailbox is shared 
or public.  Is there any way to have this behaviour with a private 
mailbox?  Ideally the filenames are not touched after creation in which 
case rsync will work nicely to my NAS.


The last issue is that since Dovecot went live, I've started receiving 
mail.  For the time being, I have left my "yet to be migrated" mail 
outside of maildir.  So Dovecot doesn't know about it.  I will need to 
merge this newly received mail with my "yet to migrated mail" once I 
sort out exactly how to migrate that mail.  And I would really like all 
the mail to sort properly.


From what I can tell doveadm is the tool for the job, but I just can't 
make sense of the documentation.


Thanks,
Stef

Dovecot version:  2.3.4.1 (f79e8e7e4)

Re: Help with migration to Dovecot from raw files

[Aki Tuomi]

You could try something simple like just

for n in *; do cat $n | doveadm save -u targetuser; done

or if the IceWarp has IMAP interface, you could try

doveadm -o imapc_host=hostname -o imapc_user=username -o 
imapc_password=password sync -1R targetuser imapc:

Aki

> On 26/03/2020 12:11 Stefano Antonelli  wrote:
> 
>  
> Hello list,
> 
> I recently migrated to Dovecot from a very old version of IceWarp 
> running on failing hardware running Windows XP.  I had a lot of issues 
> downloading mail through imap.  What I have instead are the email files 
> directly off the harddrive from IceWarp's mail store.
> 
> I rearranged the mail files to look like a Dovecot maildir and tried to 
> use these emails directly.  Dovecot did not complain, but I ran into my 
> first problem.  The emails were out of order on the client side.  None 
> of my imap clients store email.  They all just fetch headers and 
> download mail as it's required.  So a resync is not a problem.  I just 
> can't figure out how and I don't know if the filenames are a problem.
> 
> The files I'm working with have this format:
> 
> 2020032512105639B0-0100D2D45079615C.imap
> 
> The first part is the date and all the the filenames appear to be 
> unique.  Is it okay to leave them this way or should I rename them in 
> the same fashion as Dovecot?  If so, is there a tool I can use or do I 
> need to write a script?  If I have to write a script, could someone help 
> me decipher this filename:
> 
> 1585197800.M440284P10160.hostname,S=2812,W=2875:2,S
> 
> I understand the first bit is a unix timestamp.  I couldn't find out 
> what M440284P10160 means or how I might create that number myself to use 
> in a renaming script.  Additionally, I'd like to customise the hostname 
> without changing the machine's hostname and I understand that the S and 
> W can also be omitted.  I'm not using quotas.  I couldn't find anywhere 
> in the documentation where it talked about any configuration options for 
> the filenames.  Are there any or do I have to patch the source?
> 
> I was also planning on directly backing up the email using rsync, but 
> the changing filenames with flags is obviously an issue.  I understand 
> that dovecot can store flags in a file instead, if the mailbox is shared 
> or public.  Is there any way to have this behaviour with a private 
> mailbox?  Ideally the filenames are not touched after creation in which 
> case rsync will work nicely to my NAS.
> 
> The last issue is that since Dovecot went live, I've started receiving 
> mail.  For the time being, I have left my "yet to be migrated" mail 
> outside of maildir.  So Dovecot doesn't know about it.  I will need to 
> merge this newly received mail with my "yet to migrated mail" once I 
> sort out exactly how to migrate that mail.  And I would really like all 
> the mail to sort properly.
> 
>  From what I can tell doveadm is the tool for the job, but I just can't 
> make sense of the documentation.
> 
> Thanks,
> Stef
> 
> Dovecot version:  2.3.4.1 (f79e8e7e4)

Re: Not able to start dovecot

[Olaf Hopp]

On 3/20/20 4:43 PM, Alexander Dalloz wrote:


Use a proper systemd unit file instead of the initd script.

Alexander



Use e.g. the following for  getting started:

# This file is part of Dovecot
#
# DO NOT CUSTOMIZE THIS FILE, INSTEAD
# create the file:
#   `/etc/systemd/system/dovecot.service.d/service.conf'.
# or copy this as
#   `/etc/systemd/system/dovecot.service` and edit then
# and put your changes there

[Unit]
Description=Dovecot IMAP/POP3 email server
Documentation=man:dovecot(1)
Documentation=http://wiki2.dovecot.org/
After=local-fs.target network-online.target dovecot-init.service
Requires=dovecot-init.service

[Service]
Type=simple
ExecStartPre=/usr/libexec/dovecot/prestartscript
ExecStart=/usr/sbin/dovecot -F
PIDFile=/var/run/dovecot/master.pid
ExecReload=/usr/bin/doveadm reload
ExecStop=/usr/bin/doveadm stop
PrivateTmp=true
NonBlocking=yes
# this will make /usr /boot /etc read only for dovecot
ProtectSystem=full
ProtectHome=no
PrivateDevices=true

# You can add environment variables with e.g.:
#Environment='CORE_OUTOFMEM=1'
# If you have trouble with `Too many open files', increase
LimitNOFILE=65535
# If you want to allow the Dovecot services to produce core dumps, use:
#LimitCORE=infinity

[Install]
WantedBy=multi-user.target


--
Karlsruher Institut für Technologie (KIT)
ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik

Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -

Am Fasanengarten 5, Gebäude 50.34, Raum 009
76131 Karlsruhe
Telefon: +49 721 608-43973
Fax: +49 721 608-46699
E-Mail: olaf.h...@kit.edu
www.atis.informatik.kit.edu

www.kit.edu

KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft

Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Not able to start dovecot

[Sami Ketola]

> On 18 Mar 2020, at 19.55, Adam Raszkiewicz  wrote:
> 
> Freshly compiled dovecot 2.3.9.3 on Centos 7 for some reason not able to start
>  
> sudo systemctl status dovecot.service
> ● dovecot.service - LSB: Starts the Dovecot POP3/IMAP service
>Loaded: loaded (/etc/rc.d/init.d/dovecot; bad; vendor preset: disabled)
>Active: failed (Result: exit-code) since Wed 2020-03-18 16:32:17 UTC; 15s 
> ago
>  Docs: man:systemd-sysv-generator(8)
>   Process: 2868 ExecStart=/etc/rc.d/init.d/dovecot start (code=exited, 
> status=127)
>  
> Mar 18 16:32:17 systemd[1]: Starting LSB: Starts the Dovecot POP3/IMAP 
> service...
> Mar 18 16:32:17 dovecot[2868]: /etc/rc.d/init.d/dovecot: line 34: 
> log_info_msg: command not found
> Mar 18 16:32:17 dovecot[2868]: /etc/rc.d/init.d/dovecot: line 35: 
> start_daemon: command not found
> Mar 18 16:32:17 dovecot[2868]: /etc/rc.d/init.d/dovecot: line 36: 
> evaluate_retval: command not found
> Mar 18 16:32:17 systemd[1]: dovecot.service: control process exited, 
> code=exited status=127
> Mar 18 16:32:17 systemd[1]: Failed to start LSB: Starts the Dovecot POP3/IMAP 
> service.
> Mar 18 16:32:17 systemd[1]: Unit dovecot.service entered failed state.
> Mar 18 16:32:17 systemd[1]: dovecot.service failed.
>  
> I have compiled the same code on Centos 6 and had no problem to run it.
>  
> Any thoughts on that?

Can you use the community packages instead?

https://repo.dovecot.org/ 

Sami

doveadm rebuild: Failed to add attachment keywords

[Patrick Ben Koetter]

Greetings,

I'm using the current dovecot release 2.3.10 and I try to rebuild mailboxes
with broken attachment paths.

When I run this command:

doveadm rebuild attachments -u u...@example.com ALL

I runs fine for 6199 of 6223 messages and fails then at message 6200 with this
message:

doveadm(u...@example.com): Error: Mailbox INBOX: UID=6200: 
read(attachments-connector(/MailStore/mail/mailboxes/CFEE0E67-6269-4AD3-8DED-6AB71A8E4BD7/dbox/mailboxes/INBOX/dbox-Mails/u.6200))
 failed: 
read(/MailStore/mail/attachment/b9/52/b952b91441e53d3bb57453af2df66bb003e551f4-e94b170ff8bb685e943f162bf795-5d0b3129034c8957284d7dfcf92d-6200[base64:19
 b/l]) failed: 
open(/MailStore/mail/attachment/b9/52/b952b91441e53d3bb57453af2df66bb003e551f4-e94b170ff8bb685e943f162bf795-5d0b3129034c8957284d7dfcf92d-6200)
 failed: No such file or directory (read reason=)
doveadm(u...@example.com): Error: Mailbox INBOX: UID=6200: 
read(attachments-connector(/MailStore/mail/mailboxes/CFEE0E67-6269-4AD3-8DED-6AB71A8E4BD7/dbox/mailboxes/INBOX/dbox-Mails/u.6200))
 failed: 
read(/MailStore/mail/attachment/b9/52/b952b91441e53d3bb57453af2df66bb003e551f4-e94b170ff8bb685e943f162bf795-5d0b3129034c8957284d7dfcf92d-6200[base64:19
 b/l]) failed: 
open(/MailStore/mail/attachment/b9/52/b952b91441e53d3bb57453af2df66bb003e551f4-e94b170ff8bb685e943f162bf795-5d0b3129034c8957284d7dfcf92d-6200)
 failed: No such file or directory (read reason=)
doveadm(u...@example.com): Error: Mailbox INBOX: UID=6200: Failed to add 
attachment keywords: mail_get_parts() failed: Mailbox INBOX: UID=6200: 
read(attachments-connector(/MailStore/mail/mailboxes/CFEE0E67-6269-4AD3-8DED-6AB71A8E4BD7/dbox/mailboxes/INBOX/dbox-Mails/u.6200))
 failed: 
read(/MailStore/mail/attachment/b9/52/b952b91441e53d3bb57453af2df66bb003e551f4-e94b170ff8bb685e943f162bf795-5d0b3129034c8957284d7dfcf92d-6200[base64:19
 b/l]) failed: 
open(/MailStore/mail/attachment/b9/52/b952b91441e53d3bb57453af2df66bb003e551f4-e94b170ff8bb685e943f162bf795-5d0b3129034c8957284d7dfcf92d-6200)
 failed: No such file or directory (read reason=)
6200 error

Assuming the attachments are still there, what would I need to do to 'fix'
that? And if I can't fix it, can 'rebuild' skip the error and go on rebuilding
the rest?

TIA,

p@rick

-- 
[*] sys4 AG

https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein

Re: Help with migration to Dovecot from raw files

[Stefano Antonelli]

On 2020-03-26 03:37, Aki Tuomi wrote:

You could try something simple like just

for n in *; do cat $n | doveadm save -u targetuser; done


Thank you.  This "works" in that it renames the files for me.  However,

1. the filename timestamps are all similar (ie. time of import)
2. the emails showed up in new rather than cur
3. the emails are unseen.

The seen state can probably be fixed after the fact using doveadm flags, 
but there doesn't appear to be a way to do it with the doveadm save 
command.


I'd also like the filenames to reflect their original timestamps, not 
the time of import.  Is that possible using doveadm?



or if the IceWarp has IMAP interface, you could try

doveadm -o imapc_host=hostname -o imapc_user=username -o
imapc_password=password sync -1R targetuser imapc:


The server is down for good.  All I have are files.

Thanks,
Stef