Hello Aki, Thanks for looking into these.
I will as requested attempt the relevant procedures under Dovecot 2.3.6. To make the test fair, I will need to fork the relevant production groupware stack (which is now stable and in operation, with our enterprise (email) data successfully migrated from Microsoft Exchange) to a new staging server; given that the current staging server is now of a materially different configuration to the production server (where the most controlled observations of these bugs were made). Kindly give me some time, as I have an urgent internal openstack deployment project to kick-off, that has been delayed by an overrun of this internal groupware stack deployment project. Best regards, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldo...@parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Aki Tuomi via dovecot <dovecot@dovecot.org> Reply-To: Aki Tuomi <aki.tu...@open-xchange.com> To: arnoldo...@parallaxict.com, Arnold Opio Oree < arnold.o...@parallaxict.com>, Arnold Opio Oree via dovecot < dovecot@dovecot.org> Cc: debian-rele...@lists.debian.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Mon, 8 Jul 2019 08:48:08 +0300 (EEST) Hi! Thank you for reporting these. We will look into them. In the mean time, can you see if any of these are fixed in 2.3.6? Aki > On 07/07/2019 18:12 Arnold Opio Oree via dovecot <dovecot@dovecot.org > > wrote: > > > Dovecot Team, > > I'd like to report a number of bugs, that are to my view all > critical. > > System: Replicated on multiple Debian 10 (Buster) systems > Dovecot Version(s): 2.3.4.1 > > doveadm-sync -1/general > > 1) If DIRNAMEs are not different between command line and > mail_location doveadm sync will fail, saying that the source and > destination directories are the same > > 2) The -n / -N flags do not work, and a sync will fail strangely if > location is specified in the namespace definition > > 3) Adds mbox to path name under mailbox directory (where syncing from > an mbox source) > > 4) Not having the mailboxes at source named the same as those at > destination causes errors and partial sync > > 5) Not having the target mailboxes formatted to receive the sync > (/<mailboxroot>/DIRNAME/) will cause sync errors. > > doveadm-sync > > 1) With large synchronizations UIDs are corrupted where multiple > syncs are executed and the program can no longer synchronize > > dovecot > > 1) Panics and fails to expand ~ to user home: observed cases are > where multiple namespaces are being used > > Please let me know if you need me to elaborate or to provide any > further information that you may need to replicate the bugs, or if I > can help in any other way. > > With regards to the last error that I requested help on i.e. > \Noselect. This has been resolved more-or-less by the workarounds > that I have implemented for the bugs reported above. > > I have seen a number of threads whilst researching the \Noselect > issue where people have been very confused. My finding was that > \Noselect is a function of the IMAP specification server-side > implementation RFC3501 ( > https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the > server was returning directories with \Noselect because the mailboxes > were malformed on account of dovadm-sync errors. In order to fix this > I formed a bash command to transverse the mailbox hierarchy and > create the missing folders critical to the sdbox format, namely > DIRNAME. > > Kind regards, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldo...@parallaxdt.com > > http://www.parallaxdt.com > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard its contents and disclose them to no other persons. > > -----Original Message----- > From: Arnold Opio Oree via dovecot < dovecot@dovecot.org> > Reply-To: arnoldo...@parallaxict.com, Arnold Opio Oree < > arnold.o...@parallaxict.com> > To: dovecot@dovecot.org > Cc: r...@sys4.de, aki.tu...@open-xchange.com > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 04 Jul 2019 14:52:28 +0100 > > > Hi all, > > The guidance provided so far has been really helpful, and has helped > a great deal to bringing down wasted energy on finding and executing > a viable path. I am now at the final due action to complete our > Dovecot application to our use-case, but am stuck on an issue that I > cannot find any easily accessible documentation on. > > Generally this is what has been done: > > 1. Uploaded the enterprise data PST to the target groupware server. > 2. Prepared the server by changing the mailbox format to sdbox and > the the Dovecot mail location to > mail_location=/var/vmail/domain/user/mail/ > 3. Converted the pst (on-server) to a recursive mbox hierarchy using > readpst > 4. Executed doveadm-sync to convert mbox hierarchy data into sdbox > and to copy it into the enterprise archive user's mailboxes > 4.i. The biggest issue I faced at this point was doveadm-sync saying > that the source and destination pointed to the same location, whereas > they clearly did not. > 4.i.a. I resolved this by removing the location= setting from the > target namespace, and allowing it to default to mail_location = > setting, and then using a completely different DIRNAME for the import > doveadm-sync execution (which was the desired final DIRNAME); I then > once the sync had been successful, changed the mail_location DIRNAME > so that it pointed to the imported mail DIRNAME; and hence the > imported email data was in the live mailboxes > 4.i.b. doveadm-import failed several times, and was throwing quite > inexplicable errors, so I moved onto doveadm-sync > 4.i.c. I also had to make sure that the source and destination folder > names matched, otherwise doveadm-syc threw very many errors and only > partially imported the data > 4.i.d. An issue which I decided just to live with is that an mbox > DIRNAME was added to each mailbox as well as the DIRNAME specified so > the path to mail is mbox/dbox-Mails. My thought is that with the data > live on an IMAP server it will be possible to do a dysync through TCP > to correct this problem. > > The final issue that I am facing now, is that when readpst finds > empty folders in the source pst hierarchy, it does not create an mbox > file in the mbox hierarchy folder space. This causes doveadm-sync to > not create the target data required for its mailbox structure i.e. > DIRNAME sub-folder and index file (with our configuration). At this > point either doveadm-sync or the dovecot process makes these empty > folders not selectable. > > The question now is how would I go about making all of these folders > selectable, e.g. with an internal or external command line tool to > change flags / create necessary sdbox mailbox constituent data? > > Many thanks, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldo...@parallaxdt.com > > http://www.parallaxdt.com > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard its contents and disclose them to no other persons. > > -----Original Message----- > From: Arnold Opio Oree via dovecot < dovecot@dovecot.org> > Reply-To: arnoldo...@parallaxict.com, Arnold Opio Oree < > arnold.o...@parallaxict.com> > To: Robert Schetterer < r...@sys4.de>, dovecot@dovecot.org > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 27 Jun 2019 12:05:35 +0100 > > > Also you may run into client limits.... i ve seen this with > > outlook, > > apple mail, thunderbird via imap in the past > > Thanks for this note Robert, it was not really an aspect that I was > considering. > > We are operating our groupware services user access through both > Evolution Groupware and KDE Kontact / KMail on Debian Linux > workstations. Hopefully if there is a client issue it should be local > to only one groupware client. > > I will be sure to study / investigate in this - client - area should > any issues that are not traceable to the server-side arise. > > Many thanks, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldo...@parallaxdt.com > > > http://www.parallaxdt.com > > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard > its contents and disclose them to no other persons. > > > -----Original Message----- > From: Robert Schetterer via dovecot < > dovecot@dovecot.org > > > Reply-To: Robert Schetterer < > r...@sys4.de > > > To: > dovecot@dovecot.org > > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 27 Jun 2019 12:53:49 +0200 > > Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot: > > On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote: > > > Hello to you all, > > > > > > I'd like to ask about my intended application of Dovecot to > > > create > > > a folder-hierarchy for storing our enterprise emails, which are > > > treated as live data rather than archives for compliance or > > > occasional / reactive retrieval. > > > > > > The data is presently not that large (a few gigabytes), but it is > > > expected to grow rapidly. Up to this stage the data has been > > > contained > > > in a Microsoft Exchange mailbox (2013), and then in an offline > > > PST. > > > The move to the offline PST was by necessity, as the large number > > > of > > > folders, and depth of hierarchy to my best understanding caused > > > the > > > exchange server / outlook / evolution mail clients to begin to > > > malfunction. To cope with this the archive was broken up and the > > > bulk stored in the offline PST and the most active components > > > stored in > > > online Exchange mailboxes. > > > > > > I have some understanding of the fs mbox format, and also the > > > mitigations to be made for certain characters / strings. My main > > > concern is whether Dovecot is likely to be able to cope well with > > > a > > > large number of folders / depth of hierarcy. > > > > > > I will really appreciate any help / advice you can give. > > > > > > Best regards, > > > > > > Arnold Opio Oree > > > > > > > Hi! > > > > Dovecot 2.2.34/2.3 supports unlimited folder depth, the only > > limiting > > factor is that the total name may not exceed 4096 bytes. Also > > individual > > folder names may not exceed 255 bytes. > > > > Prior to that the limit is 255 per folder up to 16 levels. > > > > I can't recommend using 'mbox' storage format, please consider > > using > > maildir or sdbox instead. > > > > Aki > > > > > > > > > > --- Aki Tuomi --
