Re: Solr
On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote: In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command : sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance before updating the schema.xml . Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users Additionaly, the url is http://(solr_ server):8983/solr/dovecot/ (error in wiki) After installing Solr, wherever the installation sets up there should a folder similar to: /solr/server/solr/configsets If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know. The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it. Daniel
Re: Solr
The one on the Wiki is mine...which I'm using now. So it certainly does work - but perhaps there's a setting you have differently from me. Performing a "create -c dovecot" creates a Solr instance *named* dovecot - that does *not* initialize it with the necessary schema. You need to specify "-d dovecot", with a dovecot configset already setup, to do that. The other choice is to create the instance as you show, ensure Solr is stopped, delete the "/solr/dovecot/data" folder, and copy the managed-schema file to "/solr/dovecot/conf". Again, the filename saved in the /conf folder needs to be "managed-schema" - no ".xml" suffix. If that doesn't work for you - please share the errors. Daniel On 12/10/2018 11:40 AM, Joan Moreau wrote: Hi Daniel, THere is no need of all this, just the command (on Solr 7.5) "create -c dovecot " is enough The chema.xml provided on the wiki basically does not work on 7.5 Here the latest one I am working on , but nothing works properly (bad search results, errors in ftp_solr, etc..) id positionIncrementGap="0" /> autoGeneratePhraseQueries="true" positionIncrementGap="100"> ignoreCase="true"/> generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> maxGramSize="15" /> protected="protwords.txt"/> ignoreCase="true" synonyms="synonyms.txt"/> ignoreCase="true"/> generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> maxGramSize="15" /> protected="protwords.txt"/> stored="true"/> stored="true"/> stored="true"/> stored="true"/> On 2018-12-10 21:17, Daniel Miller via dovecot wrote: On 12/4/2018 10:40 AM, Joan Moreau via dovecot wrote: In the Wiki, ( https://wiki.dovecot.org/Plugins/FTS/Solr ), it would nice to stipulate to the reader to type the command : sudo -u solr /opt/solr/bin/solr create -c dovecot # to create the dovecot instance before updating the schema.xml . Also, schema.xml is in /opt/solr/server/solr/dovecot/conf for archlinux users Additionaly, the url is http://(solr_ server):8983/solr/dovecot/ (error in wiki) After installing Solr, wherever the installation sets up there should a folder similar to: /solr/server/solr/configsets If you look there, you'll probably see folders like '_default' and 'sample_techproducts_configs'. I haven't played with the 'techproducts' sample. Copy the '_default' folder, with all its contents, to a 'dovecot' folder. In the new dovecot folder, replace the 'managed-schema' file with the file from the Dovecot Wiki https://wiki.dovecot.org/Plugins/FTS/Solr?action=AttachFile&do=view&target=solr-7.x-schema.xml after that, you should be able to run 'solr /opt/solr/bin/solr create -c dovecot' to create the instance. If things still don't work let us know. The schema is one I've tweaked and updated during my own migrations since Solr 3.3. It's possible there's something else in my config that needs documenting - but having experienced Solr search against my mailstore I never want to be without it. Daniel
Re: Solr
On 12/10/2018 10:02 PM, Joan Moreau wrote: Additionally, here the errors I get in logs: Dovecot: Dec 09 09:21:09 imap(j...@grosjo.net)<3349>: Error: fts_solr: received invalid uid '0' Dec 09 09:21:10 imap(j...@grosjo.net)<3349>: Error: fts_solr: received invalid uid '0' or 11 03:36:03 indexer-worker(j...@grosjo.net)<2093>: Error: fts_solr: Indexing failed: 500 Server Error This looks like a permissions issue. Are you using NFS? -- Daniel
Re: Solr
On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote: I shared the errors already so many times (check this mailinling for "solr" in teh title) Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml In order to create the instance, no, it copies the default config in the dovecot instance. I'm not a Solr expert by any means but I believe you are incorrect. As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it. The part that you *must* do after creating from the default template is stop the server, delete the entire "/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data. If you'll try that - explicitly "rm -rf /solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel
Re: Solr
Joan, I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging. That said, I suggest you consider an alternative viewpoint. Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem. That may well be - but I generally approach these matters from the assumption that *I* made the error in configuration and go from there. I'm not an official rep for any product nor claim to be any form of expert in these matters - but I do have a working setup and I'd like to help you if I can. If you're willing to - take a deep breath and let's try starting over. Looking back through your emails there were two items that stood out - your Dovecot config has two settings I don't use: "fts_decoder" and "fts_enforced". I also asked you earlier whether or not NFS is involved here and I didn't see an answer - please clarify. I suggest you try once more: delete Solr completely. Re-install per the directions and use *my* managed-schema. Also comment out the Dovecot directives for "fts_decoder" and "fts_enforced" so you're closer to my setup. Try running again and then post back - I'll do what I can. Based on the fact that Dovecot+Solr 7.5+my schema is working for me leads me to believe we can get it working for you as well. Daniel On 12/15/2018 2:42 PM, Joan Moreau wrote: here my latest schema.xml (remove the "long" type hich seems to be very deprecated in 7.x) id positionIncrementGap="0" /> autoGeneratePhraseQueries="true" positionIncrementGap="100"> ignoreCase="true"/> generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> maxGramSize="15" /> protected="protwords.txt"/> ignoreCase="true" synonyms="synonyms.txt"/> ignoreCase="true"/> generateWordParts="1" generateNumberParts="1" splitOnCaseChange="1" splitOnNumerics="1" catenateWords="1" catenateNumbers="1" catenateAll="1"/> maxGramSize="15" /> protected="protwords.txt"/> stored="true"/> stored="true"/> stored="true"/> stored="true"/> On 2018-12-15 20:54, Joan Moreau wrote: Daniel, I have done that so any times (deleteing the data folders, recreating the instance, restarting etc...) But this is really not the issue The issue is 1 - fts_solr reports errors in the log file (this is a pure dovecot issue) : how to have much more details on what fts_solr sends to Slor server and what does it returns ? 2 - Solr returns properly for a few hours, then starts crashing or responding non-sense after some time Additionally, is there a doc of fts-squat in order to adjust the code to new releases of dovect ? On December 12, 2018 4:44:10 PM Daniel Miller via dovecot wrote: On 12/11/2018 4:46 AM, Joan Moreau via dovecot wrote: I shared the errors already so many times (check this mailinling for "solr" in teh title) Contrary to what you say, with SOlr 7.5 and Dovecot git, I had to remove the "managed-schema" to make solr respond a bit properly. It relies on schema.xml In order to create the instance, no, it copies the default config in the dovecot instance. I'm not a Solr expert by any means but I believe you are incorrect. As of Solr 5.x the managed-schema file is the primary method for configuration. The method I detailed previously for setting up a config helps automate creating new Solr instances - but as I stated you can either setup a Solr template and then create the instance from that or create an instance using the default template and then adjust it. The part that you *must* do after creating from the default template is stop the server, delete the entire "/solr/dovecot/data" folder, then install the correct managed-schema file, then restart the server. The server will not function with mismatched schema/data. If you'll try that - explicitly "rm -rf /solr/dovecot/data", copy the managed-schema file into the conf folder, and restart - things will either work or there's something else that needs correction. -- Daniel
Re: Upgrade to 2.3.1 has failed
As a LetsEncrypt user myself, I have: ssl_cert = So nothing further should be required. You say Dovecot fails to start - have you tried simply executing "dovecot -F"? Daniel On 12/16/2018 6:19 AM, C. Andrews Lavarre wrote: Phil hi. Thank you for explaining what the symbol does... so it is like the BASH *from* symbol. OK.That is new information. So without it dovecot reads the *path/to/file* as if it were a hashed cert, which of course doesn't work. So *with* the symbol dovecot tries to follow the path to read the cert but for some reason cannot read it. Now, that is curious, since I can *cat* the path/to/file and read the cert or key... Now, while the /path/to/file permission is presently *root:root 0777 *(yes, I know 0777 is not good, but I was trying to eliminate any prevention to reading it)**it is actually a soft link to yet another file. Let'sEncrypt has to be renewed every so often so the cert engine (*certbot*) recreates the softlink to the new cert so that we don't need to edit *10-ssl.conf*. So I have entered the actual full path/to/file for the cert and key (not the softlinks) to eliminate that possibility, buty it didn't help. So it's something else. As you say, focus on the problem: Simply put, why can 2.3.1 not read a file while we can list and print out (*ls, cat*) the file? What changed in that regard from 2.2.x to 2.3.1? I'm very grateful for the time folks have spent on this, including my own time. I'm not being rude, just factual. This is what is happening. But "something is wrong with your configuration", while equally factual, is also equally ineffective. OTOH, in my experience factually describing an anomaly can lead to someone wondering why it might be, and if they are more knowledgeable of the inner workings of the system be better able to understand why that might be. For example, I didn't know anything about AppArmor before, now I do, have gone down that rabbit hole, and seem to be able to say, nope, that's not the problem. So now I can move on to checking out something else. Similarly, under BASH the path/to/files are all correct and I can read them from the command line. And 2.2.x didn't have any problem with them. So why might 2.3.1 not be able to read them? So we all need to leave this alone, for now. I'll work along, and when/if I figure it out shall return to report. I'm sure it's something simple: Easy when you know how. :-) Thanks again. Andy On Sun, 2018-12-16 at 07:41 -0500, Phil Turmel wrote: Andy, This is just rude. You have been told multiple times that the less-than symbol is required to read the certificate from the file. Otherwise, the filename is parsed as if it is the certificate itself. Which yields garbage. If dovecot can't read that file, it is *not* dovecot's fault. You are simply not going to succeed until *you* figure out what security differences you have in your new installation. So dovecot can read the files. Every single attempt to connect via openssh depends on dovecot reading your certificate and key files. They are pointless exercises until dovecot actually loads your files. Focus on the real problem if you wish to fix your service. On 12/15/18 5:12 PM, C. Andrews Lavarre wrote: Alexander, Thanks, as described before, if I include the "<" then Dovecot fails to start at all. Thank you again for your time. I have forwarded my latest to Aki to the group. Regards, Phil
ssh_dh?
Don't know if this was corrected in 2.3.4 (haven't upgraded yet but didn't see it in the notes) - but in 2.3.3 I see this in my log: imap-login: Error: Diffie-Hellman key exchange requested, but no DH parameters provided. Set ssh_dh= So...either there's an undocumented feature of SSH-over-IMAP (that's Dovecot - always on the cutting edge!) or someone had a coffee shortage during a coding session... -- Daniel
Possible attack?
I found an error in my log today...
Dec 17 12:03:30 bubba dovecot:
imap(us...@amfes.com)<23017>: Error: fts_solr:
received invalid uid '0'
Dec 17 12:04:44 bubba dovecot:
imap(us...@amfes.com)<25004>: Fatal: master:
service(imap): child 25004 killed with signal 11 (core dumps disabled -
https://dovecot.org/bugreport.html#coredumps)
I've now enabled core dumps (I think) and restarted - if it comes back
hopefully I can get a backtrace. But reading that fts_solr message, and
some other comments, leads me to wonder - could this be caused by
someone/thing trying to authenticate as root?
On that theory - I tried doing so via telnet - and received:
Dec 17 15:06:02 bubba dovecot: auth: Error:
plain(ultradeitytypeper...@amfes.com,127.0.0.1,<4kQr0z99UMZ/AAAB>): user
not found from any userdbs
Dec 17 15:06:02 bubba dovecot: imap: Error: Authenticated user not found
from userdb, auth lookup id=3522297857 (auth connected 1 msecs ago,
handshake 0 msecs ago, request took 1 msecs, client-pid=29572 client-id=1)
I have root's email aliased to a valid user's email. I'm not sure how
I'm able to authenticate as root - there isn't a root user defined in my
LDAP database and that should be the only auth backend enabled for
Dovecot. Or do I need to explicitly block local users from /etc/passwd
on the server? The only auth databases shown in doveconf -n:
userdb {
driver = prefetch
}
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
passdb {
args = /usr/local/etc/dovecot/master-users
driver = passwd-file
master = yes
}
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
and "master-users" doesn't list root either.
--
Daniel
SIS feature request
I tried SIS a couple years ago - I was very excited with the resulting decrease in storage requirements but the undiagnosed intermittent issues became too significant to ignore so I switched away. Recently I was thinking about it again. The primary issue with SIS seemed to be links would be deleted even though the source attachment files and related mails still existed. It was possible to either manually re-build the links or have a script scan the mail error log and perform such. I haven't looked at the code - but a thought for a possible "temporary" fix: 1. Whatever function in dbox code that performs the deletion of links - prior to actually deleting call a new function that will verify if any mails exist that reference it. A new function, without modifying existing code, may catch something the existing functions don't - and if it logs the fact that it was called and found something...perhaps we can find the flaw in the original algorithm. Just a thought. 2. In the mail retrieval function, if the attachment link doesn't exist - perform the relevant scan through the attachment database and if found re-create the link automatically. This should log an error but indicate the recovery. -- Daniel
Re: Solr
Joan, The reason for dropping squat, I'm assuming, is that Lucene and Solr potentially provide superior features & performance and as they are 3rd-party libraries & apps it reduces the maintenance responsibilities and let's the Dovecot team focus on mail server specific stuff - and let others focus on FTS. There is a *huge* difference between a functional Solr setup & squat - and if I'm able to get it working we should be able to get you there as well. I don't recall what OS you're running - I'm on Ubuntu 18.04. My Java version is OpenJDK 10.0.2. Attached is my complete Solr config. Try one more time - stop the server, delete the data folder, unpack the attached into the conf folder - and restart. I also have /etc/default/solr.in.sh: SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000" SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=6" SOLR_PID_DIR=/run/solr SOLR_HOME=/usr/local/lib Adjust the above folders as appropriate - or don't use them at all if you're using the defaults. /etc/systemd/system/solr.service: # put this file in /etc/systemd/system/ as root # below paths assume solr installed in /opt/solr, SOLR_PID_DIR is /data # and that all configuration exists in /etc/default/solr.in.sh which is the case if previously installed as an init.d service # change port in pid file if differs # note that it is configured to auto restart solr if it fails (Restart=on-faliure) and that's the motivation indeed :) # to switch from systemv (init.d) to systemd, do the following after creating this file: # sudo systemctl daemon-reload # sudo service solr stop # if already running # sudo systemctl enable solr # systemctl start solr # this was inspired by https://confluence.t5.fi/display/~stefan.roos/2015/04/01/Creating+systemd+unit+(service)+for+Apache+Solr [Unit] Description=Apache SOLR 7.5.0 After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket Before=multi-user.target graphical.target nginx.service dovecot.service Conflicts=shutdown.target [Service] LimitNOFILE=65000 User=vmail Group=mail ExecStartPre=/bin/mkdir -p /run/solr ExecStartPre=/bin/chown -R vmail.mail /run/solr PermissionsStartOnly=true PIDFile=/run/solr/solr-8983.pid Environment=SOLR_INCLUDE=/etc/default/solr.in.sh ExecStart=/opt/solr/bin/solr start ExecStop=/opt/solr/bin/solr stop Restart=on-failure RestartSec=15s TimeoutStopSec=30s [Install] WantedBy=multi-user.target graphical.target dovecot.service If you don't use systemd disregard - but see if any of the above applies for your setup. Let me know what happens. I agree this can be a mortal pain to setup - but it's worth it. Daniel On 12/21/2018 4:33 AM, Joan Moreau wrote: Dear Daniel. Thank you for your kind reply. Regarding NFS, no, there is nothing like this in my setup. Deleteing SOLR and recreating it, I did it so many times already. I started with *your* setup in the first place, as FTS_squat (which actually works very well and very straightforward, I have no clue why going for SOlr which is just a pain and not maintaining squat), and it leads to totally funny results (for instance, I type "emirates" in my "Air Companies" subfolder and get a lot of results .. but of competing companies :D ) I added the fts_enforce following AKi advice. I removed fts_decoder for the time being. I don't know where to go now. Dovcot still returning errors and SOlr still companinig with "Out of range" and other Java errors. Bottom line, I am back to squat, but as it is not maintained so crashed also times to times. I think we should discuss on (1) Why the damn choice of Solr has been main. As you empahised, maintainend so many independent software is a pain (2) If there is a real reason why going for SOlr, how to have a working (i.e. getting the right results to the end user) setup ? (3) If there iare no tangible reason, what about maintaining fts_squat , which did the job nicely for years and no complains about. On 2018-12-16 08:51, Daniel Miller via dovecot wrote: Joan, I understand and sympathize with your frustration - trying to get multiple applications to work together, particularly given the lack of documentation for some of them, can be extremely challenging. That said, I suggest you consider an alternative viewpoint. Frequently being misunderstood myself I apologize in advance if I'm reading you wrong - but it appears your view towards the situation is there is a bug in Dovecot related to this problem. That may well be - but I generally approach these matters from the assumption that *I* made the error in configuration and go from there. I'm not an official rep for any product nor claim to be any form of expert in these matters - but I do have a working setup and I'd like to help you if I can. If you're willin
Segfault report
Ubuntu 18.04, AMD Opteron, Dovecot Version 2.3.3, local file storage. I
believe it's one of my users checking mail remotely via mobile - don't
remember if it's an iPhone or Android.
gdb backtrace:
Reading symbols from /usr/local/libexec/dovecot/imap...done.
[New LWP 13852]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `dovecot/imap [kkhany@amfes.c'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 event_want_debug_log (event=event@entry=0x0,
source_filename=source_filename@entry=0x7efd84178aa3 "mail-storage.c",
source_linenum=source_linenum@entry=1261) at event-log.c:120
120 if (event->forced_debug)
(gdb) bt full
#0 event_want_debug_log (event=event@entry=0x0,
source_filename=source_filename@entry=0x7efd84178aa3 "mail-storage.c",
source_linenum=source_linenum@entry=1261) at event-log.c:120
ctx = {type = LOG_TYPE_DEBUG, exit_status = 0, timestamp = 0x0,
timestamp_usecs = 0, log_prefix = 0x0,
log_prefix_type_pos = 0}
#1 0x7efd83dc0986 in event_want_debug (event=event@entry=0x0,
source_filename=source_filename@entry=0x7efd84178aa3
"mail-storage.c", source_linenum=source_linenum@entry=1261)
at event-log.c:140
No locals.
#2 0x7efd840bf270 in mailbox_open_full
(box=box@entry=0x55704dc81058, input=input@entry=0x0) at mail-storage.c:1259
_tmp_event = 0x0
ret =
#3 0x7efd840bf57a in mailbox_open_full (input=0x0,
box=0x55704dc81058) at mail-storage.c:1368
ret =
ret =
_tmp_event =
_data_stack_cur_id =
_data_stack_cur_id =
#4 mailbox_open (box=0x55704dc81058) at mail-storage.c:1349
No locals.
#5 0x55704c36a31b in select_open (readonly=false,
mailbox=, ctx=0x55704dc13bc8) at cmd-select.c:288
client = 0x55704dc11de8
status = {messages = 1830951344, recent = 32766, unseen =
2391910144, uidvalidity = 1475818629, uidnext = 1830951424,
first_unseen_seq = 32766, first_recent_uid = 1832402502,
last_cached_seq = 32766, highest_modseq = 0,
highest_pvt_modseq = 4294967296, keywords = 0x55704dbf1380,
permanent_flags = 1280910144, flags = 21872,
permanent_keywords = false, allow_new_keywords = false,
nonpermanent_modseqs = false, no_modseq_tracking = false,
have_guids = false, have_save_guids = true, have_only_guid128
= false}
flags =
---Type to continue, or q to quit---
ret = 0
client =
status =
flags =
ret =
#6 cmd_select_full (cmd=, readonly=) at
cmd-select.c:417
client = 0x55704dc11de8
ctx =
args = 0x55704dbef690
list_args = 0x5d006e
mailbox = 0x55704dbe1540 "shared"
error = 0x55704dc11de8 ""
ret =
__func__ = "cmd_select_full"
#7 0x55704c371e30 in command_exec (cmd=cmd@entry=0x55704dc13a38) at
imap-commands.c:201
hook = 0x55704dbeb0f0
finished =
__func__ = "command_exec"
#8 0x55704c3701d2 in client_command_input (cmd=,
cmd@entry=0x55704dc13a38) at imap-client.c:1152
client = 0x55704dc11de8
command =
__func__ = "client_command_input"
#9 0x55704c370274 in client_command_input (cmd=) at
imap-client.c:1215
client = 0x55704dc11de8
command =
__func__ = "client_command_input"
#10 0x55704c370675 in client_handle_next_command
(remove_io_r=, client=0x55704dc11de8) at
imap-client.c:1257
---Type to continue, or q to quit---
No locals.
#11 client_handle_input (client=0x55704dc11de8) at imap-client.c:1271
_data_stack_cur_id = 3
ret =
remove_io = false
ret =
remove_io =
client = 0x55704dc11de8
handled_commands =
_data_stack_cur_id =
ret =
remove_io =
_data_stack_cur_id =
#12 0x55704c370ccc in client_input (client=0x55704dc11de8) at
imap-client.c:1317
cmd = 0x55704dc0bcb0
output = 0x55704dc2d150
bytes = 17
__func__ = "client_input"
#13 0x7efd83ddae0f in io_loop_call_io (io=0x55704dc13910) at
ioloop.c:698
ioloop = 0x55704dbe9ee0
t_id = 2
__func__ = "io_loop_call_io"
#14 0x7efd83ddc7c6 in io_loop_handler_run_internal
(ioloop=ioloop@entry=0x55704dbe9ee0) at ioloop-epoll.c:221
ctx = 0x55704dbedc00
events =
event =
list = 0x55704dc13970
---Type to continue, or q to quit---
io =
tv = {tv_sec = 1799, tv_usec = 999365}
events_count =
msecs =
ret =
i = 0
j =
call =
__func__ = "io_loop_handler_run_internal"
#15 0x7efd83ddaf1c in io_loop_handler_run (ioloop=)
at ioloop.c:750
No locals.
#16 0x7efd83ddb138 in io_loop_run (ioloop=0x55704dbe9ee0) at
ioloop.c:723
__func__ = "io_loop_run"
#17 0x7efd83d50873 in ma
Re: Segfault report
On 12/26/2018 1:32 AM, Aki Tuomi wrote: On 26 December 2018 at 11:26 Daniel Miller via dovecot wrote: Ubuntu 18.04, AMD Opteron, Dovecot Version 2.3.3, local file storage. I believe it's one of my users checking mail remotely via mobile - don't remember if it's an iPhone or Android. I believe this is fixed with https://github.com/dovecot/core/commit/4fcd4e8fad45dcaa637e4cb36a9f99204d69badf.patch on v2.3.4. Aki Just to be clear - fixed with v2.3.4, or need to apply a patch on top of it (that will be included in next point release)? Daniel
Re: Solr
On 12/29/2018 4:46 PM, Joan Moreau wrote: Hi Daniel, I am on Archlinux. Anyway, I adapted the scripts. 2 questions: 1 - It looks like we are not on the same version . I am on 7.5.0. Which version are you running ? Solr 7.5.0. 2 - Your conf shows that you let managed-schema but deleted schema.xml. What is the meaning of each ? schema.xml is the legacy configuration file. managed-schema is the config file used by current Solr versions. -- Daniel
Re: Solr
On 12/29/2018 4:49 PM, Joan Moreau wrote: Also : - Java is 10.0.2 Same as me. - If i delete schema.xml but create only managed-schema, the solr refuses to start with a java error "schema.xml missing" Ok...so we need to do some more digging. How did you install Solr? (I downloaded a "binary" installation and unpacked it) How did you create the dovecot instance? (I've provided explicit instructions for how I did it - did you follow those exactly or something different)? How are you starting Solr? (I use the provided "solr/bin/solr start" command, wrapped inside a systemd service). -- Daniel
Re: Solr
I'm running 7.5.0. The solrconfig.xml file is what I've modified over
time - I haven't started one from scratch for a while but perhaps I'll try.
Have you tried using the complete config that I sent you? With *all*
the files I included - and *none* of yours?
--
Daniel
On 1/1/2019 4:12 PM, Joan Moreau wrote:
The real main differecne seems coming from "diffconfig.xml"
When I put yours, Solr delete (!) schema.xml and create a
"manage-schema" and starts complaining about useless types (tdates,
booleans, etc..) that are not needed for Mail fileds
When I put mine (from standard distribution of Arch), it keeps things
as they are (yeah !), does not complains about those useless types and
startup properly.
I attach my diffconfig
But these are the configurations that one should adjust as per his/her
own use.
The main problem is : After some time of indexing from Dovecot,
Dovecot returns errors (invalid SID, etc...) and Solr return "out of
range indexes" errors
On 2019-01-02 07:49, Joan Moreau wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the
systemd installation script is included (and it is launching
/opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this
creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
On 2018-12-31 14:09, Daniel Miller wrote:
On 12/29/2018 4:49 PM, Joan Moreau wrote:
Also :
- Java is 10.0.2
Same as me.
- If i delete schema.xml but create only managed-schema, the
solr refuses to start with a java error "schema.xml missing"
Ok...so we need to do some more digging.
How did you install Solr? (I downloaded a "binary" installation
and unpacked it)
How did you create the dovecot instance? (I've provided explicit
instructions for how I did it - did you follow those exactly or
something different)?
How are you starting Solr? (I use the provided "solr/bin/solr
start" command, wrapped inside a systemd service).
--
Daniel
--
--
Daniel
Re: Solr
On 1/2/2019 12:59 AM, M. Balridge wrote: So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the following functionality: 1) The ability to search for a string within any of the structured fields (from/subject) that returns correct results? Yes. 2) The ability to search for any string within the BODY of emails, including the MIME attachment boundaries? Yes. 3) The ability to do "ranging" searches for structures within emails that decompose to "dates" or other simple-numeric data? Dunno - I don't think I've needed that and I'm not sure how to do it. My mail clients are Thunderbird and AquaMail (on Android). If you'll give me either the desired Thunderbird steps or telnet-based IMAP command I'm happy to test. OPTIONALLY, and this is probably way outside of the scope of the above, despite the fact that it's listed as a "selling point" of SOLR versus other full text search engines: 4) The ability to do searches against any attachments that are able to be post-processed and hyper-indexed by SOLR+Tika? Haven't tried. SOLR seems to have "brand cachet", so presumably it actually works (for somebody). It works - just sometimes needs more effort to setup than it should. Dovecot has not a little "brand cachet", and for me, I have innate faith and trust in Timo and his software. I think we're all in agreement here. But please, level with us faithful users. Does this morass of Java B.S. actually work, and if not, please just deprecate and remove this moribund software, and stop trying to bury the only FTS plugin many of us HAVE actually gotten to work. (Pretty please?) I respect that Messr. Moreau has made an earnest effort to get this JAVA B.S. to actually work, as I have. He persevered where I'd given up. He's vocal about it, and now I'm chiming in that this ornate collection of switchblades only cuts those who try to use them. Short answer - it actually works. Longer answer - I've gone through a hate/love/hate/like relationship with Solr myself. The transition from v3 to v4 was a major headache - and I gave up for a while. But versions 6 & 7 have been pretty good for me. I'm neither a Dovecot nor a Solr developer - just enough of a fiddler to get them working to fulfill my own needs. If my unreliable memory serves I believe the Dovecot fts-solr plugin hasn't needed to change much (I recall one significant change required when Solr changed it's protocol - I think an XML/JSON thing). So having a stable interface let's Timo & Co. forget about on-going FTS development and continue focusing on things not provided by other tools. Hopefully they'll revisit SIS... I recall reading something about the Lucene library (which Squat & Solr are based on) and again my memory is the C version(s) weren't getting maintained as well as might be desired. I think having the Solr/Lucene team focusing on Java development was another point of consideration for Dovecot's squat - but I could be totally off here. Based on the errors reported by Joan I believe that system's problems are due to configuration - either Solr, Dovecot, or both. They don't sound like Java related issues (which are a *major* pain to deal with!). I've provided a copy of what is a working configuration *for me*. I'm happy to continue helping as best I can - and if Joan, you, or anyone else would like my aid I'll do my best. If you're crazy I-mean-trusting enough to have me SSH or remote view to your system I'm willing to take a look. I've had enough people help me over the years for various packages that I'd like to pay it forward where I can. -- Daniel
Re: Solr
On 1/1/2019 3:49 PM, Joan Moreau via dovecot wrote:
Hi
Solr is a standard package in ArchLinux. ("pacman -S solr") . the
systemd installation script is included (and it is launching
/opt/solr/bin/solr.in.sh)
Instance : sudo -u solr /opt/solr/bin/solr create -c dovecot -> this
creates a separate folder with default solrconfig.xml, schema.xml, etc..
I made a symlink of the data folder to a second drive (ext4) much bigger
I'm using that nasty word *should*...in that the above installation
*should* yield working results. But...since I don't use Arch and have
no insight into it I suggest downloading a binary tarball from the Solr
site and do a clean install. It may behave identically...or maybe
something will be different.
--
Daniel
Re: Solr
On 1/3/2019 10:56 AM, Tanstaafl wrote: On 12/21/2018, 11:19:42 AM, Daniel Miller via dovecot wrote: There is a *huge* difference between a functional Solr setup & squat Interesting. Care to elaborate? This is one of those things that has to be experienced to be understood. When you can perform an FTS search across (pause while I check current stats...): du -c -h /var/mail 136G Solr numDocs: 520102 and using any IMAP client that supports server-side searches (like Thunderbird & AquaMail) the results are basically instantaneous...it's worth the effort. And that's searching a Dovecot virtual folder defined as "* all", including all my archives, all my list subscriptions, and all the shared Inbox/Sent folders from my other users. But I certainly wish it was easier to setup. -- Daniel
Re: Solr
On 1/5/2019 9:58 AM, Tanstaafl wrote: Thanks Daniel... So, as one who has no experience of the benefit of either... How does this compare with Squat? Meaning, Is it exponentially faster? Twice as fast? It's been many years since I last had a Squat setup - but that's my memory. -- Daniel
Re: Rsync to backup dbox with SIS
On 1/25/2019 1:33 PM, ash-dove...@comtek.co.uk wrote: We will be deploying a replacement Dovecot server soon, and we are planning to use maildir for the primary storage, but with an archive namespace using mdbox (or perhaps sdbox), and SIS. Our backup servers and (luke)warm spare server need to obtain full copies of the mail store. For the maildirs I know I can simply use rsync (we already use it here). I'm a little wary of using rsync with mdbox and SIS though. Significantly limited knowledge opinions below: Probably not the answer you want - but I would strongly suggest using Dovecot replication. Dovecot replication Just Works - so don't reinvent the wheel when Timo provided such a polished tool already. And based on my previous SIS experience - while dbox is nice I would suggest avoiding SIS until there are reports of more development. Sdbox will be solid - which is what you want for an archive - though maildir would be the safest. Archives don't need to be rapid-access - they need to be dependable. SIS is wonderful for space saving - but until there's more safety checks built-in I'd suggest avoiding it for production backups. Drives are cheap - lost data, lost time, lost hair, lost sanity...is not. I'm aware of "doveadm backup", and (although it currently throws up a few errors) it seems like it might be a valid solution for our warm spare server. Our backup servers, on the other hand, aren't supposed to be visible to the production machines, with the exception of the backup machine sshing in to do an rsync each night. We can't install dovecot on them. The backups don't have to be "visible" to other machines - don't even have to be running IMAP/POP services (I think). And the replication command is run - via ssh (see https://wiki.dovecot.org/Replication) - so what's the problem? -- Daniel
Re: Dovecot and FTS experiment
On 1/29/2019 9:15 AM, Tomasz Nowak wrote: Hello, I'm trying to experiment with Dovecot and Solr server. I have >30k email addresses that I want to index to speed up searching and save IOPS on mail servers. For now - I'm doing some experiments and I'm testing how it is working. I'm thinking about adding one additional server with Solr and configure all mail servers to use that server. I have some questions. 1. I have 15 mail servers. It will be good If I add new server with Solr and use it on all Dovecot servers? Or maybe I should install Solr on all mail servers? You need to start somewhere. If you've never played with Solr before I suggest you start with one and get it working before you explore "sharding". When you're ready for that you should consult the solr mailing list. The importance of enough RAM for Solr cannot be overstated. 2. I notice - I have mail account with 3GB of mail. Index files in mail dir has 5MB. After indexing mailbox in Solr - index files has 15MB. What changes in those files? FTS indexing adds something to that files - but what? What mail storage format are you using? dbox? Thinking...I believe that Dovecot records which mails have been reported to the FTS. That may help account for the increased size. -- Daniel
Re: Sis to deduplicate attachments does not work?
On 4/23/2019 1:53 AM, luckydog xf via dovecot wrote: Hi, I use sis to deduplicate attachments, here is my `doveconf -n` [...] mail_location = maildir:/var/mail/%n/Maildir [...] SIS is a function of dbox. You're using Maildir. -- Daniel
Re: Sis to deduplicate attachments does not work?
On April 23, 2019 10:54:38 PM luckydog xf wrote: Is it worthwile to use dbox? seeing from http://www.linuxmail.info/mbox-maildir-mail-storage-formats/ it may cause file lock and easy to corrupt. As with everything - it depends. You're asking me so these are *my* opinions - and I do not claim to be anything more than a hobbyist/tinkerer when the comes to this. mbox has potential use for long term read-only archives - I see no reason to use it for live mailboxes. maildir is undoubtedly the least susceptible to corruption. It's also the slowest format for reading. How slow is "slow" depends on your hardware - it may be imperceptible with enough RAM and SSD's - or it may result in user complaints with large mailboxes. dbox is Dovecot's preferred format. I know Timo has put a lot of effort into it. sdbox is similar to maildir in that each mail is a separate file. mdbox significantly reduces the number of files which can make file-based backups faster. Both dbox formats are dependent on their index files. If you've got good hardware, including a proper UPS, I'd recommend dbox (my server is presently using sdbox). With large mailboxes and file-based backups you'll benefit from mdbox. When reliability is the #1 concern above anything else - use maildir. Depending on your use SIS can have significant impact on storage requirements - but storage these days is relatively cheap. I haven't seen much feedback from users actively using SIS - I'd love to hear from high traffic sites with SIS experience to know if the corruption issues have been resolved. In my case there was at least a 30% reduction in space but I had too many errors - admittedly it's been a couple years since I last tried it. -- Daniel
Re: Understanding virtual mailboxes (examples in 15-mailboxes.conf)
On 4/30/2019 11:13 PM, MRob via dovecot wrote:
The examples in 15-mailboxes.conf
# If you have a virtual "All messages" mailbox:
#mailbox virtual/All {
# special_use = \All
# comment = All my messages
#}
# If you have a virtual "Flagged" mailbox:
#mailbox virtual/Flagged {
# special_use = \Flagged
# comment = All my flagged messages
#}
They seem to reference some kind of virtual mailbox setup that doesn't
compare to the docs for the "virtual" plugin. That plugin says we should
create a separate namespace instead, like "namespace virtual" and put
files representing the virtual folders into user maildirs. What if we
use mdbox? add the files to user/mailboxes director I will guess.
Is there a way to use the mailbox examples in the inbox namespace in the
default config? Does it use some other method different from the virtual
plugin? maybe more config hints for those examples would be helpful.
Thank you.
You will indeed need to setup a virtual namespace. The virtual mailboxes
will exist in a folder alongside but separate from your primary
mailstore. If your default namespace is:
namespace inbox {
type = private
separator = /
prefix =
location = maildir:/var/mail/%d/%n/Maildir
inbox = yes
hidden = no
list = yes
subscriptions = yes
}
then add
namespace virtual {
prefix = virtual/
separator = /
location = virtual:/var/mail/%d/%n/virtual
subscriptions = no
list = children
}
So for user dan...@somedomain.org there will exist:
/var/mail/somedomain.org/daniel/Maildir
/var/mail/somedomain.org/daniel/virtual
And then you'll need to create the virtual definition files for each
user's mailbox as needed.
--
Daniel
Re: dynamic virtual mailboxes?
On 5/2/2019 12:47 PM, MRob via dovecot wrote: hi, I spent time learning about virtual mailboxes. Is there some way to create dynamic virtual mailboxes? I mean, when I look at a mailbox, I want to see only unread messages or flagged messages in that mailbox. contents of /var/mail/mydomain/myuser/virtual/Flagged/dovecot-virtual: * flagged contents of /var/mail/mydomain/myuser/virtual/Unread/dovecot-virtual: * unseen contents of /var/mail/mydomain/myuser/virtual/Unread-Flagged/dovecot-virtual: * unseen flagged -- Daniel
Re: Understanding virtual mailboxes (examples in 15-mailboxes.conf)
On 5/3/2019 11:18 AM, MRob via dovecot wrote:> Thank you, but question is about the example mailbox settings in 15-mailboxes.conf I found I can put those mailbox definitions in the new virtual namespace, still not sure if they would work if I kept them in the inbox namespace, maybe the documentation in the example file can include clarification No - you need to keep them in a separate namespace. -- Daniel
Re: dynamic virtual mailboxes?
On 5/3/2019 11:22 AM, MRob via dovecot wrote:> That is not dynamically generated and it isn't limited to just one mailbox (dovecot terminology here is confusing, normally a mailbox is a mail account (user), but in this context "mailbox" I guess mean "folder" which is how I am using it, as it is used in 15-mailboxes.conf) What does "dynamically generated" mean? Are you asking to create virtual mailboxes via your mail client? If so - then there's no native method. The examples I gave will indeed give you virtual "folders" which respectively contain ALL your flagged (regardless of seen status), unseen (regardless of flagged status), or only unseen flagged messages from all folders. And they will auto-update. Virtual mailboxes are defined per user - so indeed the examples I gave will only exist for the user(s) that have such files and will only apply to their folders. -- Daniel
Re: dynamic virtual mailboxes?
On 5/5/2019 10:50 PM, MRob via dovecot wrote:> Thank you for helping but- Again, Dovecot terminology here, mailbox means 'folder' not the whole account Dynamic- https://www.mail-archive.com/dovecot@dovecot.org/msg71091.html Ahh...I understand what you want now. Yes it would be nice - no that ability does not exist. -- Daniel
FTS Xapian
For my primary namespace this is working fine - thanks to the developers! It also appears to work great for shared folders as well. But my virtual folders aren't returning results - at least not to the client. The logs show FTS Xapian opening several DB files and getting results - but nothing is being returned to client. Is this a config issue on my side or is this a current limitation of the plugin? -- Daniel
Re: FTS Xapian
ot;dovecot" OR
bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot")
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: 15 results in
39 ms
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: Opening DB (RO)
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_d5359c092c8b584ee25d3bc41c5f
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: Query=
(subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR
bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot")
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: 9 results in
37 ms
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: Opening DB (RO)
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_d6359c092c8b584ee25d3bc41c5f
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: Query=
(subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR
bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot")
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: 49 results in
35 ms
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: Opening DB (RO)
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_e84b2f0bed746259565f3bda95b5
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: Query=
(subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR
bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot")
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: 11 results in
18 ms
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: Opening DB (RO)
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_e8a36d2782404c56de4b9db5accb
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: Query=
(subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR
bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot")
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: 80 results in
54 ms
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: Opening DB (RO)
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_701a2a2d6848815c750e9db5accb
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: Query=
(subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR
bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot")
Jun 5 06:02:29 bubba dovecot:
imap(dmil...@amfes.com)<25877>: FTS Xapian: 54 results in
43 ms
doveconf -n:
# 2.3.6 (7eab80676): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.6 (92dc263a)
# OS: Linux 4.15.0-50-generic x86_64 Ubuntu 18.04.2 LTS
# Hostname: bubba.amfes.lan
auth_cache_size = 4 k
auth_master_user_separator = *
auth_mechanisms = plain login
default_login_user = nobody
default_vsz_limit = 2 G
dict {
acl = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
imap_client_workarounds = tb-extra-mailbox-sep
imap_idle_notify_interval = 29 mins
listen = *
login_trusted_networks = 192.168.0.0/24
mail_attachment_hash = %{sha512}
mail_plugins = fts fts_xapian acl zlib virtual
mail_prefetch_count = 10
mail_shared_explicit_inbox = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date index ihave
duplicate mime foreverypart extracttext
namespace inbox {
hidden = no
inbox = yes
list = yes
location =
mailbox "Deleted Messages" {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox INBOX/Archives {
auto = no
special_use = \Archive
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Items" {
auto = no
special_use = \Sent
}
mailbox "Sent Messages" {
auto = no
special_use = \Sent
}
mailbox Trash {
auto = subscribe
autoexpunge = 30 days
special_use = \Trash
}
mailbox virtual/Flagged {
comment = All my flagged messages
special_use = \Flagged
}
prefix =
separator = /
subscriptions = yes
type = private
}
namespace usershares {
list = children
location = sdbox:/var/mail/%%d/%%n/sdbox
prefix = shared/%%n/
separator = /
subscriptions = no
type = shared
}
namespace virtual {
list = children
location = virtual:/var/mail/%d/%n/virtual
prefix = virtual/
separator = /
subscriptions = no
}
passdb {
args = /usr/local/etc/dovecot/master-users
driver = passwd-file
master = yes
}
passdb {
args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
acl = vfile:/usr/local/etc/dovecot/global-acls:cache_secs=300
acl_shared_dict = proxy::acl
fts = xapian
fts_autoindex = yes
fts_autoindex_exclude = \Trash
fts_autoindex_exclude2 = \Junk
fts_autoindex_exclude3 = \Spam
fts_enforced = no
fts_index_timeout = 20
fts_xapian = partial=2 full=20
mailbox_alias_new = Sent Messages
mailbox_alias_new2 = Sent Items
mailbox_alias_new3 = Deleted Messages
mailbox_alias_old = Sent
mailbox_alias_old2 = Sent
mailbox_alias_old3 = Trash
sieve = file:~/sieve;active=~/.dovecot.sieve
vsz_limit = 4G
}
protocols = imap lmtp sieve
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = mail
mode = 0600
user = vmail
}
}
service dict {
unix_listener dict {
group = mail
mode = 0660
user = vmail
}
}
service imap-login {
process_min_avail = 10
service_count = 1
}
service imap-postlogin {
executable = script-login /usr/local/etc/dovecot/post-login.sh
user = $default_internal_user
}
service imap {
executable = imap imap-postlogin
}
service indexer-worker {
process_limit = 3
}
service lmtp {
process_min_avail = 5
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = mail
mode = 0666
user = vmail
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
inet_listener sieve_deprecated {
port = 2000
}
process_min_avail = 0
service_count = 1
}
ssl_cert = mail_plugins = fts fts_xapian acl zlib virtual sieve postmaster_address =
postmas...@amfes.com
}
protocol lda {
mail_plugins = fts fts_xapian acl zlib virtual sieve}
protocol imap {
mail_max_userip_connections = 50
mail_plugins = fts fts_xapian acl zlib virtual imap_acl imap_zlib
mailbox_alias
}
local 192.168.0.2 {
protocol imap {
ssl_cert = On June 4, 2019 10:03:47 PM Joan Moreau via dovecot
wrote:
Hi
Can you post your dovecot conf file and the subset of the log files related
to the issue ?
thanks
On June 5, 2019 9:29:13 AM Daniel Miller via dovecot
wrote:
For my primary namespace this is working fine - thanks to the developers!
It also appears to work great for shared folders as well.
But my virtual folders aren't returning results - at least not to the
client. The logs show FTS Xapian opening several DB files and getting
results - but nothing is being returned to client. Is this a config
issue on my side or is this a current limitation of the plugin?
--
Daniel
Re: FTS Xapian
Yes, latest git version. The logs show (as I read them) returned results - yet nothing shows in the client. The logs look the same (with different numbers) when querying "regular" folders - but results are shown in clients. -- Daniel On June 6, 2019 12:16:08 AM Joan Moreau wrote: Hi Are you using the latest git version ? WHich part exactly of your logs relates to "virtual folders do not work" ? On 2019-06-05 13:08, Daniel Miller via dovecot wrote: Logs: Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2857830c70c844e2f1d3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 1 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_78544714f3f1ae5b9b0d3bda95b5 Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 53 results in 40 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_bdcb8e2172fadf4db50b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 12 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_be25c00241fedf4de00b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 3 results in 32 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_a7e75820d9fadf4dd90b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 11 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6fa78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 21 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6ea78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes
Re: Some questions
On 7/9/2019 6:17 AM, Jérôme Bardot via dovecot wrote: Hello, This is my first email here. I want to understand well how dovecot is integrate with ldap in a postfix/dovecot/ldap setup. I use a debian server. Perfectly! More specifically what dovecot need in ldap to work. I saw we can use several "mode" related to virtual domain, etc. For "start" i only need one domain with several address. I currently use fusiondirectory for manage my ldap users. i guess i can use that schema to auto create users email (name.firstn...@domain.tld for ie) ? I also want to setup some aliases and share directory based on ldap group/role can i do it ? An other question is can we have two domain name for imap.domain.tld && smtp.domain.tld ? Yes. Dovecot & Postfix have no "hard" schema, or database definition, or particular fields. You need to create map files which tell each server how to use the information from LDAP (or any other database). Each server (Postfix & Dovecot) have their own configuration which is separate from each other. So you need to start with one or the other. Postfix questions should be asked on the Postfix list. Everything you asked for above is easily doable - just start with one step at a time. Ask specific questions when you get stuck. -- Daniel
Namespace structure
Is the following "legal" for Dovecot? And...is this separation
recommended or a bad idea? Particularly I'm asking about the "archives"
namespace - I haven't actually implemented this yet and I'm checking
before I break something.
10-mail.conf
# Primary private namespace
# Using sdbox for storage
namespace inbox {
type = private
separator = /
prefix =
location = sdbox:/var/mail/%d/%n/sdbox
inbox = yes
hidden = no
list = yes
subscriptions = yes
}
# For long-term archival
namespace archives {
type = private
separator = /
prefix = Archives/
location = mdbox:/var/mail/%d/%n/Archives/mdbox
subscriptions = no
list = children
}
# Shared mailboxes
mail_shared_explicit_inbox = yes
namespace usershares {
type = shared
separator = /
prefix = shared/%%n/
location = sdbox:/var/mail/%%d/%%n/sdbox
subscriptions = no
list = children
}
# Virtual mailboxes - for server-side searches
namespace virtual {
prefix = virtual/
separator = /
location = virtual:/var/mail/%d/%n/virtual
subscriptions = no
list = children
}
--
Daniel
doveadm mailbox list
It's quite likely I'm doing it wrong, but... Given a valid mailbox... doveadm mailbox list -u realmb returns "realmb" doveadm mailbox list -u real* returns "realmb" Seems reasonable. Now, with a non-existent mailbox... doveadm mailbox list -u bogus returns "bogus" doveadm mailbox list -u bogus* returns "" Is this a bug or correct behavior? -- Daniel
Re: fts_solr: Error: fts_solr: received invalid uid '0'
On 9/13/2019 1:21 AM, Fabian via dovecot wrote: Hi, we are trying to add full text search functionality with Solr to our Doveoct setup. Our Versions: OS: Debian 9 Tried versions: - Dovecot 2.2.7 with Solr 3.6 - Dovecot 2.3.4 with Solr 8.2 (2.2.7 from offical Debian repository, 2.3.4 from backports) Search is working mostly of the time perfrectly smooth. But sometimes following message appears in mail.err: dovecot: imap(username)<16189>: Error: fts_solr: received invalid uid '0' If this error occurs our webmail frontend delivers most of the time a timeout. Sometimes the search only takes really long. Are there any ideas why this error occurs? We are not able to reproduce the error in such a way that it would always be reproducible. However, we can reproduce the behavior in some form over and over again - but we do not know exactly what is decisive. Are you limiting Solr's memory usage? How much available memory is on your server? To shortcut the conversation - if you don't have at least 16G of *free* RAM it's time to upgrade. My own server has 32G installed - I used to have 16G. My own Solr problems basically disappeared after adding RAM. And I only serve a few users - my own mailstore is the largest as I keep most of my mails. If you're serving 20+ users you'd probably benefit from doubling to at least 64G. -- Daniel
Namespace overlap
Given an existing default namespace:
namespace inbox {
type = private
separator = /
prefix =
location = sdbox:/var/mail/%d/%n/sdbox
inbox = yes
hidden = no
list = yes
subscriptions = yes
}
And mailboxes like:
INBOX
INBOX/Archives
INBOX/Archives/2018
if I then define a new namespace:
namespace archives {
type = private
separator = /
prefix = Archives/
location = mdbox:/var/mail/%d/%n/Archives/mdbox
subscriptions = no
list = children
}
What will happen to the previous existing mailboxes & mails? Will they
simply be "masked" by the new namespace and remain pending other
operations? Or would they be moved/deleted?
If they remain - is it possible to refer to the old mailboxes via either
IMAP or doveadm?
--
Daniel
Re: Imaptest stall
On 9/17/2019 12:58 AM, Marc Roos via dovecot wrote: I have been testing with imaptest and getting 'stalls', I tried even building from source and static. Even running it on the same host. Anyone knows what I could doing wrong? [@~]# ./imaptest - append=100,0 logout=0 host=192.168.10.44 port=143 user=test2 pass= seed=100 secs=240 clients=1 mbox=64kb.mbox box=INBOX/test What are you trying to test? Do the Dovecot logs show any connections? -- Daniel
Re: Imaptest stall
If you're just speed testing for writing probably sdbox or maildir would be the fastest. Daniel On 9/17/2019 1:09 PM, Marc Roos via dovecot wrote: Yes dovecot is showing the inserted messages until the stall. Looks like it is an issue with imap test because I am able to empty the mailbox again via thunderbird. I am comparing write tests to different backends. -Original Message- From: Daniel Miller [mailto:dmil...@amfes.com] Sent: dinsdag 17 september 2019 22:06 To: Marc Roos; dovecot Subject: Re: Imaptest stall On 9/17/2019 12:58 AM, Marc Roos via dovecot wrote: I have been testing with imaptest and getting 'stalls', I tried even building from source and static. Even running it on the same host. Anyone knows what I could doing wrong? [@~]# ./imaptest - append=100,0 logout=0 host=192.168.10.44 port=143 user=test2 pass= seed=100 secs=240 clients=1 mbox=64kb.mbox box=INBOX/test What are you trying to test? Do the Dovecot logs show any connections? -- Daniel
Re: fts_solr: Error: fts_solr: received invalid uid '0'
On 9/19/2019 6:28 AM, Fabian via dovecot wrote: Thanks for your response! No we are not limiting Soli’s memory usage. After your tip, we've also upgraded the memory to 32GB. But the behavior remains the same. I have also already considered that Dovecot may index the UID incorrectly. But if I search the index directly, I don't find any entries with UID = 0, so I have no idea where this "fts_solr: received invalid uid '0"" message might come from. In our test environment we actually indexed only one user. The user's mailbox contains about 100.000 mails. This means that there is not really much data in the index. Are there any other hints or tips regarding this „invalid uid ‚0‘"-message? Logfile: Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: Debug: http-client[1]: peer 172.17.10.12:8983: Creating 1 new connections to handle requests (already 0 usable, connecting t$ Your post has truncated the lines (right margin). Re-post with the full lines. -- Daniel
Re: File manager or browser for IMAP?
Not defending Thunderbird - but I don't understand your "taking hours to load my Dovecot IMAP". I suppose if you have sync enabled then the first time you connect to a large mailstore there would be an initial download. But...I always disable sync immediately upon setting up accounts in Thunderbird so that's never been an issue for me. Being unable to prevent downloads or utilize server-side searches is why some other clients have been disappointing for me - like EM Client and Mailbird. Daniel On 9/23/2019 5:36 PM, Steve Litt via dovecot wrote: Thunderbird is an absolute pig, taking hours to load my Dovecot IMAP. Claws-mail is good, but I have some problems with it. Alpine appears not to be ready for prime time to act as a window into IMAP. Same with the rest I've tried. SteveT On Tue, 24 Sep 2019 00:21:33 +0200 Ionel Spanachi wrote: Why not use thunderbird (or any other IMAP talking client)? :-) Ionel On 24.09.19 00:14, Steve Litt via dovecot wrote: Hi all, I could really use a file manager or browser to browse my Dovecot IMAP. Ideally it would have hotkeys to move, copy, delete and send. The send part needn't be coded: Just a call to a shellscript which can handle the send the way it's locally the most convenient. Anyone know of such a file manager or browser for IMAP? SteveT Steve Litt Author: The Key to Everyday Excellence http://www.troubleshooters.com/key Twitter: http://www.twitter.com/stevelitt
Re: Password issue
On 10/9/2019 6:58 PM, @lbutlr via dovecot wrote: On Oct 9, 2019, at 5:23 PM, @lbutlr wrote: Postfix logs "Client host rejected: Access denied” but as I said, other accounts can submit and there’s nothing special in the submission service in master.cf. submission inet n - n - - smtpd -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_sasl_type=dovecot -o smtpd_sasl_security_options=noanonymous -o smtpd_sasl_path=private/auth -o smtpd_milters= -o milter_connect_macros= -o milter_macro_daemon_name=ORIGINATING -o syslog_name=postfix/submit -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_data_restrictions= -o smtpd_relay_restrictions=permit_sasl_authenticated,reject -o smtpd_helo_restrictions= -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject I suggest you re-post this to the Postfix as this is a Postfix issue. However, before doing so, reference http://www.postfix.org/DEBUG_README.html To begin with, I'd suggest adding a "-v" to the smtpd command above, followed by a Postfix reload, and test sending again. If that doesn't reveal your issue re-post to the Postfix list, and include the output of "postconf -n". BTW - I'm assuming the duplicate smtpd_recipient_restrictions line at the end is an email artificat. -- Daniel
Re: Still trying to get past authorization problems
In conf.d/10-logging.conf, set: auth_debug_passwords = yes mail_debug = yes verbose_ssl = yes You might try setting them one-by-one as having all three will give a ton of info, and auth_debug_passwords will expose all passwords used while set, but those settings should show you what the problem is. Daniel On 10/24/2019 6:23 AM, Steve Matzura via dovecot wrote: That's already in conf.d/10-auth.conf. On 10/24/2019 1:31 AM, Aki Tuomi via dovecot wrote: On 24.10.2019 6.18, Steve Matzura via dovecot wrote: Got all the Postfix errors fixed but maybe one, so I don't think that's involved in this mix any more. I had a domain definition problem, got that sorted. The accounts' logins are correct. I tried several from the shell, and they let me in. Here's the minus-n output, not very different from the first time I posted it: Try adding auth_mechanisms = PLAIN LOGIN and do not use [x] secure password in your MUA. Aki
subscription namespace
The current documentation makes mention of a "special" subscription
namespace. The example given:
namespace subscriptions {
subscriptions = yes
prefix = ""
list = no
hidden = yes
}
namespace inbox {
inbox = yes
location =
subscriptions = no
[...]
results in a startup error as both namespaces have the same prefix. Was
the intent for the "inbox" namespace to have an explicit "INBOX/" prefix?
If this is configured for an existing server that previously had no such
"INBOX/" prefix namespace - will clients need to be reconfigured?
--
Daniel
SQL iterate_query
I've been hunting some ghost mailboxes - and I *think* I found the source. I use the complete email address as the username, and store such in a database. The storage structure is location=/var/mail/%d/%n. Not unusual I think. So all I *should* see from "ls /var/mail" would be a list of domains. But I keep seeing empty mailboxes being created at this level. Having corrected a few other errors I *hope* I've found the last one - but if I'm right I believe the docs need updating: The examples given for SQL userdb's include: iterate_query = SELECT userid AS username, domain FROM users So this means the username is returned for *both* the username and domain. Even if I'm wrong as to the cause of my own troubles this can't be right. It just can't. Or am I mistaken? So, given that the complete address is used as the username I now use: iterate_query = SELECT username FROM mailbox (I'm using postfixadmin to administer this - and "mailbox" is the default user table name) I believe the alternative would be an explicit: iterate_query = SELECT username, domain AS username, domain FROM users I don't *think* that would make any security difference for my use case so why add the extra processing? I believe the documentation should be updated, or at least clarified, on this issue. -- Daniel
Re: MariaDB database for users and passwords?
There is some ambiguity in the setting names, however: In the "upper" authentication config file (possibly conf.d/auth-sql.conf.ext) you define which "internal" driver the authentication system will use. These are...more of a top-level engine selection if you will - perhaps not what you'd consider a "true" driver. In the "lower" authentication config file (like dovecot-sql.conf.ext), which is referenced by the 'args' setting in the userdb & passdb sections of the "upper" file, is where you explicitly specific the "true" driver, the actual database, and any field mappings. If you're just getting things setup I suggest you check out: http://postfixadmin.sourceforge.net/ Very clean & simple admin GUI for mail services. It includes documentation for setting up Dovecot. Daniel On 11/8/2019 11:12 PM, Aki Tuomi via dovecot wrote: On 09/11/2019 05:44 Ken Wright via dovecot wrote: On 11/8/19 3:40 PM, Alexander Dalloz via dovecot wrote: Am 08.11.2019 um 21:23 schrieb Ken Wright via dovecot: On 11/8/19 3:14 PM, @lbutlr via dovecot wrote: On 08 Nov 2019, at 11:56, Ken Wright wrote: Nov 8 13:28:53 grace dovecot: auth: Fatal: Unknown passdb driver ‘ You do not have Dovecot compiled with support for mysql' But the dovecot-mysql package is installed! Why can't it see that? The driver is called "sql". See https://doc.dovecot.org/configuration_manual/authentication/sql/ Alexander Are you sure? I looked at that page, and it says there are different drivers for MySQL and PostgreSQL: mysql and pgsql respectively. I also checked dovecot.conf, and there the driver is called "sql." Ken SQL is the **authentication** database, which has mysql **driver**. So in dovecot.conf you use sql, and in the config file for the sql authentication, you specify the driver. See https://github.com/dovecot/core/blob/master/doc/example-config/dovecot-sql.conf.ext#L32 Aki
Re: http API for IMAP
On 11/13/2019 11:59 PM, Thomas Güttler via dovecot wrote: Am 13.11.19 um 17:21 schrieb Ralph Seichter via dovecot: * Thomas Güttler via dovecot: AFAIK you can't sent a link/URL to a mail on a shared folder to a friend. Like "Hi bob, she loves me. See this message from here https:/./" Regards, Thomas Güttler Actually - why not? It doesn't seem that difficult (at an abstract level) to implement such with available tools. PHP has built-in support for IMAP - so creating an interface that maps HTTP URI's to IMAP commands doesn't look too bad. I might even suggest leveraging existing platforms like Nextcloud - instead of creating a whole new authentication, authorization, processing, and presentation framework you'd "simply" write a Nextcloud add-on that publishes IMAP folders/messages in whatever manner you prefer. Nextcloud already provides for file-sharing - so I see this as a good fit. Daniel
Re: Possible hack via doveadm
I only allow explicit service traffic through. IMAPS, SMTPS, etc. If doveadm is communicating via the IMAP(S) ports then all I can do via firewall is block countries. Which of course I can but I'm asking about any additional hardening for Dovecot itself. -- Daniel On May 13, 2023 6:25:06 PM jeremy ardley via dovecot wrote: On 14/5/23 09:14, Daniel L. Miller via dovecot wrote: May 12 15:45:58 cloud1 dovecot: doveadm(194.165.16.78): Error: doveadm client not compatible with this server (mixed old and new binaries?) May 13 03:44:31 cloud1 dovecot: doveadm(45.227.254.48): Error: doveadm client not compatible with this server (mixed old and new binaries?) Since I don't recognize those IPs, the first is out of Panama and the other is Belize, I assume these are hostile attackers trying to exploit something. How can I defend against this? Set up a firewall rule that only allows access from an IP range you control. For any other source, simply drop the connection. You can get really fancy and use port forwarding using ssh to connect from remote but appear as localhost to the server. This access can be configured in dovecot as well as firewall Jeremy ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
