Problem with tika
Hello all,
For some time now we've bin using solr as a search engine (working
great). I have added tika for searching inside documents, however it
keeps crashing when indexing. When just indexing mails that arrive works
good, but a reindex on all mail constantly crashes with below stacktrace.
When I observe the packets with wireshark I see a HTTP-flow going to
tika and suddenly, midway a document dovecot (or the server) sends a
RST. Sometimes after thousands of documents, sometimes after a few.
Usually after a few hundred.
It appears there are less errors using http than https.
Relevant config:
OS: CentOS6, fully updated
plugin {
fts = solr
batch_size = 1
fts_solr =
url=https://username:passw...@solr-01.vevida.net:443/solr/dovecot/
#fts_tika = https://solr-01.vevida.net:443/tika/
batch_size = 1000
fts_autoindex=yes
soft_commit=no
}
# dovecot --version
2.3.9.3 (9f41b88fa)
# Configure options:
--docdir=%{_docdir}/dovecot \
--disable-static \
--with-nss \
--with-shadow \
--with-pam \
--with-gssapi=plugin \
--with-ldap=plugin \
--with-sql=plugin \
--with-pgsql \
--with-sqlite \
--with-zlib \
--with-bzlib \
--with-lzma \
--with-libcap \
--with-ssl=openssl \
--with-ssldir=%{ssldir} \
--with-solr \
--with-docs
# It is compiled agains the latest openssl
# Tika and Solr: Both latest versions.
#Stacktrace:
doveadm(i...@samenmetrenske.nl): Info: Sent: Caching mails seq=1..161
doveadm(i...@samenmetrenske.nl): Panic: file http-client-request.c: line
1173 (http_client_request_send_more): assertion failed:
(req->payload_input != NULL)
doveadm(info@xxx.x): Error: Raw backtrace:
/usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x2f)
[0x7f95d805acbf] ->
/usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x26) [0x7f95d805add6]
-> /usr/lib64/dovecot/libdovecot.so.0(+0xe90ba) [0x7f95d80660ba] ->
/usr/lib64/dovecot/libdovecot.so.0(+0xe9161) [0x7f95d8066161] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x41158) [0x7f95d7fbe158] ->
/usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_more+0x424)
[0x7f95d8005094] ->
/usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0x11a)
[0x7f95d800a24a] ->
/usr/lib64/dovecot/libssl_iostream_openssl.so(+0x8f6a) [0x7f95d57a2f6a]
-> /usr/lib64/dovecot/libdovecot.so.0(+0x114483) [0x7f95d8091483] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x61)
[0x7f95d807e581] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xdc)
[0x7f95d808076c] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5c)
[0x7f95d807e67c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f95d807e8c8] -> /usr/lib64/dovecot/libdovecot.so.0(+0x89105)
[0x7f95d8006105] ->
/usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_payload+0x1f)
[0x7f95d80063cf] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xd31d)
[0x7f95d6ad931d] ->
/usr/lib64/dovecot/lib20_fts_plugin.so(fts_parser_more+0x1a)
[0x7f95d6ad83ca] ->
/usr/lib64/dovecot/lib20_fts_plugin.so(fts_build_mail+0x761)
[0x7f95d6ad6401] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x114ca)
[0x7f95d6add4ca] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_precache+0x2a)
[0x7f95d835ab4a] -> doveadm(+0x31e75) [0x55e7c1052e75] ->
doveadm(+0x321fb) [0x55e7c10531fb] -> doveadm(+0x2c321) [0x55e7c104d321]
-> doveadm(+0x2c577) [0x55e7c104d577] ->
doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x1e8) [0x55e7c104ec38] ->
doveadm(doveadm_cmd_run_ver2+0x52e) [0x55e7c105fafe] ->
doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55e7c105fb97] ->
doveadm(main+0x21a) [0x55e7c1062aca] ->
/lib64/libc.so.6(__libc_start_main+0x100) [0x7f95d7983d20] ->
doveadm(+0x1c479) [0x55e7c103d479]
signature.asc
Description: OpenPGP digital signature
Dovecot v2.3.10 Released
We are pleased to release v2.3.10 of Dovecot. Please find it from
locations below:
https://dovecot.org/releases/2.3/dovecot-2.3.10.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.10.tar.gz.sig
Binary packages in https://repo.dovecot.org/
Docker images in https://hub.docker.com/r/dovecot/dovecot
Aki Tuomi
Open-Xchange oy
---
* Disable retpoline migitations by default. These can cause severe
performance regressions, so they should be only enabled when
applicable.
* IMAP MOVE now commits transactions in batches of 1000 mails. This
helps especially with lazy_expunge when moving a lot of mails. It
mainly avoids situations where multiple IMAP sessions are running the
same MOVE command and duplicating the mails in the lazy_expunge folder.
With this change there can still be some duplication, but the MOVE
always progresses forward. Also if the MOVE fails at some point, the
changes up to the last 1000 mails are still committed instead of
rolled back. Note that the COPY command behavior hasn't changed,
because it is required by IMAP standard to be an atomic operation.
* IMAP EXPUNGE and CLOSE now expunges mails in batches of 1000 mails.
This helps especially with lazy_expunge when expunging a lot of mails
(e.g. millions) to make sure that the progress always moves forward
even if the process is killed.
* Autoexpunging now expunges mails in batches of 1000 mails. This helps
especially with lazy_expunge when expunging a lot of mails
(e.g. millions) to make sure that the progress always moves forward
even if the process is killed.
+ Add tool for generating sysreport called dovecot-sysreport.
This generates a bundle of information usually needed for support
requests.
+ Add support for the new IMAP \Important SPECIAL-USE flag (RFC 8457).
+ Add metric { group_by } setting. This allows automatically creating
new metrics based on the fields you want to group statistics by.
NOTE: This feature is considered experimental and syntax is subject
to change in future release.
+ auth: Support SCRAM-SHA-256 authentication mechanism.
+ imap: Support the new IMAP STATUS=SIZE extension.
+ Use TCP_QUICKACK to reduce latency for some TCP connections.
+ quota-status: Made the service more robust against erroneous use with
Postfix ACL policies other than smtpd_recipient_restrictions.
+ Add "revision" field support to imap_id_send setting. Using
"revision *" will send in IMAP ID command response the short commit
hash of the Dovecot git source tree HEAD (same as in dovecot --version).
+ IMAP ENVELOPE includes now all addresses when there are multiple
headers (From, To, Cc, etc.) The standard way of having multiple
addresses is to just list them all in a single header. It's
non-standard to have multiple headers. However, since MTAs allow these
mails to pass through and different software may handle them in
different ways, it's better from security point of view to show all
the addresses.
+ Event filters now support using "field_name=" to match a field that
doesn't exist or has an empty value. For example use "error=" to match
only events that didn't fail.
- acl: INBOX ACLs shouldn't apply for IMAP GETMETADATA/SETMETADATA
commands.
- cassandra: CASS_ERROR_SERVER_WRITE_FAILURE error should also be
treated as "uncertain write failure".
- dict-redis: Using quota_clone configured with dict-redis could have
crashed when Redis responded slowly.
- imap-hibernate: Communication trouble with imap-master leads to
segfault.
- imap-hibernate: Unhibernation retrying wasn't working.
- imap: Fixed auth lookup privilege problem when imap process was reused
and user was being un-hibernated.
- Fix potential crash when copying/moving mails within the same folder.
This happened only when there were a lot of fields in dovecot.index.cache.
- lib-index: Recreating dovecot.index.cache file could have crashed when
merging bitmask fields.
- lib-index: Using public/shared folders with INDEXPVT configured to use
private \Seen flags, trying to search seen/unseen in an empty folder
crashes with segfault.
- lib-mail: Large base64-encoded mails weren't decoded properly.
This could have affected searching/indexing mails and message snippet
generation.
- lib-mail: Message with only quoted text could have caused message
snippet to ignore its 200 character limit and return the entire
message. This was added also to dovecot.index.cache file, which
increased disk space and memory usage unnecessarily.
v2.3.9.2 regression (previous versions cached the quoted snippet as
empty). In a large mail quoted text could have become wrongly added
to the snippet, possibly mixed together with non-quoted text.
- lib-smtp: client could have assert-crashed if STARTTLS handshake
finished earlier than usually.
- lib-ssl-iostream: remove -static flag for lib-ssl-iostream linking to
prevent a compile issue.
- lib-storage: Mailbox synchronization may have assert-crashed in some
rare situations.
Pigeonhole v0.5.10 Released
We are doing a pigeonhole 0.5.10 release to maintain version number parity with dovecot. You can download it from locations below: https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.10.tar.gz https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.10.tar.gz.sig Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot There are no other changes than version number. --- Aki Tuomi Open-Xchange oy signature.asc Description: OpenPGP digital signature
Re: Dovecot - spam training through Outlook
On 05.03.2020 17:50, Jerry wrote: > Again, these are obsoleted versions of MS Outlook. The latest official > release is the 2019 version. I am actually using the beta version, but > I doubt that it would make much difference. I would highly recommend > updating to the latest version and then retrying the script. If it still > fails, then file a bug report. > > I actually read the posts in the link you supplied. The last post was > in regards to Outlook 2013, and it is nearly three years old. > Complaining about the the operation of an old version is like beating a > dead horse. > Did you verify that the issue is gone in the new version or are you just pitching Outlook to me? As I said we don't use it anymore. Last I saw this issue hasn't been resolved in any recent Outlook version. Also Outlook 2013 is still supported until 2023 so I don't know what you are complaining about people complaining. >Good luck. Thanks, same to you.
Re: Dovecot v2.3.10 Released
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 > On 06/03/2020 12:53 Aki Tuomi wrote: > > > We are pleased to release v2.3.10 of Dovecot. Please find it from > locations below: > > https://dovecot.org/releases/2.3/dovecot-2.3.10.tar.gz > https://dovecot.org/releases/2.3/dovecot-2.3.10.tar.gz.sig > Binary packages in https://repo.dovecot.org/ > Docker images in https://hub.docker.com/r/dovecot/dovecot > > Aki Tuomi > Open-Xchange oy > > --- > We forgot this from the NEWS: - fts-solr: The XML response parser fails to parse large/chunked responses correctly. This leads to spurious parse errors, most notably: "Error: fts_solr: received invalid uid '0'". Aki -BEGIN PGP SIGNATURE- Version: BCPG v1.62 iQEcBAABCgAGBQJeYmbVAAoJEBk7Y2OzREIGY8kH/0IaPC6r2X0WK8SgFE68q+Ti 8fyrXnJCNUguKO6xfU9oDfPswJ/ylzhIarMRR/A9OsC+o6C0uV9LQcRhXEz3qdVh F8afqiJB7pF6sq8G166ol8pGSPEBx21+r99SW33vwJdzFpHYQRod6iqomPZoaNCT y6Rrx+O/9PK849j7SiM+Fhs+f7mbmA2dpZow0A4bG9VQ8fgieI39raJwx3KMeHgh Jq7vfOvacD114dPv8vI+p9BafjmfO81oRoVsxz+hDTiL5TW7AvNy2Q+kBlI4W/mf 2eYUNS5Yjoxv3TfnzlkIRJ7P+dKBEArfDm3wE3JTZUKY7BQpMJmvvKIVf/1mIow= =rTq6 -END PGP SIGNATURE-
Updated Roadmap for Dovecot?
Hello, Just wanted to go take a peek at the latest Roadmap for dovecot, and note that the current wiki page is flagged as obsolete: https://wiki.dovecot.org/Roadmap Is there an updated version somewhere? I'm interested in the list of new features being worked on - especially JMAP support - and a rough idea of when said features might be expected to make it into a release. Thanks!
Re: Dovecot - spam training through Outlook
On Fri, 6 Mar 2020 14:23:09 +0100, Claudius stated: >On 05.03.2020 17:50, Jerry wrote: >> Again, these are obsoleted versions of MS Outlook. The latest >> official release is the 2019 version. I am actually using the beta >> version, but I doubt that it would make much difference. I would >> highly recommend updating to the latest version and then retrying >> the script. If it still fails, then file a bug report. >> >> I actually read the posts in the link you supplied. The last post was >> in regards to Outlook 2013, and it is nearly three years old. >> Complaining about the the operation of an old version is like >> beating a dead horse. >> >Did you verify that the issue is gone in the new version or are you >just pitching Outlook to me? Actually, I have the beta version of Outlook. However, since I don't use it in the same manner as you do, I obviously have not proof either way as 6to its behavior. Microsoft maintains several chat forums that offer advice. I have used several of them myself. Your question might best be asked, and hopefully answered there. >As I said we don't use it anymore. Last I saw this issue hasn't been >resolved in any recent Outlook version. > >Also Outlook 2013 is still supported until 2023 so I don't know what >you are complaining about people complaining. I am not complaining, just pointing out that it is by no means current. Microsoft, as well as many other software authors, rarely back-ports a non-security feature unless it is a real show-stopper. This obviously is not. >>Good luck. > >Thanks, same to you. -- Jerry pgpGUC95BLB_p.pgp Description: OpenPGP digital signature
