Package: offlineimap3
Version: 0.0~git20210105.00d395b+dfsg-2
Severity: important
Dear Maintainer,
I sent a message composed in mutt, afterwards offlineimap3 crashed
while trying to upload the sent message to IMAP server
with message "UnicodeDecodeError: 'utf-8' codec can't
decode byte 0xfc in position 317: invalid start byte".
The message was composed in mutt and contained German umlauts.
My system uses utf-8 encoding, however mutt has a send_charset
option that defaults to “us-ascii:iso-8859-1:utf-8” and thus the
message was encoded as "Content-Type: text/plain; charset=iso-8859-1".
Full error message (for real):
OfflineIMAP 7.3.0
Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.05, Python v3.9.1+, OpenSSL 1.1.1i 8 Dec 2020
Account sync foo:
*** Processing account foo
Establishing connection to mail.foo.com:993 (fooRemote)
Folder INBOX [acc: foo]:
Syncing INBOX: IMAP -> Maildir
Copy message UID -2 (1/2) fooLocal:INBOX -> fooRemote:INBOX
ERROR: Copying message -2 [acc: foo]
'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
ERROR: while syncing INBOX [account foo]
'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
ERROR: ERROR in syncfolder for foo folder INBOX: Traceback (most recent call
last):
File "/usr/share/offlineimap3/offlineimap/accounts.py", line 666, in
syncfolder
localfolder.syncmessagesto(remotefolder, statusfolder)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 1186, in
syncmessagesto
action(dstfolder, statusfolder)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 1013, in
__syncmessagesto_copy
self.copymessageto(uid, dstfolder, statusfolder, register=0)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 902, in
copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 262, in
getmessage
retval = file.read()
File "/usr/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 317:
invalid start byte
'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
Account sync foo:
*** Finished account 'foo' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: Copying message -2 [acc: foo]
'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
Traceback:
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 902, in
copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 262, in
getmessage
retval = file.read()
File "/usr/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
ERROR: while syncing INBOX [account foo]
'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
Traceback:
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 1186, in
syncmessagesto
action(dstfolder, statusfolder)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 1013, in
__syncmessagesto_copy
self.copymessageto(uid, dstfolder, statusfolder, register=0)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 902, in
copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 262, in
getmessage
retval = file.read()
File "/usr/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
ERROR: ERROR in syncfolder for foo folder INBOX: Traceback (most recent call
last):
File "/usr/share/offlineimap3/offlineimap/accounts.py", line 666, in
syncfolder
localfolder.syncmessagesto(remotefolder, statusfolder)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 1186, in
syncmessagesto
action(dstfolder, statusfolder)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 1013, in
__syncmessagesto_copy
self.copymessageto(uid, dstfolder, statusfolder, register=0)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 902, in
copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 262, in
getmessage
retval = file.read()
File "/usr/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 317:
invalid start byte
'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
After debugging a bit I came up with this hotfix to get me out of
the situation without deleting the offending email:
--- offlineimap/folder/Maildir.py.orig 2021-01-05 09:55:15.000000000 +0100
+++ offlineimap/folder/Maildir.py 2021-01-31 20:25:39.570660781 +0100
@@ -23,6 +23,7 @@
from sys import exc_info
from threading import Lock
from hashlib import md5
+from email.parser import BytesParser
from offlineimap import OfflineImapError, emailutil
from .Base import BaseFolder
@@ -258,7 +259,8 @@
filename = self.messagelist[uid]['filename']
filepath = os.path.join(self.getfullname(), filename)
- file = open(filepath, 'rt')
+ m = BytesParser().parse(open(filepath, 'rb'), headersonly=True)
+ file = open(filepath, 'rt', encoding=m.get_content_charset('utf-8'))
retval = file.read()
file.close()
# TODO: WHY are we replacing \r\n with \n here? And why do we
This is likely not the best way to fix it, please consult with upstream.
Best Regards,
Johannes
-- System Information:
Debian Release: bullseye/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 5.8.18 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=de_DE.utf-8 (charmap=UTF-8),
LANGUAGE=en_US:en
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)
Versions of packages offlineimap3 depends on:
ii python3 3.9.1-1
ii python3-distro 1.5.0-1
ii python3-imaplib2 2.57-5.2
offlineimap3 recommends no packages.
offlineimap3 suggests no packages.
-- no debconf information
