Re: [Dovecot] Master user and missing folders
Anders <[EMAIL PROTECTED]> writes: > It logs the correct value for "home" with mail_debug=yes. It is also > able to print the correct quota values, so Dovecot seems to end up > at the right place. It was actually auth_debug that I turned on, not mail_debug. Here is the mail_debug log. It led me to suspect the acl plugin and, indeed, if I turn that one off, I can list the folders even with a master user login. Cheers, Anders. IMAP(am): Loading modules from directory: /usr/local/dovecot-1.1rc4/lib/dovecot/imap IMAP(am): Module loaded: /usr/local/dovecot-1.1rc4/lib/dovecot/imap/lib01_acl_plugin.so IMAP(am): Module loaded: /usr/local/dovecot-1.1rc4/lib/dovecot/imap/lib10_quota_plugin.so IMAP(am): Module loaded: /usr/local/dovecot-1.1rc4/lib/dovecot/imap/lib11_imap_quota_plugin.so IMAP(am): Module loaded: /usr/local/dovecot-1.1rc4/lib/dovecot/imap/lib20_expire_plugin.so IMAP(am): Module loaded: /usr/local/dovecot-1.1rc4/lib/dovecot/imap/lib20_mail_log_plugin.so IMAP(am): Effective uid=5000, gid=5000, home=/var/mail/vhosts/example.invalid/am IMAP(am): Quota root: name= backend=maildir args= IMAP(am): Quota rule: root= mailbox= bytes=104857600 (0%) messages=0 (0%) IMAP(am): Quota warning: bytes=99614720 (95%) messages=0 (0%) command=/usr/local/bin/quota-warning 95 IMAP(am): Quota warning: bytes=83886080 (80%) messages=0 (0%) command=/usr/local/bin/quota-warning 80 IMAP(am): maildir: data=/var/mail/vhosts/example.invalid/am/Maildir IMAP(am): maildir++: root=/var/mail/vhosts/example.invalid/am/Maildir, index=, control=, inbox=/var/mail/vhosts/example.invalid/am/Maildir IMAP(am): acl: initializing backend with data: vfile:/etc/dovecot-acls:cache_secs=300 IMAP(am): acl: acl username = master IMAP(am): acl: owner = 0 IMAP(am): acl vfile: Global ACL directory: /etc/dovecot-acls imap-login: Login: user=, method=LOGIN, rip=127.0.0.1, lip=127.0.0.1, secured IMAP(am): acl vfile: file /etc/dovecot-acls//.DEFAULT not found IMAP(am): acl vfile: file /var/mail/vhosts/example.invalid/am/Maildir/dovecot-acl not found IMAP(am): Disconnected: Logged out bytes=78/379
Re: [Dovecot] Delete antispam emails
Johannes Berg wrote: >> This is what I found in syslog... >> >> Apr 12 21:28:15 kosh imap: antispam: plugin initialising >> Apr 12 21:28:15 kosh imap: antispam: "trash" is trash folder >> Apr 12 21:28:15 kosh imap: antispam: "Trash" is trash folder >> Apr 12 21:28:15 kosh imap: antispam: "Deleted Items" is trash folder >> Apr 12 21:28:15 kosh imap: antispam: "SpamTrain" is spam folder >> Apr 12 21:28:15 kosh imap: antispam: "INBOX.SpamTrain" is spam folder >> ... >> >> It looks like all the folders are being recognised correctly. > > Good, yes, that confirms at least the configuration is working. > >> Is it >> possible to get more info when messages are being processed? > > Not as-is, but you can try something like the patch below: Apologies for taking so long to reply, but have been travelling... Anyways, this is what I see with your patch (date/time stamp truncated to allow the details to be on a single line)... ... map: antispam: mail copy: from trash: 0, to trash: 0 ... map: antispam: mail copy: src spam: 1, dst spam: 0, src unsure: 0 So, it looks as though my trash detection is broken. I guess I need to look at the mailbox_is_trash() function next? Can you suggest a patch to debug that? I'll try and reply quicker as have stopped travelling for the moment! david
Re: [Dovecot] Delete antispam emails
> Anyways, this is what I see with your patch (date/time stamp truncated
> to allow the details to be on a single line)...
>
> ... map: antispam: mail copy: from trash: 0, to trash: 0
> ... map: antispam: mail copy: src spam: 1, dst spam: 0, src unsure: 0
>
> So, it looks as though my trash detection is broken. I guess I need to
> look at the mailbox_is_trash() function next? Can you suggest a patch to
> debug that?
Hmm. Odd. Let's try this. Are you using namespaces?
johannes
diff --git a/antispam-plugin.c b/antispam-plugin.c
index c83d225..1f33820 100644
--- a/antispam-plugin.c
+++ b/antispam-plugin.c
@@ -60,7 +60,10 @@ static bool mailbox_in_list(struct mailbox *box, char **list)
if (!list)
return FALSE;
+ debug("mailbox_in_list");
+
while (*list) {
+ debug(" - list item: %s, box: %s\n", *list,
mailbox_get_name(box));
if (mailbox_equals(box, box->storage, *list))
return TRUE;
list++;
@@ -76,6 +79,7 @@ bool mailbox_is_spam(struct mailbox *box)
bool mailbox_is_trash(struct mailbox *box)
{
+ debug("checking trash %s\n", mailbox_get_name(box));
return mailbox_in_list(box, trash_folders);
}
Re: [Dovecot] Delete antispam emails
Johannes Berg wrote: >> Anyways, this is what I see with your patch (date/time stamp truncated >> to allow the details to be on a single line)... >> >> ... map: antispam: mail copy: from trash: 0, to trash: 0 >> ... map: antispam: mail copy: src spam: 1, dst spam: 0, src unsure: 0 >> >> So, it looks as though my trash detection is broken. I guess I need to >> look at the mailbox_is_trash() function next? Can you suggest a patch to >> debug that? > > Hmm. Odd. Let's try this. Are you using namespaces? I needed to add an additional entry to the list of trash folders, INBOX.Trash, and then all was fine. IMAP folder names and hierachy do confuse me a bit at times. Thanks for your help and apologies for the noise. BTW, your debugging patch was really helpful, so maybe it could be added with a conditional? If I code it up, would you be interested? david
Re: [Dovecot] Delete antispam emails
On Sat, 2008-04-19 at 12:17 +0100, David Reid wrote: > Johannes Berg wrote: > >> Anyways, this is what I see with your patch (date/time stamp truncated > >> to allow the details to be on a single line)... > >> > >> ... map: antispam: mail copy: from trash: 0, to trash: 0 > >> ... map: antispam: mail copy: src spam: 1, dst spam: 0, src unsure: 0 > >> > >> So, it looks as though my trash detection is broken. I guess I need to > >> look at the mailbox_is_trash() function next? Can you suggest a patch to > >> debug that? > > > > Hmm. Odd. Let's try this. Are you using namespaces? > > I needed to add an additional entry to the list of trash folders, > INBOX.Trash, and then all was fine. IMAP folder names and hierachy do > confuse me a bit at times. Thanks for your help and apologies for the noise. Ah :) I'm not entirely sure about INBOX. vs. just plain names myself. > BTW, your debugging patch was really helpful, so maybe it could be added > with a conditional? If I code it up, would you be interested? I could add a debug_verbose() statement that you need to enable in the .config, what do you think? johannes signature.asc Description: This is a digitally signed message part
[Dovecot] building a simple antispam plugin
Hello, I'm trying to build another antispam plugin, but some things are a bit unclear. In backend_handle_mail I put store the signature returned from signature_extract into my antispam_transaction_context. In backend_commit it appears that the signature is empty. (It wasn't in backend_handle_mail). Somehow between backend_handle_mail en backend_commit it has been cleared... When I use i_strdup it appears to work. Anyone can tell me where this clearing happens? Is i_strdup the right way to do it then? Thanks, Alexander
[Dovecot] Getting dovecot to recognize existing folders, was Re: Okay, so what's with "mail storage creation failed", was Cannot connect to dovecot imap server -- nothing in log
On Fri, 2008-04-18 at 00:27 -0400, Bill Cole wrote:
> At 8:36 PM -0700 4/17/08, David Benfell wrote:
>
> (reduced to the key pieces of information.)
>
> >dovecot: 2008-04-17 20:24:51 Error: IMAP(benfell): Ambiguous mail
> >location setting, don't know what to do with it:
> >/home/benfell/imap-folders (try prefixing it with mbox: or maildir:)
>
> [...]
>
> >namespace:
> > type: private
> > separator: /
> > prefix: folders/
> > location: ~/imap-folders
>
> Apply the clue in the log line to the namespace 'location' parameter.
>
> http://wiki.dovecot.org/Namespaces?highlight=%28location%29
>
>
Once again, thanks. I was trying to understand the configuration file
and I guess that just didn't work. But I still seemed to be missing
something.
I wanted to get it to recognize my existing folders. I created a test
folder and saw that in the actual file space, dovecot uses periods to
separate layers of the hierarchy, and that the folder name at the top of
the hierarchy also begins with a period.
If this is the case, I'm not sure what defining "/" as a separator is
for.
So, though I'm not much of script writer, I created a script to go
through and set symbolic links to those folders (Maildirs):
#!/usr/bin/zsh
cd ${HOME}/imap
find . -type d | grep -v "\/\." \
| grep -v "\/tmp$" \
| grep -v "\/cur$" \
| grep -v "\/new$" | while read folder
do
target=$(print "${folder}"|sed 's_/_._'g|sed 's_\.\._._'g)
if [[ ! -e "${target}" ]]
then
ln -s "${folder}" "${target}"
ls -al "${target}"
fi
done
It still didn't appear to work. There are thousands of these folders in
a nested hierarchy, because I rotate mail folders by year, month,
and--in some cases--day. Thus, not only do I need to get these folders
recognized initially, but I have an ongoing automatic process that
generates new folders daily, that I want dovecot to also recognize.
So about the time I was just about to fire this off, evolution greyed
out. When it came back, I could see that it was going through my
folders. I sure hope it doesn't have to go through all *this* every
time, because it is taking a while.
--
David Benfell <[EMAIL PROTECTED]>
signature.asc
Description: This is a digitally signed message part
Re: [Dovecot] building a simple antispam plugin
Hi, > I'm trying to build another antispam plugin, but some things are a bit > unclear. What are you trying to achieve? Just build another backend for my antispam plugin? > In backend_handle_mail I put store the signature returned from > signature_extract into my antispam_transaction_context. In > backend_commit it appears that the signature is empty. (It wasn't in > backend_handle_mail). > > Somehow between backend_handle_mail en backend_commit it has been > cleared... When I use i_strdup it appears to work. Anyone can tell me > where this clearing happens? Is i_strdup the right way to do it then? Well, how did you put it in there? It's well possible that the memory was allocated on a stack or in a pool that has been cleared/freed... johannes signature.asc Description: This is a digitally signed message part
Re: [Dovecot] building a simple antispam plugin
Johannes Berg wrote:
> Hi,
>
>> I'm trying to build another antispam plugin, but some things are a bit
>> unclear.
>
> What are you trying to achieve? Just build another backend for my
> antispam plugin?
Yes, just another backend. It would be logging the signatures to a mysql
database, so that I can process the signatures with dspam at a
convenient time.
I'll use the sql functions available in the sql-api.h from dovecot. I'll
probably store a global sql_db, so I don't need to open/close
connections for every message.
>
>> In backend_handle_mail I put store the signature returned from
>> signature_extract into my antispam_transaction_context. In
>> backend_commit it appears that the signature is empty. (It wasn't in
>> backend_handle_mail).
>>
>> Somehow between backend_handle_mail en backend_commit it has been
>> cleared... When I use i_strdup it appears to work. Anyone can tell me
>> where this clearing happens? Is i_strdup the right way to do it then?
>
> Well, how did you put it in there? It's well possible that the memory
> was allocated on a stack or in a pool that has been cleared/freed...
My antispam_transaction_context contains a const char* signature.
in backend_handle_mail:
ast->signature = signature_extract(t, mail);
Then in backend_commit:
debug("backend_commit %p SIG: %s CLASS: %u", ast, ast->signature ?
ast->signature : "/", ast->classification);
signature appears to be an empty string there. When using i_strdup in
backend_handle_mail it seems to work.
Thanks,
Alexander
Re: [Dovecot] building a simple antispam plugin
On Sat, 2008-04-19 at 15:12 +0200, Alexander Prinsier wrote:
> Johannes Berg wrote:
> > Hi,
> >
> >> I'm trying to build another antispam plugin, but some things are a bit
> >> unclear.
> >
> > What are you trying to achieve? Just build another backend for my
> > antispam plugin?
> Yes, just another backend. It would be logging the signatures to a mysql
> database, so that I can process the signatures with dspam at a
> convenient time.
>
> I'll use the sql functions available in the sql-api.h from dovecot. I'll
> probably store a global sql_db, so I don't need to open/close
> connections for every message.
> >
> >> In backend_handle_mail I put store the signature returned from
> >> signature_extract into my antispam_transaction_context. In
> >> backend_commit it appears that the signature is empty. (It wasn't in
> >> backend_handle_mail).
> >>
> >> Somehow between backend_handle_mail en backend_commit it has been
> >> cleared... When I use i_strdup it appears to work. Anyone can tell me
> >> where this clearing happens? Is i_strdup the right way to do it then?
> >
> > Well, how did you put it in there? It's well possible that the memory
> > was allocated on a stack or in a pool that has been cleared/freed...
> My antispam_transaction_context contains a const char* signature.
>
> in backend_handle_mail:
> ast->signature = signature_extract(t, mail);
>
> Then in backend_commit:
> debug("backend_commit %p SIG: %s CLASS: %u", ast, ast->signature ?
> ast->signature : "/", ast->classification);
>
> signature appears to be an empty string there. When using i_strdup in
> backend_handle_mail it seems to work.
You have to i_strdup() it and also free it later, the signature code
just gives you a pointer that is valid as long as the mail struct lives
(IIRC)
johannes
signature.asc
Description: This is a digitally signed message part
[Dovecot] mailtrain with dovecot antispam plugin
Hello all, and Johannes if you're around, I was trying to set up the "new" antispam plugin with the mailtrain backend, and i was trying to use the sendmail binary that's part of a qmail system. The message wouldn't get sent and i kept getting exit error 100 from the /var/qmail/bin/sendmail binary. To try to figure out what was happening, i mean a fake sendmail program, which is nothing more than a bash script to capture the command call input. The script is rather simple, it just places the calling parameters into a txt file and the stdin into another. code is #!/bin/bash echo $* > /tmp/sendmail-parms.txt exec cat<&0 >> /tmp/sendmail-msg.txt exit 0 So i tried a retrain and went to see what those txt files had. The parms.txt file was fine apparently, with cat /tmp/sendmail-parms.txt i got -f [EMAIL PROTECTED] [EMAIL PROTECTED] Then i did the same to the message file, with cat /tmp/sendmail-msg.txt, and i got the follwing Return-Path: <[EMAIL PROTECTED]>Delivered-To: [EMAIL PROTECTED] Received: (qmail 21616 invoked from network); 19 Apr 2008 16:21:17 - Received: (simscan 1.4.1 ppid 21609 pid 21611 t 0.1451s) (scanners: regex: 1.4.1 attach: 1.4.1 clamav: 0.92.1 /m:46/d:6803); 19 Apr 0108 16:21:17 - Received: from femme-fatal.resnet.costabasto.com (HELO femme-fatal) (192.168.0.20) by hmonteiro.net with SMTP; 19 Apr 2008 16:21:17 - Subject: buy some viagra HURRAY X-DSPAM-Result: Innocent X-DSPAM-Processed: Sat Apr 19 17:21:17 2008 X-DSPAM-Confidence: 0.4945 X-DSPAM-Probability: 0.0330 X-DSPAM-Signature: 480a1bfd216125429498786 viagra sale really HURRAY cheap! only $5! As you can see, there is no new line in the first line. Return-Path and Delivered-To are set in a single line, which makes qmail-inject (which is later used by /var/qmail/bin/sendmail) to abort like this cat /tmp/sendmail-msg.txt | /var/qmail/bin/sendmail -f [EMAIL PROTECTED] [EMAIL PROTECTED] qmail-inject: fatal: unable to parse this line: Return-Path: <[EMAIL PROTECTED]>Delivered-To: [EMAIL PROTECTED] Before i go into digging the plugin code i was wondering if i'm not seing things clearly and thinking this is in fact a bug in the plugin code. Thank you all in advance, Hugo Monteiro. -- ci.fct.unl.pt:~# cat .signature Hugo Monteiro Email: [EMAIL PROTECTED] Telefone : +351 212948300 Ext.15307 Centro de Informática Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa Quinta da Torre 2829-516 Caparica Portugal Telefone: +351 212948596 Fax: +351 212948548 www.ci.fct.unl.pt [EMAIL PROTECTED] ci.fct.unl.pt:~# _ signature.asc Description: OpenPGP digital signature
Re: [Dovecot] How to deal with mail to unknown virtual recipients?
Andreas Ntaflos wrote:
Thank you for your reply! Reading through it and re-reading the Postfix
documentation gave me the clue I needed.
Unfortunately I forgot to mention that I didn't have a virtual_mailbox_maps
directive defined in Postfix's main.cf, so Postfix couldn't know anything
about which virtual recipients were valid and which weren't. Since that
particular map was optional and everything worked fine during my tests (which
didn't include sending mail to an invalid virtual recipient, shame on me) I
quickly forgot about it.
Naturally that was the root of problem I was facing.
Defining a virtual_mailbox_maps file with all valid virtual users solves the
problem in principle, however now I need to maintain two files with valid
virtual users for valid virtual domains: the passwd-file for Dovecot
(/etc/dovecot/passwd) plus the vmailbox file for Postfix
(hash:/etc/postfix/vmailbox).
use a script to generate the virtual mailbox maps files from a single
source file. you can use a Makefile to rebuild only when the source file
changed.
I suppose this is unavoidable when using flat files and would be better solved
using a real database backend, but it makes me wonder if there really isn't
anything Dovecot could do about this?
Seeing as Dovecot recognises when a virtual recipient is invalid, couldn't it
somehow put the to-be-rejected mail back to Postfix (or whichever MTA)?
as already said, you should reject invalid recipient during the smtp
transaction. there's already too much backscatter out there, and the
borked sites are listed in backscatterer.org.
Or
should it just drop it?
No, because people sometimes mistype addresses and you don't want to
blackhole such mail. if I send mail to [EMAIL PROTECTED], and you discard
that mail, I'll assume that you received it. now suppose this mail is
important ("Hi Andreas, there is a problem at foo bar, tell me if I
should handle it...").
I don't know what the SMTP (whichevery protocol is to
be followed here) says about this, so I pose the question again:
if you accept the message during the smtp transaction, you take
responsibility and you should not lose it "frivoulously". This
requirement is what makes email as reliable as it is, at the expense of
disk IO overhead (before saying "ok", the MTA must sync the file to
permanent storage, so that it survives server crashes... etc).
of course, servers should never bounce spam and viruses. as a result, it
is important to reject as much junk during the smtp transaction and to
minimize failures after a message was accepted. In particular, it is no
more acceptable to bounce mail because the recipient is invalid. quota
bounces and exceptional server errors are still acceptable because they
don't occur too often.
Note that this also applies to auto-responders (vacation, ...).
auto-responders must follow the recommendations of rfc3834 as well as
other best practices.
Is there anything Dovecot itself can do to reject or handle mail to invalid
virtual users? Or does this situation *have* to be dealt with earlier, by the
MTA?
yes. see www.backscatterer.org for an incentive ;-p
Re: [Dovecot] mailtrain with dovecot antispam plugin
Hugo Monteiro wrote:
Hello all, and Johannes if you're around,
I was trying to set up the "new" antispam plugin with the mailtrain
backend, and i was trying to use the sendmail binary that's part of a
qmail system. The message wouldn't get sent and i kept getting exit
error 100 from the /var/qmail/bin/sendmail binary. To try to figure
out what was happening, i mean a fake sendmail program, which is
nothing more than a bash script to capture the command call input.
The script is rather simple, it just places the calling parameters
into a txt file and the stdin into another.
code is
#!/bin/bash
echo $* > /tmp/sendmail-parms.txt
exec cat<&0 >> /tmp/sendmail-msg.txt
exit 0
So i tried a retrain and went to see what those txt files had. The
parms.txt file was fine apparently, with cat /tmp/sendmail-parms.txt i
got
-f [EMAIL PROTECTED] [EMAIL PROTECTED]
Then i did the same to the message file, with cat
/tmp/sendmail-msg.txt, and i got the follwing
Return-Path: <[EMAIL PROTECTED]>Delivered-To: [EMAIL PROTECTED]
Received: (qmail 21616 invoked from network); 19 Apr 2008 16:21:17 -
Received: (simscan 1.4.1 ppid 21609 pid 21611 t 0.1451s)
(scanners: regex: 1.4.1 attach: 1.4.1 clamav: 0.92.1
/m:46/d:6803); 19 Apr 0108 16:21:17 -
Received: from femme-fatal.resnet.costabasto.com (HELO femme-fatal)
(192.168.0.20)
by hmonteiro.net with SMTP; 19 Apr 2008 16:21:17 -
Subject: buy some viagra HURRAY
X-DSPAM-Result: Innocent
X-DSPAM-Processed: Sat Apr 19 17:21:17 2008
X-DSPAM-Confidence: 0.4945
X-DSPAM-Probability: 0.0330
X-DSPAM-Signature: 480a1bfd216125429498786
viagra sale really HURRAY cheap! only $5!
As you can see, there is no new line in the first line. Return-Path
and Delivered-To are set in a single line, which makes qmail-inject
(which is later used by /var/qmail/bin/sendmail) to abort like this
cat /tmp/sendmail-msg.txt | /var/qmail/bin/sendmail -f
[EMAIL PROTECTED] [EMAIL PROTECTED]
qmail-inject: fatal: unable to parse this line:
Return-Path: <[EMAIL PROTECTED]>Delivered-To: [EMAIL PROTECTED]
Before i go into digging the plugin code i was wondering if i'm not
seing things clearly and thinking this is in fact a bug in the plugin
code.
Thank you all in advance,
Hugo Monteiro.
I've managed to get it working with the following patch.
There may be a better way, deeper in the code, to do it. Timo or
Johannes ... any sugestions?
--- dovecot-antispam/mailtrain.c2008-04-19 22:20:32.0 +0100
+++ dovecot-antispam-qmail/mailtrain.c 2008-04-19 22:25:45.0 +0100
@@ -274,7 +274,8 @@
mail_storage_set_error(t->box->storage,
"Failed to write line to
temp");
goto failed_to_copy;
- }
+ } else
+ o_stream_send_str(outstream, "\n");
if (o_stream_send_istream(outstream, mailstream) < 0) {
ret = -1;
Cheers,
Hugo Monteiro.
--
ci.fct.unl.pt:~# cat .signature
Hugo Monteiro
Email: [EMAIL PROTECTED]
Telefone : +351 212948300 Ext.15307
Centro de Informática
Faculdade de Ciências e Tecnologia da
Universidade Nova de Lisboa
Quinta da Torre 2829-516 Caparica Portugal
Telefone: +351 212948596 Fax: +351 212948548
www.ci.fct.unl.pt [EMAIL PROTECTED]
ci.fct.unl.pt:~# _
signature.asc
Description: OpenPGP digital signature
[Dovecot] using mysql from within a plugin
Hello, I'm trying to access a mysql database in a plugin. -What special compiler flags do I need? I discovered that when I don't explicitly link lib-sql/libsql.a in, then I get an undefind symbol error about sql_init when loading the plugin. Shouldn't the imap process itself have sql support linked in, rather than each plugin? -It appears that I need to call sql_drivers_init() and sql_drivers_register_all() before sql_init. Again, shouldn't the imap process do this, instead of each plugin? It should probably only be called once, so if one plugin already does it, then the second one will create problems. -When linking libsql.a in, and calling the functions mentioned above, then I get this error when starting dovecot: undefined symbol: mysql_ssl_set I'm guessing it's part of the mysql library itself, not dovecot. Can someone give me some hints on how to use dovecot's mysql library in a plugin? Thanks, Alexander
