[Dovecot] Is this really a user agent issue?

[Phil Howard]

I suspect this is a user agent issue, given that Evolution is flaky in
so many areas.  When I deliver mail to a subfolder/subbox (e.g. the -m
option in the deliver command), and Evolution doesn't know of it, yet,
creating it fails, and Evolution still can't get to it.  Doing things
the other way around (create it in Evolution first, then deliver to
it) works fine.  Seems to be silliness to me.  Just wondered if any
Dovecot aspects might be involved ... such as the fact that the case
is different (e.g. "INBOX" in the filesystem, but "Inbox" in
Evolution).

-- 
sHiFt HaPpEnS!

[Dovecot] Moving email between folders outside of IMAP context

[Phil Howard]

I'd like to run a script from cron that will scan certain mailboxes
and move selected mail from one folder (usually INBOX) to another
folder.  The basic idea is to move previously detected spam based on
headers prepended to the mail.  Since I am not able to figure out how
to get Postfix to invoke deliver with the -m option in these cases,
and not in others, this is one idea I have come up with to deal with
it.  The big (hopefully obvious) complication seems to be getting the
indexes updated.  Is there a way to move mail this way AND then have
the indexes updated to reflect that move (maybe even if regenerating
the entire index all over again)?

-- 
sHiFt HaPpEnS!

Re: [Dovecot] Moving email between folders outside of IMAP context

[Phil Howard]

On Tue, Dec 21, 2010 at 16:41, Tom Hendrikx  wrote:

> Do not use an external script, but add a global sieve script that sorts
> messages into some directory, based on your preferred header.

How does that get executed.  I take it I can't do it via cron.  Does
it run at deliver time for each mail message?  Is it doable in version
1.1.11 (that's what's in my version of Ubuntu)?  Do you know which
Ubuntu 9.10 package to install (we're stuck on 9.10 until the new
server arrives around February).

-- 
sHiFt HaPpEnS!

Re: [Dovecot] Moving email between folders outside of IMAP context

[Phil Howard]

On Tue, Dec 21, 2010 at 17:09, Tom Hendrikx  wrote:

> I am not familiar with Ubuntu specifics, but iirc Stephan Bosch (creator
> of sieve plugin) has a repo with debian builds of dovecot+sieve.

I would need something that is sieve only, as Dovecot is already
installed, configured, and running.

I'm also considering writing a program to be run by Postfix in place
of "deliver", which will read the mail to determine how to run
"deliver".  As long as mail comes in on stdin and is written on
stdout, it should be relatively easy (pipe(), fork(), maybe setsid(),
execve() in child, read() and write() in parent).  But this would take
some thorough testing to be sure it is safe.

Re: [Dovecot] Moving email between folders outside of IMAP context

[Phil Howard]

On Tue, Dec 21, 2010 at 17:41, Tom Hendrikx  wrote:

> That software is already written and tested, it is the sieve plugin. If
> you're willing to write, test and run a complete new piece of software,
> you might as well replace dovecot with a sieve enabled version of the
> same package you're running now, and putting your current configuration
> in place.

I would not be willing to take the risk of uninstalling and
re-installing Dovecot or Postfix.


> Anyhow, maybe sieve is already available in your current package. I'm
> not familiar with (contents of) ubuntu packages, so maybe someone more
> knowledgeable in this area can step up?

Ah ... apparently so.  These files exist, so I'll go do reading on CMU sieve.

meitner/root/x0 /root 299# ( dpkg -l | fgrep dovecot | awk '{print
$2;}' | while read n ; do dpkg -L ${n} | fgrep -i sieve | prefix
"package ${n} has " ; done )
package dovecot-common has /usr/share/doc/dovecot-common/wiki/LDA.Sieve.txt.gz
package dovecot-common has /usr/share/doc/dovecot-common/wiki/ManageSieve.txt.gz
package dovecot-common has /usr/share/doc/dovecot-common/managesieve.README.gz
package dovecot-common has /usr/lib/dovecot/modules/lda/lib90_cmusieve_plugin.so
package dovecot-common has /usr/lib/dovecot/modules/lda/lib90_cmusieve_plugin.la
package dovecot-common has /usr/lib/dovecot/modules/lda/lib90_cmusieve_plugin.a
package dovecot-common has /usr/lib/dovecot/sievec
package dovecot-common has /usr/lib/dovecot/sieved
package dovecot-common has /usr/lib/dovecot/managesieve
package dovecot-common has /usr/lib/dovecot/managesieve-login
meitner/root/x0 /root 300#

-- 
sHiFt HaPpEnS!

Re: [Dovecot] Is this really a user agent issue?

[Phil Howard]

2010/12/21 Karsten Bräckelmann :

> Creating the new mail folder is entirely on the IMAP server side. The
> MUA (Evolution in your stated case) is irrelevant. If the creation of
> the new folder fails, it is a server side problem.
>
> However, once a new folder has been created (server side, mind you),
> Evolution won't know about that folder until it is restarted. Probably
> the same with other MUAs, too. In the worst case, restarting Evo twice
> should show the new folder.
>
> (That is assuming you are not limiting your MUA to subscribed folders
> only, or, as IIRC is the default, deliver auto-subscribes the user to
> the just created folder.)

Then I would call this a user agent issue.  If IMAP provides a way for
the user agent to discover the folder already exists, then the user
agent should do this, at least if an error is encountered when trying
to create it.  Evolution clearly does not (or doesn't act on knowing
if it does).  Other user agents, I don't know.  But it is simple
logic:

if (create folder $name succeeds) OR (folder $name can be accessed) then
set up local reference to folder $name
else
report error about creating folder $name
endif

That, of course, depends on IMAP having a means to detect if the
folder exists (even though the folder was not given in a previous list
of existing folders).  Trying to open it should be a way to do that
test.

But if the IMAPD process itself can't or won't even try to access that
folder because it wasn't in the list when it started, then I see that
as a server side issue, or if the protocol doesn't provide any such
means to ask for a folder not previously seen, then I see that as a
protocol design issue.

-- 
sHiFt HaPpEnS!

[Dovecot] trying to make cmusieve happen globally

[Phil Howard]

In Dovecot 1.1.11 cmusieve is apparently integrated in the Ubuntu 9.10
package "dovecot-common" since the files are there.  I am wanting to
right now just do a very basic test of the setup to see how it runs
before doing more sophisticated steps that could obscure any errors or
bugs (e.g. unit testing).

So I took the following example sieve script from the wiki1 documentation:

require "fileinto";
if header :contains "X-Spam-Flag" "YES" {
  fileinto "spam";
}

and recoded it to skip the test and just always do the fileinto step like so:

require "fileinto";
fileinto "spamtest";

Then I configured my dovecot.conf file (via dovecot-postfix.conf it
really points to) by setting:

protocol lda {
...
mail_plugins = cmusieve
mail_plugin_dir = /usr/lib/dovecot/modules/lda
...
}

and:

plugin {
sieve_global_path = /etc/dovecot/sieve-global.d/spam.sieve
}

I manually compiled /etc/dovecot/sieve-global.d/spam.sieve so there
also exists /etc/dovecot/sieve-global.d/spam.sievec in the same
directory.  Then I restarted both Dovecot and Postfix.

Mail is being delivered, but still only to the same place it always
has.  The filesystem shows these sieve files are not being accessed at
all.  I've read documents http://wiki1.dovecot.org/LDA/Sieve and
http://wiki1.dovecot.org/LDA/Sieve/CMU to come up with this.  But it
just doesn't seem to be happening.  So I guess it is a good thing I've
broken my testing down into smaller units.  Anything standout that I
misinterpreted about the documentation?  If not, I can post the whole
config files (in case there is a conflict with something else I have
done).

FYI, I'm only trying to do a global sieve, not per-user sieve scripts.
 I'll tackle per-user stuff at a later date, after the global is
working.

-- 
sHiFt HaPpEnS!

Re: [Dovecot] trying to make cmusieve happen globally

[Phil Howard]

On Wed, Dec 22, 2010 at 11:46, Anton Dollmaier
 wrote:

>> plugin {
>>         sieve_global_path = /etc/dovecot/sieve-global.d/spam.sieve
>
> Use "sieve_before", which is not documented in the wiki.

Changed.


>
> Only this way the sievescript really gets executed.
>
>
> The global path requires a sieve script per home directory, including the
> global file.

I changed "sieve_global_path" to "sieve_before", restarted both
Dovecot and Postfix (though if "deliver" reads the config file each
time, I guess I should not need to restart for this stuff), but
there's still no effect.  Mail is delivered, but still just to the
regular inbox.

Do I need to have per-user sieves set up, too, just to get it to work?
 Are there any log messages that I should expect to see when it is
working?


-- 
sHiFt HaPpEnS!

Re: [Dovecot] trying to make cmusieve happen globally

[Phil Howard]

On Wed, Dec 22, 2010 at 13:12, Stephan Bosch  wrote:
> Op 22-12-2010 17:46, Anton Dollmaier schreef:

>>> plugin {
>>>         sieve_global_path = /etc/dovecot/sieve-global.d/spam.sieve
>>
>> Use "sieve_before", which is not documented in the wiki.
>
> Eh.. no. CMUSieve does not have multiscript support, so sieve_before is not
> available.
>
>> Only this way the sievescript really gets executed.
>
> The global default is not executed when the user has a script of his own,
> otherwise it is executed.

That sounds like a reasonable way to do it.


>> The global path requires a sieve script per home directory, including the
>> global file.
>
> You are confusing the sieve_global_dir and sieve_global_path settings. The
> first is used by include, the latter specifies the default script executed
> when the user has none. Maybe these settings should be named differently,
> probably naming the latter sieve_default.

Yeah, since it behaves as a default, that name would have made sense.
That, or a sequence of path names that may or may not have %u to test
for an available script, and stop at the first one found.

sieve_global_path should have the file name?  Should it be the source
or the compiled sieve?  Does it need chmod +x ?

-- 
sHiFt HaPpEnS!

Re: [Dovecot] trying to make cmusieve happen globally

[Phil Howard]

On Wed, Dec 22, 2010 at 13:27, Thomas Leuxner  wrote:

> May be a good idea to post some snippets of the actual Postfix delivery 
> attempts just to make sure it really uses the external LDA. You can also 
> crank up 'mail_debug = yes' to log more verbosely. IIRC it should show CMU 
> Sieve warnings too (it's way old). Going forward using Pigeonhole would make 
> more sense IMHO as it spits out more useful information. Can you also post 
> the relevant Postfix 'main.cf' and 'master.cf' sections?

If it is not using the LDA, I don't know what it is doing.  There have
been several hundred thousand messages going from Postfix to Dovecot
since I installed this a few months ago, all successfully arriving in
the appropriate user INBOX as far as I can tell.  Note that I am using
a gmail account for this mailing list because I set this up way back
when I was installing the mail server, and chose to leave this set up
just in case things break.  Right now, the mail server is working
"fine" except for the sieve part behaving as if there is no sieve
part.  I just started trying to run it today.

part of main.cf is:
 begin
# Use Dovecot SASL for authentication to submit email.
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/dovecot-auth
smtpd_sasl_auth_enable = yes

# Use Dovecot for transport delivery.
virtual_transport = dovecot
dovecot_destination_recipient_limit = 1
virtual_mailbox_domains = cdb:/var/maildb/domains
virtual_mailbox_maps = cdb:/var/maildb/mailbox
 end

part of master.cf is:
 begin
#
# This is used by "virtual_transport = dovecot" in main.cf
#
dovecot   unix  -   n   n   -   -   pipe
  flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -a
${recipient} -d ${us...@${nexthop} -f ${sender}
 end

Note that the -a ${recipient} part was added on today, due to this
being shown in the http://wiki1.dovecot.org/LDA/Sieve page.  I tested
that it did not break ordinary delivery.  But things have been working
(sans sieve) for months without it.  I just assumed sieve might need
it, or benefit from it.

The filesystem access timestamps are not showing any accesses
happening on either sieve file except my own accesses from the command
line.

I do see the following as suspicious, that the settings in "protocol
lda" do not show up in "dovecot -n" at all.  But I don't know if they
should or not.  Of particular concern was that "main_plugin" was
missing.  But here (domain name redacted with ) is the
whole output of "dovecot -n" just in case something is relevant (the
"protocol lda" part of dovecot-postfix.conf follows it):
 begin
# 1.1.11: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.31-22-server x86_64 Ubuntu 9.10 ext3
base_dir: /var/run/dovecot/
log_path: /var/log/dovecot/error.log
info_log_path: /var/log/dovecot/info.log
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap pop3 imaps pop3s
listen: 172.30.0.28, [fcca::1c], 127.0.0.1, [::1]
ssl_cert_file: /etc/ssl/certs/imapx..crt
ssl_key_file: /etc/ssl/private/imapx..key
ssl_parameters_regenerate: 24
ssl_cipher_list:
ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM
login_dir: /var/run/dovecot//login
login_executable(default): /usr/lib/dovecot/imap-login
login_executable(imap): /usr/lib/dovecot/imap-login
login_executable(pop3): /usr/lib/dovecot/pop3-login
login_greeting: AUTHORIZED USERS ONLY -- unauthorized access strictly prohibited
login_greeting_capability(default): yes
login_greeting_capability(imap): yes
login_greeting_capability(pop3): no
mail_max_userip_connections(default): 10
mail_max_userip_connections(imap): 10
mail_max_userip_connections(pop3): 3
verbose_proctitle: yes
first_valid_uid: 250
mail_privileged_group: mail
mail_uid: vmail
mail_gid: vmail
mail_location: maildir:/home/mail/%Ld/%Ln/mail
mail_debug: yes
mail_executable(default): /usr/lib/dovecot/imap
mail_executable(imap): /usr/lib/dovecot/imap
mail_executable(pop3): /usr/lib/dovecot/pop3
mail_process_size: 768
mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
imap_client_workarounds(default): outlook-idle delay-newmail
imap_client_workarounds(imap): outlook-idle delay-newmail
imap_client_workarounds(pop3):
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
auth default:
  mechanisms: plain login
  username_chars:
abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz01234567890...@+
  username_format: %...@%ld
  verbose: yes
  debug: yes
  debug_passwords: yes
  passdb:
driver: passwd-file
args: scheme=crypt username_format=

Re: [Dovecot] trying to make cmusieve happen globally

[Phil Howard]

On Wed, Dec 22, 2010 at 13:28, Stephan Bosch  wrote:
> Op 22-12-2010 19:12, Phil Howard schreef:
>>
>> On Wed, Dec 22, 2010 at 11:46, Anton Dollmaier
>>   wrote:
>>
>>>> plugin {
>>>>         sieve_global_path = /etc/dovecot/sieve-global.d/spam.sieve
>>>
>>> Use "sieve_before", which is not documented in the wiki.
>>
>> Changed.
>>
> Don't. Check my other e-mail.

I backed that out.  It is now back to "sieve_global_path".


>> Mail is delivered, but still just to the regular inbox.
>
> The main question is: did you read and apply the following:
>
> http://wiki.dovecot.org/LDA/Postfix

You mean did I set up Postfix to run Dovecot deliver as the virtual
transport?  Yes.  It has been doing its thing for several months and
hundreds of thousands of emails.  I did add the -a option as described
in the http://wiki1.dovecot.org/LDA/Sieve document.

Parts of some files, and "dovecot -n" output, in a previous message.

-- 
sHiFt HaPpEnS!

Re: [Dovecot] trying to make cmusieve happen globally

[Phil Howard]

On Wed, Dec 22, 2010 at 13:28, Stephan Bosch  wrote:
> Op 22-12-2010 19:12, Phil Howard schreef:

>> Do I need to have per-user sieves set up, too, just to get it to work?
>
> No.

I did some debugging.  It looks like I do need to have that set up:
 begin
deliver(p...@ham.org): 2010-12-22 15:10:03 Error: Per-user script path
is unknown. See http://wiki.dovecot.org/LDA/Sieve#location
 end

However, this message itself is broken, because that page (redirects
to wiki1, now) does not have a label called "location".  Or is it that
the page is broken by not having a label that works for code already
distributed.  Whatever.

So I changed it to this, adding the setup for per-user scripts:
 begin
plugin {
sieve_global_path = /home/sieve-scripts/DEFAULT.sieve
sieve = /home/sieve-scripts/%u
}
 end

With this /home/sieve-scripts/DEFAULT.sieve compiled:
 begin
require "fileinto";
fileinto "INBOX.spamtest";
 end

And now I get this:
 begin
deliver(p...@ham.org): 2010-12-22 15:32:42 Info: Loading modules from
directory: /usr/lib/dovecot/modules/lda
deliver(p...@ham.org): 2010-12-22 15:32:42 Info: Module loaded:
/usr/lib/dovecot/modules/lda/lib90_cmusieve_plugin.so
deliver(p...@ham.org): 2010-12-22 15:32:42 Info: maildir:
data=/home/mail/ham.org/phil/mail
deliver(p...@ham.org): 2010-12-22 15:32:42 Info: maildir++:
root=/home/mail/ham.org/phil/mail, index=, control=,
inbox=/home/mail/ham.org/phil/mail
deliver(p...@ham.org): 2010-12-22 15:32:42 Info: cmusieve:
/home/sieve-scripts/p...@ham.org doesn't exist
deliver(p...@ham.org): 2010-12-22 15:32:42 Info: cmusieve: Using sieve
path: /home/sieve-scripts/DEFAULT.sieve
deliver(p...@ham.org): 2010-12-22 15:32:42 Info: cmusieve: Executing
script /home/sieve-scripts/DEFAULT.sievec
deliver(p...@ham.org): 2010-12-22 15:32:42 Info:
msgid=<1293049962.4148.20.ca...@lorentz>: saved mail to INBOX.spamtest
 end

Success!

-- 
sHiFt HaPpEnS!

Re: [Dovecot] Is this really a user agent issue?

[Phil Howard]

I think this issue has been entirely misunderstood.  Have I explained it wrong?

2010/12/22 Karsten Bräckelmann :
> On Wed, 2010-12-22 at 09:34 -0500, Phil Howard wrote:
>> 2010/12/21 Karsten Bräckelmann :
>>
>> > Creating the new mail folder is entirely on the IMAP server side. The
>> > MUA (Evolution in your stated case) is irrelevant. If the creation of
>> > the new folder fails, it is a server side problem.
>> >
>> > However, once a new folder has been created (server side, mind you),
>> > Evolution won't know about that folder until it is restarted. Probably
>> > the same with other MUAs, too. In the worst case, restarting Evo twice
>> > should show the new folder.
>> >
>> > (That is assuming you are not limiting your MUA to subscribed folders
>> > only, or, as IIRC is the default, deliver auto-subscribes the user to
>> > the just created folder.)
>>
>> Then I would call this a user agent issue.
>
> You snipped the part of your OP where you stated that dovecot deliver
> fails creating a mail folder. This is NOT an MUA issue.

Then whose issue it is?  There is clearly an issue because an existing
folder cannot be accessed by this MUA.


>> If IMAP provides a way for
>> the user agent to discover the folder already exists, then the user
>> agent should do this, at least if an error is encountered when trying
>> to create it.  Evolution clearly does not (or doesn't act on knowing
>> if it does).  Other user agents, I don't know.  But it is simple
>> logic:
>
> There is a way for the MUA to discover folders existing, LIST. Existing
> on the IMAP server, mind you, again. So this is not an MUA limitation.

But it would be an MUA issue if the MUA fails to do LIST after an
error from CREATE or whatever command it is.


> Also, MUAs don't care about the full list of folders, if they are in
> subscription mode. In that case, they ask for subscribed folders, LSUB,
> not all folders.

It was not in subscription mode.


> Which method do you use? (That's a client option.)

Whatever it starts out as.  I don't do subscriptions because so far it
hasn't needed to do that.  I don't want a subset of folders.  I want
all folders.  How would subscription help there?


>> if (create folder $name succeeds) OR (folder $name can be accessed) then
>>         set up local reference to folder $name
>
> This is unrelated. *sigh*  In your case, the creation is server side.
> And yes, unless you use subscription only, Evo does exactly that on the
> next start -- ask for all folders on the server.

The point is, if it tries to create a folder, and there is a failure,
and because a developer should know and understand that an existing
folder is a cause for a failure of a create operation, it should try
to distinguish this.  A user expects the folder to show up in the list
when created when there is no error.  If the folder already exists,
then there is no valid reason to not put that folder in the list now,
just as a successful create would do.  Since the protocol apparently
provides no means to reflect this situation accurately, the MUA needs
to have some means to get access to the existing folder.  Restarting
the MUA is not an acceptable means.

>
>> else
>>         report error about creating folder $name
>> endif
>
> Impossible. The error creating the folder occurred on the server, during
> delivery, which does not even need to be doveot deliver but any number
> of other MDAs.

No.  The delivery successfully created the folder.  The error happened
later when the MUA tried to create the same folder, due to the MUA not
being aware the folder had been created during delivery.  But in the
case of Evolution, because of that failure in the create operation, it
did not allow any access to that folder.  The existing folder it tried
to create did not get added to the list.  It could have been had the
logic I described been used.  Or it could have been had IMAP provided
for a better explain error (more often referred to as "an exceptional
condition" in the classic programming methods I originally learned,
rather than a pure unexplained error).  Still, the MUA could get
around that with the suggested logic.


> Either way, no client needs to be running while mail is being delivered,
> and often is not. The server just does not even tell the client about a
> failure, even *if* it potentially could know about that. This was purely
> a server-side operation and failure, and the client, any client, is not
> going to be informed about it.
>
> Only a successfully created folder will be known to the MUA during the
> LIST command.

If the folder was created by delivery of mail after the MUA was
started, and then how do 

Re: [Dovecot] Is this really a user agent issue?

[Phil Howard]

On Thu, Dec 23, 2010 at 10:27, David Warden  wrote:
> Out of curiosity, are you using the "autosubscribe" feature of Dovecot LDA
> to automatically subscribe people who whatever new folder the Dovecot LDA
> automatically creates? It looks like this is the -s flag to LDA binary in
> 1.1.3+ or lda_mailbox_autosubscribe in 2.0+ (not sure exactly what point
> release...)

I didn't purposely use it.  I see nothing in /etc/dovecot/dovecot.conf
that suggests that.  I see no -s flag on any running binary.  There is
no -s flag configured for deliver in Postfix's master.cf.  So I
conclude that answer to be "no".


> I don't have a lot of experience with Evolution but I know Thunderbird can
> be difficult when it comes to new folders showing up on the server that it
> didn't create. The best you can do is make sure the client gets
> automatically subscribed to that new folder and hope that the client checks
> the list of subscribed folders (the LSUB IMAP command) eventually. I would
> hope that clicking "check/get new mail" would be enough to trigger that in
> most clients.

I don't know the semantics of this, and would not want to suddenly
change things on users.  Maybe they are doing subscribes on their own
that might highlight what they subscribe to and ignore all else ...
and that doing autosubscribe might interfere with that.

Evolution and Thunderbird seem to be fine with no subscriptions going
on.  On startup they seem to find all the folders.  The issue I'm
encountering (tried only on Evolution since that is what I use right
now) is that when I want to access a folder that was created during
delivery, I cannot.  There is no folder shown and no option to refresh
it (unless doing a subscribe to an as-yet unseen folder would do it,
after which I could then unsubscribe) short of quitting Evolution and
restarting.  Doing "Send/Receive" doesn't refresh (I didn't really
expect it to, but I did try it just in case).  The most obvious option
is doing a right button click on the parent folder and selecting
refresh.  That didn't add the folder, either, so I conclude it only
refreshes the count of emails.  So I tried to create it within
Evolution.  I figured that would at least set up the folder name in
the tree on the left panel.  But, since IMAP reported a failure to
create (because it existed at that time, due to the earlier create
during delivery), Evolution just gave up and did not add the folder.
I believe what it should do when creating a folder fails is to check
to see if that folder already exists, and if so, either add the folder
anyway, or ask the user if the "already created" folder should be
added.

When programming in C, when I want to create a file in a directory,
and want to create the directory if it does not yet exist, what I code
is a mkdir() call with error handling that ignores EEXIST.  The
intended directory name might be something other than a directory.
But I know that open() naming a file in that directory to be created,
will fail if it wasn't a directory.  So I do not need to explicitly
stat() the directory to see if it is a directory or not.  If open()
fails, then I sort out the error to the level that program calls for.
Such a program would create the file in that directory whether the
directory exists or not.  Extending this concept to a program using a
protocol like IMAP ... I don't know if that's beyond the ability of
the application developer(s) to grasp, or not.  But it is the case
that I find very little "goofy logic" in systems and server daemons,
and find a lot of that in GUI applications.  I just didn't want to
jump to even an obvious conclusion in this case, knowing that I did
not know details about IMAP (if I had known IMAP, I probably would
have come to some conclusion, quickly).


-- 
sHiFt HaPpEnS!

Re: [Dovecot] Is this really a user agent issue?

[Phil Howard]

On Thu, Dec 23, 2010 at 12:28, Charles Marcus  wrote:
> On 2010-12-23 10:27 AM, David Warden wrote:
>> I know Thunderbird can be difficult when it comes to new folders showing
>> up on the server that it didn't create.
>
> Main reason I always uncheck the 'show only subscribed folders' option
> in the Advanced settings...

If folders were automatically subscribed when created, and while the
user agent was running and connected to imapd, would there be a
notification sent?  Or does the user agent in such a case always ask
for "all subscribed folders" and see the new one at that time.  It
would not help if the new folder did not show up.

-- 
sHiFt HaPpEnS!

Re: [Dovecot] Is this really a user agent issue?

[Phil Howard]

On Thu, Dec 23, 2010 at 17:08, Willie Gillespie
 wrote:
> Phil Howard wrote:
>>
>> I think this issue has been entirely misunderstood.  Have I explained it
>> wrong?
>
> I think there's been a bit of confusion here.  Everyone is saying similar
> things in slightly different ways.
>
> The IMAP protocol has no way to "push" to the MUA that a folder (mailbox)
> has been freshly created.  This information must be "pulled" by the client,
> ie: LISTing all folders.

Given that it appeared to be a request/response class protocol, I was
expecting that it had no such push ability.


> So, say we have an MUA connected and -something- (whether it be deliver or
> another MUA connected elsewhere) creates a new folder.  With IMAP the
> original MUA has no way of knowing that this just happened.  When it tries
> to create the same folder, the CREATE fails, because the folder already
> exists.

Right.


> How the MUA handles this situation is up to the MUA.
>
> I see a few possibilities:
> 1) it could ignore the situation and just show an error message to the user*
> 2) it could do a LIST and get an updated list of folders**
> 3) it could add the folder to its display***
>
> * Sounds like what your MUA is doing.

Yes.

> ** This could be fine and dandy, but many MUAs use the subscription list
> (LSUB) instead of showing all the folders (LIST).  So just because the MUA
> now knows the folder exists doesn't mean it will show it to you unless you
> SUBSCRIBE to it.

However, if I am not doing subscriptions, shouldn't it show me ALL
folders (per what Charles Marcus said in his message just before
yours)?  Why would this folder be handled differently if it is showing
me all the other folders?


> *** Whether this means that the MUA auto-SUBSCRIBEs you to the mailbox or
> not depends on what mode the MUA is running in.  It seems like this is what
> you want your MUA to do instead of #1.

No, that is not what I want it to do.  What I want it to do is #2 ...
and show me ALL the folders, with the new one included from the most
recent LIST.  It should do LIST as a result of there being an error
from CREATE ... to determine if the error was because the folder had
been created by other than the MUA.


> If I were a programmer, #1 would definitely be the easiest to do.  Then I
> wouldn't have to care WHY the CREATE failed, I just show an error message no
> matter what.

But #2 is not really harder.  It's another step.  I don't think of
such logic has "harder".  To me, "hardness" of programming is the
difficulty level of figuring out what algorithm to use ... e.g. what
works and is expected to work.

I would do #2.  If as a programmer I was trying to make it easier, I'd
just not write any of it at all.  When I do programming, though, I
consider that the effort to meet reasonable human expectations is part
of the job/project.  If the developer believes humans expect to not be
able to get to a folder because it had previously been created by
something else, it should at least be informative ... "Sorry, you
cannot access folders that were created by other than you, without
restarting the client".  It's just so much simpler, even with the need
to do a whole LIST request, to give the human the realistic
expectation of seeing the folder show up after a folder creation
dialog, regardless if something else created it first.


> So in answer to the question in the subject, "Is this really a user agent
> issue?" Yes.  The server is doing nothing wrong according to protocol.

I really didn't think it was.  But I was wondering if there was some
possibility the IMAP protocol had a limitation that completely
prevented this (e.g. it wouldn't be in the LIST response, either, or
LIST is only allowed once when connecting, or whatever).  It appears
IMAP is a minimal but reasonable protocol, and provides sufficient
means for MUA logic to be reasonable, and Evolution fell short of
that.  I wanted to be sure that assumption was correct.

-- 
sHiFt HaPpEnS!

[Dovecot] re-indexing moved mail

[Phil Howard]

I have a huge block of email that was misdelivered into the wrong
folder.  Moving it where it should be is not just all from one folder
to being all of another folder.  Basically, the index files need to be
rebuilt.  Looking in the wiki I cannot find how to do this.  It all
seems to be about installation, and no administrative tools.

Or would it be just as simple as delete certain index files and they
would be automagically rebuilt?

-- 
sHiFt HaPpEnS!

[Dovecot] case (non-)sensitivity for folder names

[Phil Howard]

Is IMAP supposed to be case sensitive or case in-sensitive?  It seemed
it would be case sensitive because I've had different cases of
folders.  But today I found I had two folders "Spam" and "spam", with
directories ".INBOX.Spam" and ".INBOX.spam" on the server.  Messages
existed in each directory on the server and they were different.  The
messages could be read from Evolution.  However, the list of messages
had info mixed up between the folders.  I then selected all messages
in one of the folders and trashed them (click on trash icon).
Messages disappeared in BOTH folders.  On the server, after a couple
minutes delay, messages in BOTH directories were flagged "T".  That
doesn't seem right.  But I don't know enough about what IMAP is
supposed to do, or on which end the names of folders get remapped into
directory names, to even guess which is messed up here.  I do know
Evolution keeps a track of what it thinks the folder has, because in
the past I have just "rm -fr" on a folder directory and Evolution
still showed each message in its list, although it could not access
them.

Anyone know what's the deal with that?

Unfortunately, there seems to be no operation in Evolution to tell it
to discard its own cache.

I'm also wondering if life would be any better if I used Thunderbird.

-- 
sHiFt HaPpEnS!

Re: [Dovecot] case (non-)sensitivity for folder names

[Phil Howard]

On Wed, Jan 5, 2011 at 12:05, Timo Sirainen  wrote:
> On Wed, 2011-01-05 at 10:09 -0500, Phil Howard wrote:
>> Is IMAP supposed to be case sensitive or case in-sensitive?
>
> Case sensitive, except for INBOX. (Or if the server is using
> case-insensitive filesystem then they're case-insensitive.)

So what will Dovecot do when both directories exist:  .INBOX.Spam and
.INBOX.spam


>>  It seemed
>> it would be case sensitive because I've had different cases of
>> folders.  But today I found I had two folders "Spam" and "spam", with
>> directories ".INBOX.Spam" and ".INBOX.spam" on the server.  Messages
>> existed in each directory on the server and they were different.  The
>> messages could be read from Evolution.  However, the list of messages
>> had info mixed up between the folders.  I then selected all messages
>> in one of the folders and trashed them (click on trash icon).
>> Messages disappeared in BOTH folders.  On the server, after a couple
>> minutes delay, messages in BOTH directories were flagged "T".  That
>> doesn't seem right.
>
> My guess is that Evolution messes this up.

And it might be confused by odd data it gets due to both above
directories existing.


>> Unfortunately, there seems to be no operation in Evolution to tell it
>> to discard its own cache.
>
> rm -rf ~/.evolution/mail/imap/

Thanks.  I'll give that a try soon.

-- 
sHiFt HaPpEnS!

Re: [Dovecot] case (non-)sensitivity for folder names

[Phil Howard]

On Wed, Jan 5, 2011 at 13:20, Timo Sirainen  wrote:
> On 5.1.2011, at 19.33, Phil Howard wrote:
>
>> On Wed, Jan 5, 2011 at 12:05, Timo Sirainen  wrote:
>>> On Wed, 2011-01-05 at 10:09 -0500, Phil Howard wrote:
>>>> Is IMAP supposed to be case sensitive or case in-sensitive?
>>>
>>> Case sensitive, except for INBOX. (Or if the server is using
>>> case-insensitive filesystem then they're case-insensitive.)
>>
>> So what will Dovecot do when both directories exist:  .INBOX.Spam and
>> .INBOX.spam
>
> Same as if they were called .INBOX.Spam and .INBOX.HelloWorld. They are two 
> completely different mailboxes.

But if for INBOX names are case IN-sensitive, then there will be
confusion because anything obeying that concept would see "Inbox >>
spam" and "Inbox >> Spam" as being the same box.

There are different ways to deal with case IN-sensitive name matching.
 A lame way is to translate all names to a specific case and do a
lookup at the OS layer (which on Unix is case sensitive, so which name
is found, if any, depends on how the case-ify is done).  A somewhat
smarter way is to scan the list of names and case-ify each, and
compare that the case-ified search name (where they are case-ified the
same way, example always upper, or maybe always lower).  When a match
is found, that leaves two alternatives: either stop there and use that
name (which means which is used depends on the order names were
obtained), or continue for more.  If continuing for more and more are
found, are they merged is the next question.  The logic can get more
convoluted with this.

The behavior I have seen, through Evolution (I just installed
Thunderbird and will be trying that in a while), is that when I had 3
existing, "foo", "Foo", and "FOO", all under inbox (seen as ".INBOX"
on the server, and as "Inbox" in the client), there were various
situations of getting things mixed up among them as if accessing one
would really access another, though it was unclear which.  But I
definitely ran into the case where when I had ONE message in each of
"foo", "Foo" and "FOO", and deleted the message in "foo" only, it
deleted one message in all three, but left the new message count on
"FOO" as "(1)" even though it showed now messages.

The subfolders are definitely NOT kept completely distinct.  It's as
if some software was merging them, and some was not.  Since all 3 did
show up, clearly Dovecot did not merge them to LIST them to the
client.

My boss was using Apple Mail, then tried Thunderbird, and similar
effects are happening.  I'm wanting to believe that IMAP is just so
poorly defined that everyone is doing things just a bit differently,
and enough to sort of work, but mess things up.

-- 
sHiFt HaPpEnS!

Re: [Dovecot] case (non-)sensitivity for folder names

[Phil Howard]

On Wed, Jan 5, 2011 at 16:10, Timo Sirainen  wrote:
> On 5.1.2011, at 23.04, Phil Howard wrote:
>
>> But if for INBOX names are case IN-sensitive, then there will be
>> confusion because anything obeying that concept would see "Inbox >>
>> spam" and "Inbox >> Spam" as being the same box.
>
> It's case insensitive only for the INBOX itself, not its children. So 
> INBOX.spam and inbox.spam are the same, but INBOX.Spam isn't the same as 
> either of them.

Ah, I misinterpreted the original reference to be recursive.


>> The behavior I have seen, through Evolution (I just installed
>> Thunderbird and will be trying that in a while), is that when I had 3
>> existing, "foo", "Foo", and "FOO", all under inbox (seen as ".INBOX"
>> on the server, and as "Inbox" in the client), there were various
>> situations of getting things mixed up among them as if accessing one
>> would really access another, though it was unclear which.  But I
>> definitely ran into the case where when I had ONE message in each of
>> "foo", "Foo" and "FOO", and deleted the message in "foo" only, it
>> deleted one message in all three, but left the new message count on
>> "FOO" as "(1)" even though it showed now messages.
>
> I'm pretty sure this confusion comes from Evolution internally.

Given the above clarification, that appears to be the case.


>> My boss was using Apple Mail, then tried Thunderbird, and similar
>> effects are happening.  I'm wanting to believe that IMAP is just so
>> poorly defined that everyone is doing things just a bit differently,
>> and enough to sort of work, but mess things up.
>
> It's possible that multiple clients do case insensitive comparing. I doubt 
> many people have actually tested what happens then. I only know Dovecot's 
> code, and there's about zero possibility of it mixing up mailboxes with 
> different casing.

Some of this might actually be more a case of how clients handle cases
where nothing is subscribed.  Apparently Thunderbird has no such mode,
and always subscribes to what it creates.  Evolution lets you run
unsubscribed, but behaves like it had some kind of phantom
subscription state.  Some things do clear up when .evolution state is
wiped clean.

I'm moving over to Thunderbird in the next day or so.

-- 
sHiFt HaPpEnS!

[Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

I'm setting up a Postfix and Dovecot combination.  What I want to do is have
a user database that (1) is not running from some engine (so not LDAP or SQL
or such) ... and (2) is completely disassociated from system users (e.g.
most email users are not in /etc/passwd and most /etc/passwd users are not
email users).  Ideal would be a one-file solution, which can be managed by
text editing or simple command line tools.  But what I want is ONE file that
both Postfix (for valid recipients) and Dovecot (for user login
authentication) can use together.  An alternative is some way to get Postfix
to go through Dovecot to query for users (at the time of mail arriving on
SMTP so it doesn't queue anything that would later be rejected).  This is a
smallish setup on one server, with probably a max of 50 to 100 users and 50
or so role account mailboxes over the next year or two.  Any
recommendations?

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Wed, Apr 21, 2010 at 10:47 AM, Patrick Nagel wrote:

> I think /etc/passwd is as close as it gets to your requirements... why not
> just add the users as system users, and set their shell to /bin/false?
>

There would be conflicts in this, especially with multiple domain names
(sorry, forgot to mention that ... there will be about 10 domain names).

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Wed, Apr 21, 2010 at 10:58 AM, Peter Hessler  wrote:

> postfix:
>
> smtpd_sasl_type = dovecot
> smtpd_sasl_path = private/auth
>
>
> dovecot:
>
> auth default {
>  socket listen {
>client {
>  path = /var/spool/postfix/private/auth
>  user = _postfix
>  group = wheel
>  mode = 0660
>}
>  }
> }
>
> Then configure your favorite auth mechanism for dovecot.  Let bake for 20
> minutes, add salt for taste.
>

This looks interesting.  That would basically be run through Dovecot with a
named socket I take it.  How would the lookup table type be specified for
local_recipient_maps in Postfix to use this?

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Wed, Apr 21, 2010 at 1:30 PM, Rodolfo Gonzalez  wrote:

> Phil Howard escribió:
>
>  On Wed, Apr 21, 2010 at 10:47 AM, Patrick Nagel > >wrote:
>>
>>  I think /etc/passwd is as close as it gets to your requirements... why
>>> not
>>> just add the users as system users, and set their shell to /bin/false?
>>>
>>>
>> There would be conflicts in this, especially with multiple domain names
>> (sorry, forgot to mention that ... there will be about 10 domain names).
>>
>
>
> Then I think MySQL will do the job. Both postfix and dovecot support MySQL,
> and you can use SASL to authenticate SMTP with Dovecot, so Dovecot would do
> all the auth work. Finally, you could use Postfix's VDA patch if you want to
> use Maildir++.
>
> Hope this helps.
>

I don't want to use any other server engine of any kind with this.  I'm
trying to keep it small and lean, and minimize what the people that have to
monitor and fix it need to know.  So at the present time, I am excluding all
databases like any SQL or LDAP or anything else that needs to run as a
daemon/engine/service.

Ideal would be a single file both can read, be it a flat file, or a Berkeley
DB file.  Next to that would be two files where one is authoritative and the
other (for Postfix since it only needs a list) is generated from it.  That
looks like what I ultimately will do (the script that adds users will just
generate the files and test them).

I was hoping there might be a way to get Postfix to use the legacy Unix
passwd file format, but with a different file name.  It doesn't seem to have
that ability.  It would make things very simple if it did.  It would also be
simple if the system /etc/passwd file could be used, but it can't.

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Wed, Apr 21, 2010 at 3:30 PM, Thomas Leuxner  wrote:

> I'm running a setup that should be good enough for what you are trying to
> achieve. All user information is stored in flat files per domain and you may
> override per user settings individually:
>
> passdb {
>  args = username_format=%u /var/vmail/auth.d/%d/passwd
>  driver = passwd-file
> }
>
> userdb {
>  args = username_format=%u /var/vmail/auth.d/%d/passwd
>  driver = passwd-file
> }
>

What does it take to get Postfix to read this?

$ cat passwd
> u...@domain.tld:{scheme}:5000:5000::/var/vmail/domain.tld/user::userdb_quota_rule=*:storage=5G
> userdb_acl_groups=PublicMailboxAdmins
>

In which directory was this?



> I would vote against storing aliases in these files though. Reason being
> the Postfix alias files are more flexible, because you would need to setup
> NULL password/No Login users or similar in the Dovecot backend. Another
> reason to keep them in Postfix is to completely separate alias management
> from the user management and use the same for login checks.
>
> See how aliases are used for routing and to authenticate valid mail from
> senders with one file:
>
> $ cat virtual
> al...@domain.tldlo...@domain.tld
> postmas...@domain.tld   lo...@domain.tld
>

I suspect I will want to be maping virtuals between different domains, so I
might have

ab...@example.commailad...@example.net
ab...@example.netmailad...@example.net
postmas...@example.commailad...@example.net
postmas...@example.netmailad...@example.net


[main.cf]
> virtual_mailbox_domains = domain.tld, domain1.tld
> virtual_mailbox_base = /var/vmail
> virtual_minimum_uid = 100
> virtual_uid_maps = static:5000
> virtual_gid_maps = static:5000
> virtual_alias_maps = hash:/etc/postfix/virtual
> virtual_transport = lmtp:unix:private/dovecot-lmtp
> […]
> smtpd_sender_login_maps=hash:/etc/postfix/virtual
>

One thing I need to watch out for, and am concerned with because the last
time I used Postfix there were a bunch of "virtual" configurations that
really didn't work for me for a reason I cannot recall right now ... is that
the same user name in different domains is NOT always the same user.  E.g.
b...@example.com is NOT the same person as b...@example.net while
b...@example.org doesn't even exist.  So there needs to be distinct entries
for b...@example.com and b...@example.net (and not any for b...@example.org and
have Postfix reject that during incoming SMTP sessions).

There can also be cases where m...@example.com and m...@example.net are the
same person, and Mike wants to have mail to these two addresses kept in
separate mail boxes (and presumably must do separate logins, so he'd have to
set up 2 accounts in his MUA) ... as well as st...@example.com and
st...@example.net also being the same person, but Steve wants everything in
one mailbox (so he'd have to pick between st...@example.com and
st...@example.net and I'd have to set up a virtual map for the other to be
delivered to the mailbox of his choice ... in a separate lookup table in
Postfix).


If this seems suitable I can send more details to you.
>

It might well be as long the domains are fully distinct.  I'll have to go
read up on each of the virtual_* configuration parameters to be sure of the
effects.  I was thinking to use:

mailbox_command = /usr/lib/dovecot/deliver

in Postfix main.cf.  Is that workable instead of "virtual_transport =
lmtp:unix:private/dovecot-lmtp"  Or would running LMTP be a better way?

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Wed, Apr 21, 2010 at 5:45 PM, Heiko Schlittermann 
wrote:

> Hello Phil,
>
> Phil Howard  (Mi 21 Apr 2010 16:32:36 CEST):
> > I'm setting up a Postfix and Dovecot combination.  What I want to do is
> have
> > a user database that (1) is not running from some engine (so not LDAP or
> SQL
> > or such) ... and (2) is completely disassociated from system users (e.g.
> > most email users are not in /etc/passwd and most /etc/passwd users are
> not
> > email users).  Ideal would be a one-file solution, which can be managed
> by
> > text editing or simple command line tools.  But what I want is ONE file
> that
> > both Postfix (for valid recipients) and Dovecot (for user login
>
> A recent demonstration of a German postfix expert used a sed-Script to
> convert (basically cut everything behind the first „:“) the dovecot
> passdb file to a postfix readable text file (and convert this to a
> hash(?)).
>

Dozens, maybe millions, of ways to do that.  The "cut" command might be
enough.  I do stuff in C, Bash, Pike, Awk, and even a little Python (just
started learning it), as needed.  I'd just integrate this copying and
conversion into the script used to add mail users.


I'm not sure, if postfix really can't read a passdb (passwd-like) file.
> Probably it (postfix) isn't flexible enough for doing this, or the
> expert didn't want to show it.
>

It has the code to parse it.  It just assumes a specific file location (e.g.
the Unix system file /etc/passwd).


As an exim user I'd suggest using exim - and enjoing real flexiblity ;-)
> The solution I'd prefer is (d) - it makes your exim independend on the
> userdb/passdb used by dovecot, you're just talking to the auth-master.
> (Something I'd implement additionally is a „softfail“ (4xx error) in
> case the socket is not usable.)
>

I'm already more familiar with Postfix.  It's Dovecot and IMAP that's new to
me right now.  I can make this work in Postfix.  But I'm just checking for
shortcuts I don't otherwise see.

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Thu, Apr 22, 2010 at 3:33 AM, Thomas Leuxner  wrote:

> On Wed, Apr 21, 2010 at 04:34:30PM -0400, Phil Howard wrote:
> > > userdb {
> > >  args = username_format=%u /var/vmail/auth.d/%d/passwd
> > >  driver = passwd-file
> > > }
> > What does it take to get Postfix to read this?
>
> Basically these two parameters in 'main.cf':
>
> [main.cf]
> smtpd_sasl_type=dovecot
> smtpd_sasl_path=private/auth
>
> Since this will have implications when Dovecot is not running/unavailable
> as Authtentication Backend, Postfix will reject legit incoming mail in
> that case, it is better to put this in the master configuration actually
> and have Postfix use a dedicated submission port for SASL clients:
>
> [master.cf]
> submission inet n   -   -   -   -   smtpd
> smtpd_tls_security_level=encrypt
>  -o smtpd_sasl_auth_enable=yes
>  -o smtpd_sasl_type=dovecot
>  -o smtpd_sasl_path=private/auth
>  -o smtpd_sasl_security_options=noanonymous
>  -o smtpd_sasl_local_domain=$myhostname
>  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
>  -o smtpd_sender_login_maps=hash:/etc/postfix/virtual
>  -o smtpd_sender_restrictions=reject_sender_login_mismatch
>  -o
> smtpd_recipient_restrictions=reject_unknown_recipient_domain,reject_non_fqdn_recipient,permit_sasl_authenticated,reject
>

So what would local_recipient_maps look like in this case?  At this point, I
don't understand what is happening for this.  I would be expecting Postfix
to be asking Dovecot if a user is valid.  This is for mail incoming from
outside, to make the rejection decision during the SMTP session.  This looks
more like a configuration to provide a submission interface and authenticate
through Dovecot.  That's fine, and probably what is needed.  But I'm trying
to sort out the local_recipient_maps at this time.  Can this solve both
issues at the same time?


> It might well be as long the domains are fully distinct.  I'll have to go
>  > read up on each of the virtual_* configuration parameters to be sure of
> the
> > effects.  I was thinking to use:
> >
> > mailbox_command = /usr/lib/dovecot/deliver
> >
> > in Postfix main.cf.  Is that workable instead of "virtual_transport =
> > lmtp:unix:private/dovecot-lmtp"  Or would running LMTP be a better way?
>
> LMTP would be better long-term as it is more flexible and robust, e.g.
> allowing multiple recipient deliveries in parallel and has  a real
> protocol handshake compared to piping into the LDA, but both is
> feasible. Hower LMTP is available with Dovecot 2.0 only.
>

I'm doing this on Ubuntu 9.10 and it has Dovecot 1.1.11 right now (we need
to get this mail server up before we will be ready to eval Ubuntu 10.04 or
another distro).

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Thu, Apr 22, 2010 at 7:12 AM, Rainer Frey  wrote:

> This is wrong. The auth service is not queried for recipient, only for
> valid
> SASL users (that connect to the submission service as *senders*). I'm
> talking
> about determining valid *recipients* for the virtual_mailbox_domains.
>

That's what I was (initially) talking about, too.  Of course, having a
submission service would be a good idea so out-of-office users don't have to
use some remote SMTP submission.  And that will have to be authenticated.  A
combination solution would be good.  But I definitely need to have the
recipient test done based on a common database of "mail users".

If the wiki can describe how to set up both, together, that is great.  If it
has to involve extracting usernames out of a file and running makemap in
Postfix to build a btree db file, that's OK, though I was checking to see if
there was a known way to avoid that.

If Postfix will reject incoming mail when Dovecot is not up, that's could be
bad, depending on how it's rejecting.  If it can validate from a file, then,
in theory, Postfix can queue and deliver safely later (and only users that
were deleted in the interim risk a separately mailed reject).

The ideal would be a complete mail server package that handled it all in one
... SMTP, submission, IMAP(S), POP3(S).  But what I've seen as attempts to
do that so far were less than promising (even though I have no need for what
most of the MTAs do).

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Thu, Apr 22, 2010 at 5:42 AM, Thomas Leuxner  wrote:

> On Thu, Apr 22, 2010 at 11:18:09AM +0200, Rainer Frey wrote:
> > What I don't see here at all (and neither in your Wiki Howto) is how
> Postfix
> > determines the valid recipients for the domains in
> virtual_mailbox_domains.
>
> Postfix will expand possible aliases first and determine the final
> recipient handing over to Dovecot:
>
> virtual_alias_maps = hash:/etc/postfix/virtual
>
> It will query the recipients by connecting to the socket in its
> chroot provided by Dovecot:
>
> service auth {
>  unix_listener /var/spool/postfix/private/auth {
>group = postfix
>mode = 0660
>user = postfix
>  }
>  user = doveauth
> }
>
> Once it has the homedir it will send it off via LMTP or deliver,
> whichever you configured via:
>
> virtual_transport = lmtp:unix:private/dovecot-lmtp
> or
>
> virtual_transport = dovecot
>
> >
> > The correct parameter would be  virtual_mailbox_maps, but AFAIK there is
> no
> > lookup table that read the passwd format from an arbitrary file. So a
> script
> > that generates a hash/whatever postfix lookup file from the passwd-files
> would
> > still be necessary.
>
> There is no such thing as a correct parameter from my perspective. I did
> not say that alias creation was to be unified/automated. Instead I said I
> did not even think
> this is good practice to me. Anyone with at least a bit of sed/awk
> knowledge
> can kludge it from the flat-files anyway.
>

So your suggestion for local_recipient_maps is to read from a table
extracted from the file used by Dovecot.  And then use the previously
described setup (now in the Wiki) for a proper submission interface (using
SASL and asking Dovecot).'

I have Dovecot 1.1.11, and for now using package versions outside of what
Ubuntu offers isn't an option (that's something I'll be doing later with a
different distribution, but it isn't an option right now).  So I can't have
submission ask via LMTP.  If I can get Postfix to authenticate via passwords
in a btree or cdb, I guess I can just copy BOTH usernames AND passwords from
the file Dovecot reads from and build the btree or cdb db file for Postfix
from there.

Any chance Dovecot can read either btree or cdb files directly?

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Thu, Apr 22, 2010 at 10:43 AM, Rainer Frey wrote:


> If you can't wait for Dovecot 2.0, you need to use dovecot deliver, but you
> should set it up as a pipe transport in master - see
> http://wiki.dovecot.org/LDA/Postfix for virtual users. mailbox_command
> again
> is for real system users only.
>

Basically what I expect to be doing is:

1.  Postfix listens on SMTP for incoming MX mail to local (as in virtual,
not system) users.

2.  Postfix listens on Submission, encrypted only, and authenticates users
to submit mail for delivery anywhere.

3.  Dovecot listens on encrypted IMAPS and POP3S for user authenticated
access to mailboxes.

4.  Everything but MX to SMTP on port 25 shall be encrypted only.  If I can
force the use of STARTTLS on the non-encrypted ports, then it would be OK to
use them that way.  But I do not want to give any user an option to not be
encrypted.

5.  Passwords stored encrypted, such as MD5.  And it should be a scheme that
both Postfix and Dovecot can use so I don't have keep two different
encryption schemes.

6. I'd prefer not to, but it looks like I will have to copy data from one
format to another format so Dovecot can read it and Postfix can read it.  I
will most likely be using the CDB format (the constant database file format
from Dan Bernstein ... which I'd think should be easy enough for a future
version of Dovecot to support).

7.  I am wondering if I can trick Postfix into reading virtual user info by
running it chrooted where I substitute /etc/passwd and /etc/shadow with
stuff I generate from Dovecot files.

Re: [Dovecot] best choice of user database file to work with postfix

[Phil Howard]

On Thu, Apr 22, 2010 at 12:12 PM, Jerry  wrote:

> On Thu, 22 Apr 2010 17:03:00 +0200
> Rainer  articulated:
>
> > Well, it leaves out the *one tricky part* of using a flat file
> > database for virtual users with dovecot and postfix: there is no
> > common format that both understand directly.
>
> I have not been following this thread as closely as I probably should
> have; however, I was wondering what the OP's problem was with using
> MySQL? It would greatly simplify the job of constructing and
> maintaining databases. It is even possible to create tables that both
> Postfix and Dovecot can use jointly if desired. I use MySQL for several
> projects, and would never go back to using 'flat files" unless there
> was no other way to achieve my goal.
>

The administration is going to be handed off to less technical people, and
my goal is to mimize the number of elements in this.  It's not about MySQL
itself ... it's about not running yet another server/daemon.

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Thu, Apr 22, 2010 at 12:07 PM, Jim Trigg  wrote:

> On Thu, Apr 22, 2010 at 09:48:36AM -0400, Phil Howard wrote:
> > The ideal would be a complete mail server package that handled it all in
> one
> > ... SMTP, submission, IMAP(S), POP3(S).  But what I've seen as attempts
> to
> > do that so far were less than promising (even though I have no need for
> what
> > most of the MTAs do).
>
> The only one like that I know of is Courier, and I found it incredibly
> frustrating to work with.  I use dovecot and postfix, but don't have the
> virtual user issue -- all of my users (for twelve domains) have real
> accounts.
>

I would have done that, except for the fact that there is the issue of same
user name users under different domains, and I don't want the mess of
mapping those in strange ways to system users (and I don't want "@"
characters in system users).

Re: [Dovecot] best choice of user database file to work with postfix

[Phil Howard]

On Thu, Apr 22, 2010 at 12:40 PM, Jerry  wrote:

> On Thu, 22 Apr 2010 12:18:41 -0400
> Phil  articulated:
>
> > The administration is going to be handed off to less technical
> > people, and my goal is to mimize the number of elements in this.
> > It's not about MySQL itself ... it's about not running yet another
> > server/daemon.
>
> Excuse my ignorance here; however, it appears that what you are trying
> to accomplish is exceedingly harder to implement than simply using SQL
> for the job. The are numerous GUIs to administer SQL with. Personally,
> I use "phpMyAdmin" with MySQL and have found it quite a good fit. There
> are several others of course, and other flavors of SQL that you might
> want to investigate.
>
> I have personally set up MySQL with phpMyAdmin for some rather
> technically challenged individuals and they have not experienced any
> problems that I am aware of. There is also a PostfixAdmin
>  that might simplify
> your life. I don't use it myself; however, I have heard it is quite
> good.
>

The decision to go non-SQL and non-LDAP has already been made.  Only if
nothing else can work will it be re-opened.  The non-SQL/non-LDAP approach
is what my current investigation is on.

Re: [Dovecot] best choice of user database file to work with postfix

[Phil Howard]

On Thu, Apr 22, 2010 at 3:12 PM, Stan Hoeppner wrote:

> With this many lookup table types supported by Postfix, is it true that it
> has no "simple" table type in common with Dovecot?
>

There are some ... like mysql for example.  The ones I call "simple" are
ones that have a single file in some readily contructable form.  These
include btree (Berkeley DB) and cdb (Dan B's CDB).  Others could be included
but I haven't looked at them, yet (such as cidr, which has specific
purposes, only).  Things I consider "not simple" involve making a network
connection.  It can certainly be simple to configure them.  But I'm
classifying these in terms of the number of things going on to get the
answer.  Even a DB scheme to use a DNS lookup (not implemented for Postfix)
I would consider as "not simple" here, even though it is likely very simple
to do.  The tcp scheme is probably the simplest of the network schemes (but
it requires something to listen somewhere).  If I were asked what I think
could be added to this list of lookup schemes, it might be a long, long
thread.  Let's not do that here.

 unix (read-only)
> A  limited way to query the UNIX authentica-
> tion  database.  The  following  tables  are
> implemented:
>
> unix:passwd.byname
>The  table is the UNIX password data-
>base. The key is a login  name.   The
>result  is  a  password file entry in
>passwd(5) format.
>
> unix:group.byname
>The table is the UNIX group database.
>The  key is a group name.  The result
>is a group  file  entry  in  group(5)
>format.
>

This is "simple", too.  But the problem is it isn't flexible enough.
Dovecot supports this scheme and can use it on any specified file.  Postfix
is limited to just the system files (and is probably using Unix library
calls to access it, hence the source of the limitation).

I would think the general (insert some character here)-separated file should
be doable:
comma-delimited:/some/file/name
colon-delimited:/some/file/name
tab-delimited:/some/file/name
space-delimited:/some/file/name
But it hasn't been done that I can see (some third party patch not
withstanding).

Anyway, I need to re-study what is involved with the virtual users, since I
may have to be doing this in Postfix just to get submission authentication
to work, and then see how that affects using local_recipient_maps.  And at
this point it's probably more of a SASL issue than a Dovecot issue.  So back
to reading Postfix docs (including the SASL reference someone gave) and
asking on the Postfix M/L if needed (not here, this is for Dovecot).  It may
well be the description given early to use Dovecot SASL to authenticate
Postfix submission users would be just fine, as long as Postfix does a SOFT
rejection when Dovecot isn't up, so it requeues files.  On the SMTP end, I
just want an up front rejection decision to avoid backscatter.  The one
backscatter hole where an existing user is being deleted while incoming mail
is being queued is probably insignificant.

Re: [Dovecot] best choice of user database file to work with postfix

[Phil Howard]

On Thu, Apr 22, 2010 at 4:31 PM, Arne K. Haaje  wrote:

Have you looked into Postfix Admin? http://postfixadmin.sourceforge.net/
>
> It might be a good solution for you. I'm using it for a a growing database
> of users and I'm very happy with it. The setup with postfix, dovecot and
> mysql was quite straight forward, and this interface requires no particular
> technical know-how.
>
> Should be perfect if you can do the initial setup, then just give the
> admisn the password to this interface.
>

No, I had not looked into this, yet.  But it won't fit in here.
Adding/deleting users is going to be going though another system where
likely the HR person will be doing it.  The admin will then be handling the
"what's wrong with it" issues (as opposed to the clerical work of
adding/deleting users).  And it requires database engine.

Re: [Dovecot] best choice of user database file to work with postfix

[Phil Howard]

On Thu, Apr 22, 2010 at 5:19 PM, Stan Hoeppner wrote:

> One nice thing about Postfix is that the documentation is _very_ thorough,
> even if sometimes hard to digest.
>

Yes, I would agree.  Sometimes a twisty maze of passages, but you can
eventually find things.

Good luck, and please keep us up to date.  This is an interesting topic and
> the solution may be useful to others.
>

I'll try to write a wiki page on what I do, if it isn't a duplicate of one
already there.  I'd call it a "minimalist mail server".

Re: [Dovecot] best choice of user database file to work with postfix?

[Phil Howard]

On Fri, Apr 23, 2010 at 2:49 AM, Rainer Frey  wrote:

> On Thursday 22 April 2010 18:15:18 Phil Howard wrote:
> [ ... all standard stuff that is well documented ... ]
>
> > 5.  Passwords stored encrypted, such as MD5.  And it should be a scheme
> > that both Postfix and Dovecot can use so I don't have keep two different
> > encryption schemes.
>
> Postfix doesn't need any password directly. It only authenticates a user
> with
> a password in one case: SMTP SASL authentication of submission users, and
> it
> uses the dovecot auth service for that and does not read the password
> database
> itself.
>

OK, so all that is in whatever SASL configuration, and so I'll be doing that
in Dovecot.

 It is not about supporting a certain database library. There is simply a
> difference in what Dovecot and Postfix need from a user/recipient database.
> Dovecot needs information like path to mailbox, uid/gid with which to put
> files into the mailboxes, extra configuration fields ... Therefore it uses
> a
> structured  multi-value format like the passwd-file. CDB or similar don't
> work
> like this, so dovecot can't easily support using the same CDB files as
> postfix.
>

Why not have all those fields combined into the value string you get from
CDB?  If it can do it from a line out of /etc/passwd then it can do it from
CDB (the value could simply be that same line ... or defined to be something
else more convenient).


Postfix only supports name:value tables, either to use the value
> (table-style
> lookup) or to check whether there is an entry for a name (list-style
> lookup).
> Therefore it only uses file databases with such a mapping. In the case of
> valid recipients which will be handed off to dovecot for delivery, it is
> such
> a simple list lookup.
>

The value can be a tuple (see above).  We're lucky that what Postfix needs
is "does this exist".  So it can just completely ignore the value and not
need any code to parse it.



> (If it interests you: the postfix virtual delivery agent needs very similar
> information to dovecot, but it uses several key:value maps for the
> different
> information, all with the recipient as key.)
>

I've run into that before.  I think it was the source of some difficulty.
In programming I've done, I've stored structured data in Berkeley DB and
never needed but one table/file per class of index.

Re: [Dovecot] best choice of user database file to work with postfix

[Phil Howard]

On Fri, Apr 23, 2010 at 3:45 AM, Ed W  wrote:

You need to look at where you are going with this One way or another you
> need a database - call it a banana if you prefer, but it's still a database
> whether it's a flat file or a BDB file or whatever
>

One must be careful with the term "database".  I agree with you that it
could be anything.  But there are lots of people out there that assume some
server daemon or engine when you say "database".  And most of them assume
"SQL" is in there, too.  So I avoid that term.



> Your requirements appear to be badly phrased.  What you *appear* to want is
> a black box system which is as simple to maintain as possible.  However, you
> have stated a bunch of hard to meet criteria such as not allowing any long
> running code to support your needs (a daemon).  I really don't see that you
> should exclude additional daemons, simply evaluate the complete system with
> and without and choose the one which is "easiest" to maintain
>

Yes, a black box system would have suited.  But it needs to be open source
(that's not a contradiction ... I just have't seen one that I can use).  It
also needs to be able to integrate with what I will have, which is an
exported file, which I have reason to suspect will be in a spreadsheet form.


1) Yes you CAN setup something which uses a plain text database, I tried it
> for a while but at 50 users I decided it was too complicated and switched to
> a DB.
>

What complications?  Were you hand editing this file?


2) You are optimistic if you don't think some user will accidently add a
>  halfway through the text file, completely breaking it and have the
> skills to realise what they did.  Or they will add a "," in the middle of a
> password and change the meaning of all your fields.  Or they will miss a
> field save the file and then never realise their mistake.  Better yet are
> errors which cause some external (grep,sed,awk) function to choke on the
> input and cause some really wierd effects downstream...
>

Ah, yes, hand editing.  But that won't be happening in my case.


3) Why should an sql database need any monitoring and maintenance over the
> next X years?  Yes you can corrupt the files, but I would hope for very
> decent uptimes and after all they aren't going to repair a corrupted boot
> sector so you need some kind of maintenance plan in place, simply work in a
> full OS backup into the plan and this saves your DB at the same time?
> (Offlist we can discuss, but a simple rsync to some separate partition
> should do it)
>

You rsync the files an SQL database (like MySQL) works from, and don't
expect corruption?  That's only safe if the database is synced and shut
down.  I don't want to be doing that.  If I did run a database engine, it
would have to import everything ... as a single massive transaction (or
maybe a live table switch scenario between two tables).  To back it up I'd
either export the entire table to a file (and send that off to the archive),
or just back up the file I used to import with.


4) If you really, really don't want a daemon based database then you will
> have to look at bdb (if dovecot supports it?) or sqlite which I think
> postfix+dovecot both support.  You can then add convenience command line
> functions to examine and alter the data. Those convenience functions can
> error check the input also.
>

CDB would work, since I would be importing everything all at once every time
a change takes place (once every few days perhaps).  Berkeley DB would work,
too (nearly as fast).  A flat file cached in RAM by the OS would be about as
fast for these few number of users (that's a "database", too, in its own
way).


Good luck
>

It looks like the way I will be doing this is to let Dovecot handle
Postfix's SASL needs and generate a little users table for Postfix to make
the accept/reject decision on incoming email and spam.  So my task now is to
decide what form I will give the data to Dovecot.

Re: [Dovecot] best choice of user database file to work with postfix

[Phil Howard]

On Fri, Apr 23, 2010 at 5:23 AM, Ed W  wrote:

> P.S.  The "idiot" who kept breaking the plain text format file in my
> original setup was  da da ... me ... So given I think of myself as
> reasonably technical, I would claim that text format "databases" are way
> more fragile than you might expect (good luck getting a non technical
> user not to break the text format file from time to time...)
>

But this is more an issue of exposing the system to human failure than it is
a choice of using a flat file.  If the flat file is generated by some other
system, and maybe merged with other data, and never handled by a human
except in extraordinary situations, it should be fine.  This is used as a
justification for XML being a text format (that a human can read it and even
edit it, in non-routine situations).

FYI, I have done hand editing of /etc/{group,passwd,shadow} to add users.
So far I've never killed root.  I still have to at times.  Most recently I
did so with Ubuntu because the user management tool couldn't get it right (I
needed a specific UID/GID arrangement for compatibility with other systems,
and the GUI tool refused ... it would let me do one or the other, but not
both at the same time).

[Dovecot] order of variable modifiers

[Phil Howard]

I'd like to have a variable substitute take the domain name part, lower case
it, MD5 it, and take a 2-character substring of that.  Would that be like
%2MLd ?  The documentation on the H hash modifier described reversing with
R, but expresses it at %3RHu which seems to me like it would be reversing
the hash result rather than the user string.

I'm undecided about MD5 vs. hash, and could go either way.  But I was
thinking:

mail_location = /data/mail/%2MLd/%Ld/%2MLn/%Ln

[Dovecot] adding users to passwd-file

[Phil Howard]

Is there a tool equivalent to the system "passwd" command (or maybe
"adduser" or "useradd") that can support a passwd-file by setting a
password, encrypting it with the salted MD5 scheme?  The system "passwd"
command doesn't have an option to "do it to this alternate file instead of
/etc/shadow".

Re: [Dovecot] adding users to passwd-file

[Phil Howard]

On Mon, Apr 26, 2010 at 4:38 PM, Heiko Schlittermann 
wrote:

> Some people use „htpasswd“, but there not password scheme is coded into
> the password hash.
>
> It's not clear what you're seeking - some tool for generating the hashed
> string (e.g. „openssl passwd -1 "$cleartext"“) or some tool for managing
> the
> entries in the (dovecot) password file.
>

I left it vague to handle some variety.  Something to add/delete users is
nice.  Something to change password in place is nice.  Something that given
a user and plain text password and everything else Dovecot needs when using
it as userdb, and would output to stdout the line contents that would go
into the file, would do the job.

I want to avoid re-inventing the wheel.  If there are no wheels, I guess
I'll just look up the salted-MD5 format details and do it.

Re: [Dovecot] CDB Authentication database

[Phil Howard]

I'd be interested in this, too.  It might be a nicer way for smaller sites
to share between Dovecot and Postfix.  CDB is a read-only (C meaning
constant) dictionary file designed by DJB.  To update it, build a new one
and swap.  Alternatively, use 2 or more DBs with the CDB having the bulk of
information and a smaller dynamic DB first having the day's changes.  Then
the bulk data can be rebuilt daily overnight.

Another even simpler "DB" would be a directory index.  The directory is the
DB.  The key selects a file name.  The data is the file contents.  I'll
leave it to your imagination how to update it.  It could be a simple "small
first in line" DB out of 2 or more.

On Thu, Apr 29, 2010 at 9:17 AM, Luca Corti  wrote:

> Hello,
>
> I'd be interested in seeing a CDB based backend for the userdb/passdb
> lookups.
>
> Is anyone working on this? Is there any documentation (other than the
> wiki Design section and the code itself) on implementing database
> backends?
>
> thanks
>
> Luca
>
>

[Dovecot] log_path and info_log_path - ownership of parent directory

[Phil Howard]

The Ubuntu package didn't create the directory for these.  What
ownership/permissions are needed for it?  Does it need to be owned by the
dovecot user?

Re: [Dovecot] adding users to passwd-file

[Phil Howard]

Where can I find documentation on this command?  There is no "man dovecotpw"
installed, and searching for "dovecotpw" on the wiki gives 4 pages that
mention its existance.

Just executing the command with no options gives a password prompt that
cannot be broken out of (had to kill from another terminal).  The -h and -l
options give some info, but not enough to plan how to integrate this into
building the passdb/userdb.  For example how to reference the appropriate
file.

marconi/root/x0 /root 72# /usr/sbin/dovecotpw -h
> /usr/sbin/dovecotpw: invalid option -- 'h'
> usage: dovecotpw [-l] [-p plaintext] [-s scheme] [-u user] [-V]
> -lList known password schemes
> -p plaintext  New password
> -s scheme Password scheme
> -u user   Username (if scheme uses it)
> -VInternally verify the hash
> marconi/root/x0 /root 73# /usr/sbin/dovecotpw -l
> CRYPT MD5 MD5-CRYPT SHA SHA1 SHA256 SMD5 SSHA PLAIN CLEARTEXT CRAM-MD5
> HMAC-MD5 DIGEST-MD5 PLAIN-MD4 PLAIN-MD5 LDAP-MD5 LANMAN NTLM OTP SKEY RPA
> marconi/root/x0 /root 74#
>


On Mon, Apr 26, 2010 at 5:16 PM, Mike Abbott wrote:

> > Is there a tool equivalent to the system "passwd" command
>
> The dovecotpw command may be a good place to start.
>
>

[Dovecot] disable plaintext auth ... only for some addresses

[Phil Howard]

I'd like to disable plaintext authentication (e.g. only allow authentication
that does STARTTLS or connects on SSL/TLS only ports) only for certain
(most) IP addresses.  I want to exempt a few addresses (users coming over
known VPNs).

Fortunately, all this is coming in over a firewall (Sonicwall) in which I
can NAT traffic by IP address to go to specific port numbers.  So, if I can
establish a different disable_plaintext_auth policy by port number (for
extra port numbers I'll choose later), that would let me accomplish this.

If I cannot do this, then my only alternative is making the SSL/TLS only
ports the only ones open to the internet, and use the non-SSL/TLS ports only
for the VPNs (with disable_plaintext_auth = no).  But I read somewhere that
this is discouraged.  What say ye?

[Dovecot] dovecotpw documentation

[Phil Howard]

Where can I find documentation on how to use the "dovecotpw" program that I
was referred to?  No man page.  No wiki.  Very little from "dovecotpw -h".
What I need to do is generate the "passwd-file" contents with an MD5 scheme
(the one equivalent to /etc/shadow is preferred).

Re: [Dovecot] Requiring STARTTLS only on some networks

[Phil Howard]

On Fri, May 7, 2010 at 10:26, Pascal Volk <
user+dove...@localhost.localdomain.org
> wrote:

>
> By default "disable_plaintext_auth = yes" is set, in all Dovecot
> versions.
> With Dovecot v2.0 you can put something like the following in your
> dovecot.conf:
>
> remote 192.168.111.0/24 {
>  disable_plaintext_auth = no
> }
> remote fdfd::::0/48 {
>  disable_plaintext_auth = no
> }
>

Do these "remote sections" need to be in a specific place in the config
file, or can they just be put where the (global) disable_plaintext_auth is
located?

Do you know if the remote address gets passed from Postfix on to Dovecot
through the authentication connection (when Dovecot is doing the
authentication for Postfix mail submission) so that these same remote rules
apply?

[Dovecot] wrong config file?

[Phil Howard]

The sample provided config file (the one I started with by editing it)
included the following text:

# -- WARNING ---
>
> # If there's a file /etc/dovecot/dovecot-postfix.conf, which is part of
> # dovecot-postfix package, it will be used instead of dovecot.conf.
>
> # Keep in mind that, if that file exist, none of the changes in
> # /etc/dovecot/dovecot.conf will have effect on dovecot's configuration.
> # In that case you should customize /etc/dovecot/dovecot-postfix.conf.
>
> # -- WARNING ---
>

So I expected it would use that by default.  It seems to not be doing so.
Does this mean I have to modify the startup scripts to add the -c option to
force the use of this config file?  Or should I just put all my configs in
the default "/etc/dovecot/dovecot.conf" file?

When doing "dovecot -n" it became apparent that my configs were not being
used.  But "dovecot -c /etc/dovecot/dovecot-postfix.conf -n" was doing it.
Any idea why it's not behaving as those comments described?

Re: [Dovecot] Requiring STARTTLS only on some networks

[Phil Howard]

On Fri, May 7, 2010 at 11:07, Pascal Volk <
user+dove...@localhost.localdomain.org
> wrote:

> On 05/07/2010 04:35 PM Phil Howard wrote:
>
> > Do you know if the remote address gets passed from Postfix on to Dovecot
> > through the authentication connection (when Dovecot is doing the
> > authentication for Postfix mail submission) so that these same remote
> rules
> > apply?
>
> Hm, doesn't look so, as if Postfix would forward this info (remote host)
> to Dovecot. Even when I connect from a 'disable_plaintext_auth = no
> network' to Postfix (2.6.5), Postfix offers:
> 250-STARTTLS
> 250-AUTH DIGEST-MD5 CRAM-MD5
>
> But the SSL/TLS state should be forwarded from Postfix to Dovecot:
> http://www.mail-archive.com/postfix-us...@postfix.org/msg10590.html
>

Then I guess I will need to still run a separate always-SSL/TLS submission
port (e.g. 587).  I can easily restrict which ports can be reached by which
address ranges on the firewall.  But I can't (on the firewall) force use of
STARTTLS (which http://wiki.dovecot.org/SSL seems to be encouraging the use
of).

At least with IMAP and POP I can just use the one port (each ... 143 and
110) and force STARTTLS on untrusted addresses.

Re: [Dovecot] wrong config file?

[Phil Howard]

On Fri, May 7, 2010 at 11:43, Dennis Guhl  wrote:

> Ubuntu is using dovecot-postfix.conf as the working config file if you
> install the package 'dovecot-postfix' from the Ubuntu server team. But
> if you install the seperate packages 'dovecot-[common|imapd|pop3d]'
> dovecot.conf will be used.
>

This is what I have installed:

marconi/root/x0 /root 96# dpkg -l | egrep 'dovecot|postfix'
ii  dovecot-antispam  1.1+20090218.git.g28075fa-2   a
Dovecot plugin that helps train spam filte
ii  dovecot-common1:1.1.11-0ubuntu11
secure mail server that supports mbox and ma
ii  dovecot-dev   1:1.1.11-0ubuntu11
header files for the dovecot mail server
ii  dovecot-imapd 1:1.1.11-0ubuntu11
secure IMAP server that supports mbox and ma
ii  dovecot-pop3d 1:1.1.11-0ubuntu11
secure POP3 server that supports mbox and ma
ii  dovecot-postfix   1:1.1.11-0ubuntu11full
mail server stack provided by Ubuntu se
ii  postfix   2.6.5-3
High-performance mail transport agent
ii  postfix-cdb   2.6.5-3   CDB
map support for Postfix
ii  postfix-doc   2.6.5-3
Documentation for Postfix
ii  postfix-pcre  2.6.5-3   PCRE
map support for Postfix
ii  postfix-pgsql 2.6.5-3
PostgreSQL map support for Postfix
marconi/root/x0 /root 97#

So, dovecot-postfix is intended as an alternative for all the others?


> And it might be, that your master.cf does not call dovecot deliver
> with '-c /etc/dovecot/dovecot-postfix.conf' as part of argv=.
>

Thanks, I'll have to check that, too.  I did not know it also needed to be
in master.cf.  I have a reference in main.cf but it doesn't have args.

smtpd_sasl_type = dovecot
smtpd_sasl_path = private/dovecot-auth
mailbox_command = /usr/lib/dovecot/deliver

[Dovecot] CRYPT scheme and 8 character limit

[Phil Howard]

I've decided that having users supply cleartext passwords for me to encrypt
and encode is a bad idea, anyway.  So maybe I won't need dovecotpw.  The
idea is that users supply an already-encrypted password.  Most of the users
can fetch their login password from /etc/shadow on their own computer.

Wiki page http://wiki.dovecot.org/Authentication/PasswordSchemes indicates
that scheme name CRYPT is the one that uses the libc crypt() function.  But
it also says it uses no more than 8 characters.  Then there is MD5-CRYPT.
But that doesn't call crypt(), I presume.  No limit on password size is
indicated.  But this at least looks like SOME of the encrypted passwords I
have.

Other passwords I have have ids indicating SHA-256 and SHA-512.  Here is an
example from /etc/shadow that encrypted the clear password
"dovecotandpostfix" (in case it is necessary to test it):

$6$IwZzpjjj$p1VrkxQmgmTED8iQnQrV3sVEZpBmw2N8oD1ykOguXB5tf8aahICesX0TF6.VMThIW2QFs1buHjT3eDtnaAFhF1

The big question is, what scheme name should I use for these passwords?  I
can, of course, consider the id (6 in the above example, suggesting SHA-512)
if there is one (so far all are like this).  But the other consideration is
most of the passwords are longer than 8, some longer than 12, characters.
So apparently CRYPT can't be used in those cases (so I can't have the
automatic identification of crypt() to detect the scheme).

What scheme would I use for the various passwords?  What scheme would be
used for the above example for starters?  What if the id is 5 or 1 (the ones
documented for the crypt() call)?

Re: [Dovecot] CRYPT scheme and 8 character limit

[Phil Howard]

On Fri, May 7, 2010 at 15:11, Pascal Volk <
user+dove...@localhost.localdomain.org
> wrote:

> On 05/07/2010 08:16 PM Phil Howard wrote:
> > I've decided that having users supply cleartext passwords for me to
> encrypt
> > and encode is a bad idea, anyway.  So maybe I won't need dovecotpw.  The
> > idea is that users supply an already-encrypted password.  Most of the
> users
> > can fetch their login password from /etc/shadow on their own computer.
> >
> > Wiki page http://wiki.dovecot.org/Authentication/PasswordSchemesindicates
> > that scheme name CRYPT is the one that uses the libc crypt() function.
>  But
> > it also says it uses no more than 8 characters.  Then there is MD5-CRYPT.
> > But that doesn't call crypt(), I presume.  No limit on password size is
> > indicated.  But this at least looks like SOME of the encrypted passwords
> I
> > have.
> >
> > Other passwords I have have ids indicating SHA-256 and SHA-512.  Here is
> an
> > example from /etc/shadow that encrypted the clear password
> > "dovecotandpostfix" (in case it is necessary to test it):
> >
> >
> $6$IwZzpjjj$p1VrkxQmgmTED8iQnQrV3sVEZpBmw2N8oD1ykOguXB5tf8aahICesX0TF6.VMThIW2QFs1buHjT3eDtnaAFhF1
> >
> > The big question is, what scheme name should I use for these passwords?
>  I
> > can, of course, consider the id (6 in the above example, suggesting
> SHA-512)
> > if there is one (so far all are like this).  But the other consideration
> is
> > most of the passwords are longer than 8, some longer than 12, characters.
> > So apparently CRYPT can't be used in those cases (so I can't have the
> > automatic identification of crypt() to detect the scheme).
> >
> > What scheme would I use for the various passwords?  What scheme would be
> > used for the above example for starters?  What if the id is 5 or 1 (the
> ones
> > documented for the crypt() call)?
> >
>
> {CRYPT}, for SHA-256 and SHA-512. If your {,g}libc supports it, Dovecot
> can use it.
>

Is there another scheme name for it I can use that will still do those
without the 8 character limit?  I want the users to use their full passwords
even with this scheme (or at least one like it).  I know crypt() doesn't
have an 8 character limit, so a scheme that doesn't have such a limit, and
uses crypt(), I would think should work.

Re: [Dovecot] CRYPT scheme and 8 character limit

[Phil Howard]

On Fri, May 7, 2010 at 15:11, Pascal Volk <
user+dove...@localhost.localdomain.org
> wrote:

> On 05/07/2010 08:16 PM Phil Howard wrote:
> > I've decided that having users supply cleartext passwords for me to
> encrypt
> > and encode is a bad idea, anyway.  So maybe I won't need dovecotpw.  The
> > idea is that users supply an already-encrypted password.  Most of the
> users
> > can fetch their login password from /etc/shadow on their own computer.
> >
> > Wiki page http://wiki.dovecot.org/Authentication/PasswordSchemesindicates
> > that scheme name CRYPT is the one that uses the libc crypt() function.
>  But
> > it also says it uses no more than 8 characters.  Then there is MD5-CRYPT.
> > But that doesn't call crypt(), I presume.  No limit on password size is
> > indicated.  But this at least looks like SOME of the encrypted passwords
> I
> > have.
> >
> > Other passwords I have have ids indicating SHA-256 and SHA-512.  Here is
> an
> > example from /etc/shadow that encrypted the clear password
> > "dovecotandpostfix" (in case it is necessary to test it):
> >
> >
> $6$IwZzpjjj$p1VrkxQmgmTED8iQnQrV3sVEZpBmw2N8oD1ykOguXB5tf8aahICesX0TF6.VMThIW2QFs1buHjT3eDtnaAFhF1
> >
> > The big question is, what scheme name should I use for these passwords?
>  I
> > can, of course, consider the id (6 in the above example, suggesting
> SHA-512)
> > if there is one (so far all are like this).  But the other consideration
> is
> > most of the passwords are longer than 8, some longer than 12, characters.
> > So apparently CRYPT can't be used in those cases (so I can't have the
> > automatic identification of crypt() to detect the scheme).
> >
> > What scheme would I use for the various passwords?  What scheme would be
> > used for the above example for starters?  What if the id is 5 or 1 (the
> ones
> > documented for the crypt() call)?
> >
>
> {CRYPT}, for SHA-256 and SHA-512. If your {,g}libc supports it, Dovecot
> can use it.
>

OK, I looked in the source code and I cannot find any 8 character limit.  So
I'm now guessing that the documentation was really referring to the legacy
algorithm in crypt(), instead of the CRYPT scheme interface to it in
Dovecot.

Re: [Dovecot] wrong config file?

[Phil Howard]

On Fri, May 7, 2010 at 17:45, Dennis Guhl  wrote:

>  You can also give the full command here instead a transport name
> defined in the master.cf. And you append all parameters directly
> behind the command.
>

So I can do it either way?  I would prefer in main.cf if that is sufficient.

Re: [Dovecot] wrong config file?

[Phil Howard]

On Fri, May 7, 2010 at 11:43, Dennis Guhl  wrote:

> Ubuntu is using dovecot-postfix.conf as the working config file if you
> install the package 'dovecot-postfix' from the Ubuntu server team. But
> if you install the seperate packages 'dovecot-[common|imapd|pop3d]'
> dovecot.conf will be used.
>
> And it might be, that your master.cf does not call dovecot deliver
> with '-c /etc/dovecot/dovecot-postfix.conf' as part of argv=.
>

Apparently the config file selection is being done in /etc/init.d/dovecot so
it applies to the daemon and probably anything started from it, but not to
programs started elsewhere.  I'm guessing this special handling of Postfix
is distro specific, and not part of the basic Dovecot package.  But if they
are going to make such a change, they really should have made it fully
consistent and universal by changing the default file name string (and maybe
the logic to test between 2 different names) within the Dovecot programs.

That init file is testing if dovecot-postfix.conf exists, and falling back
to the default if not.

If that observation is correct, then I can simply be sure of which config
file I'm using by removing dovecot-postfix.conf and just using dovecot.conf
only.

[Dovecot] configuration user= or auth_user=

[Phil Howard]

The sample config file has:

  # User to use for the process. This user needs access to only user and
  # password databases, nothing else. Only shadow and pam authentication
  # requires roots, so use something else if possible. Note that passwd
  # authentication with BSDs internally accesses shadow files, which also
  # requires roots. Note that this user is NOT used to access mails.
  # That user is specified by userdb above.
  user = root

  # Directory where to chroot the process. Most authentication backends
don't
  # work if this is set, and there's no point chrooting if auth_user is
root.
  # Note that valid_chroot_dirs isn't needed to use this setting.
  #chroot =

So is the variable for the authentication process user supposed to be "user"
or "auth_user"?  It would make more sense to be "auth_user".  Is it just
"user" for legacy reasons, and the reference to "auth_user" is in error?  Or
can either name be used?

Also, those of you who change this, do you change it to something existing
like "dovecot" or "vmail" or "nobody" or do you make a whole new user for
it?

[Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

I'm getting this ...

May 10 12:45:01 eth0 postfix/local[3416]: A788D685F7: to=<
x...@.net>, relay=local, delay=13, delays=13/0/0/0.03,
dsn=4.3.0, status=deferred (temporary failure. Command output: Can't open
log file /var/log/dovecot/error.log: Permission denied )

So I'm looking at http://wiki.dovecot.org/LDA to see what I can do about
it.  I would have hoped some comments in the example config file would have
mentioned this and suggested a default practice solution.  Is this something
few people or many people encounter (among those using dovecot/deliver from
Postfix)?

If I do as the wiki describes and make separate log files for
dovecot/deliver to use, do I just make them owned by Postfix?  It would seem
to me a different directory might be more helpful (absent log files could be
created by having the directory permissions.  I'm also thinking in terms of
log rotations, too ... and I prefer to do log rotations by date stamping
rather than pushing sequence numbers (the legacy log rotation).

Any chance there are %-style variables I can use in log file/path names to
make log files automatically named by the date (and maybe time) ... kind of
like in a shell script I would do: `date +/path/to/tree/%Y/%m/%d.log` or
such.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

Just realized my email was not going to the list.

On Mon, May 10, 2010 at 14:20, Romer Ventura  wrote:

> I am using static uids:
> mail_uid = vmail
> mail_gid = vmail
> user = vmail
> group = vmail
>
> else it will do what you describe.
>

I have that, too.  But it's not running the right userid.  Deliver is
running as the userid Postfix starts it as.  How could it be any different
since deliver is not suid root (nor should it be, afaik).  It seems that I
need to tell Postfix a specific userid to run it as (and tell it that userid
is vmail).  I haven't found how to do that, yet.

I'm also getting wrong mail_location.  The variable %d comes up empty.  I
verified that Postfix actually is passing the full u...@domain, in the
message header, and in the -a argument (as coded in main.cf mainbox_command
=).

Maybe I need to make  /usr/lib/dovecot/deliver be suid vmail?  That would
open it up to logged in system users injecting into mailboxes.



> Thanks
> --
> Romer Ventura
>
>  On Mon, May 10, 2010 at 13:59, Romer Ventura  wrote:
>> I had to chmod 777 for it to work..
>>
>> I did chmod 777 to see what it would do, and especially, what userid the
>> log files were owned by.  Bad news from that ... they are owned by the first
>> user I sent email to.  That seems to me to be a Postfix issue where Postfix
>> still thinks I mailing to local system users, and running the deliver
>> program under such a userid.  When I start adding users which don't have
>> local system user equivalents, that's going to be a problem
>>
>> Also, I'm finding that in mail_location = the variable %d is empty.  It
>> should be the domain.  Again, this seems like Postfix is treating local
>> delivery as all-users-are-equivalent for any local domain (and that is
>> definitely not the case).  So I need to look at some Postfix config now to
>> see how to make it pass the full email address (u...@domain ... so %...@%d
>> represents the email address), and to run dovecot/deliver as user vmail.
>>
>> At least I'm not using sendmail :-)
>>
>> This old legacy "system user" thing is sure a PITA.  It should either be
>> ON or OFF.
>>  log files automatically named by the date (and maybe time) ... kind of
>> like in a shell script I would do: `date +/path/to/tree/%Y/%m/%d.log` or
>> such.
>>
>
>

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 14:42, Charles Marcus wrote:

> On 2010-05-10 1:59 PM, Romer Ventura wrote:
> > I had to chmod 777 for it to work..
>
> That's pretty much *never* a reasonable solution.
>

Absolutely right!

But it's an interim test ... in this case to see what userid the created
file would be owned by regardless of the creator's credentials.  What I
discovered is, it is not "vmail".  It is the system user that is the same as
the username part of the email address (and that is not supposed to be the
case).  Postfix is choosing the wrong userid to run dovecot/deliver as.  It
should be "vmail".  But I can't find a way to tell it that (so of course
it's going to do whatever its default is).  If it can setuid/seteuid to some
arbitrary user as it is doing now, then it must be running as root at that
point.  So it should be able to just as easily setuid/seteuid to "vmail" ...
if I can just find a way to tell it that.  More Postfix doc reading to do.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 14:44, Romer Ventura  wrote:

> What about your postfix conf..?
> mine is set to:
> virtual_gid_maps = static:1001
> virtual_mailbox_base = /srv/mail/vmail/
> virtual_mailbox_domains = $mydomain
> virtual_mailbox_maps = ldap:/etc/postfix/ldap_users.cf
> virtual_transport = dovecot
> virtual_uid_maps = static:1001
>

I saw conflicts in the docs for some of that and what I was doing.  What is
uid 1001 on yours? vmail?

What does Postfix do with virtual_mailbox_base ... or why should it care if
it is passing all deliverables to dovecot/deliver.  I'm not using LDAP, so
that's out.  Auth is via Dovecot, and a test to a non-existant user was
rejected as expected, so it seems the userdb lookup worked.  What does
"virtual_transport = dovecot" mean that ...

mailbox_command = /usr/lib/dovecot/deliver -c
/etc/dovecot/dovecot-postfix.conf -a "${RECIPIENT}"

... does not?

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 15:07, Romer Ventura  wrote:

> I dont know what else.
>
> I tried to chwon postfix:postfix, vmail:vmail, postfix:vmail, vmail:postfix
> and none of them worked. I had to go with chmod 777
>

I believe that is because Postfix is running dovecot/deliver as username
derived from the email address.  Are all your users in your /etc/passwd
file?

[Dovecot] %d is empty in mail_location

[Phil Howard]

I have this in dovecot-postfix.conf:

mail_location =
maildir:/home/mail/dnamesum=%12MLd/dname=%Ld/unamesum=%12MLn/uname=%Ln/mail

Yes, it is excessive, but that's just for testing.  The pattern I really
want is less clear for debugging.  In postfix/main.cf I have:

mailbox_command = /usr/lib/dovecot/deliver -c
/etc/dovecot/dovecot-postfix.conf -a "${RECIPIENT}"

I verified through strace that -a "${RECIPIENT}" is in fact getting a full
u...@domain address.  The problem is that %d and %Ld are coming up as empty,
and %12MLd is giving me the first 12 hex characters of an md5 of an empty
content.  It's losing the domain name somewhere.  It's in the mail headers
and in the -a option.  So what else is needed?

Re: [Dovecot] dovecot/deliver ... Can't open logfile /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 15:15, Egbert Jan  wrote:

> Could it be that selinux or apparmor are playing games with you???
>

Good idea to check.  But I don't have anything in my apparmor setup for
either postfix or dovecot.  I don't know about Romer Ventura's config.  I do
know the error trying to open the log files that I got is because the userid
Postfix ran dovecot deliver as was not allowed to open files there (before I
did chmod 777).  So it's definitely an issue of what userid dovecot/deliver
is running as (it runs as the LHS of receipient, not "vmail" as preferred).

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 15:25, Jerry  wrote:

>  From my 'master.cf' file:
>
> dovecot   unix  -   n   n   -   -   pipe
>  flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/deliver
> -f ${sender} -d ${us...@${nexthop}
>
> From 'main.cf' file: (snippet)
>
> virtual_gid_maps = static:1002
> virtual_minimum_uid = 100
> virtual_transport = dovecot
> virtual_uid_maps = static:1002
>
> Obviously, I have that user:
>
> pw showuser 1002
> vmail:*:1002:1002::0:0:Virtual Mail User:/nonexistent:/usr/sbin/nologin
>
> My log file is has 0600 permissions and its owner/group is 'vmail'.
>

So what does this make Postfix do?  Run one instance of dovecot/deliver and
pipe email to it?  Maybe that is the right solution and running it via
mailbox_command is wrong?

So what is virtual_minimum_uid doing for you if virtual_uid_maps is static?
Or why are any of these even relevant if everything is being piped to a
process started via master.cf?

And (problem I posted in a separate thread) does %d get assigned correctly
with the domain name for mail_location = if this method of running
dovecot/deliver is used?

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 15:58, Jerry  wrote:

>
> See: http://wiki.dovecot.org/LDA/Postfix
>
> Be sure to read the entire page.
>

I have a few times.  But now I'm getting a bit of a different perspective on
part of it.  The parameters are:

-d : Destination username. If given, the user information is
looked up from dovecot-auth. Typically used with virtual users, but not
necessarily with system users.
-a : Destination address (e.g. user+...@domain). Default is the
same as username. (v1.1+ only)

Well, that was actually confusing. I was passing the address via -a instead
of -d because -d was described as username.  That, and I know that the first
cases of "virtual users" (in sendmail and earlier postfix) was actually just
a twisted variant of system users, where the left hand side of @ was used
alone, and it didn't support distinct domains (e.g. b...@example.com and
b...@example.net were both just bob ... even if not the same as bob in
/etc/passwd).  And that's why I didn't use -d because in my case, I do have
different domains, where f...@example.com and f...@example.net are different
people.  So they are separate mailboxes and separate IMAP and submit
logins.  Oh, and their passwords may be different, too :-)

It's easy to continue to tie in virtual users to system users when
uniqueness is only on the LHS.  So if je...@example.com and
je...@example.net are the same user, and likewise for all users, then
storing the password in /etc/passwd or /etc/shadow suffices (for those not
wanting to use LDAP, SQL, etc).  But when the users need to be different
across different domains, even though the LHS is the same, now we have
issues with connecting them to system users.  And I have seen people map
usern...@domainname to someothername to lookup in /etc/passwd (that would be
a nightmare) or just put usern...@domainname in /etc/passwd (not sure how
well that would work).

But there is more than one semantic for "virtual users".  I believe I have
seen at least four.  In my case it will be unrelated to system users in
/etc/passwd or the setuid() or seteuid() calls.  Security will depend on the
mail application codes, not the underlying OS, to keep one user out of
another's mailbox (or sieve scripts,etc).


> > So what is virtual_minimum_uid doing for you if virtual_uid_maps is
> > static? Or why are any of these even relevant if everything is being
> > piped to a process started via master.cf?
>
> Not really sure. I was told it has something to do with Postfix itself.
>

The description of virtual_minumum_uid seemed to suggest that it was a bound
applied to what you get from virtual_uid_maps in case something was bad in
the map.


> > And (problem I posted in a separate thread) does %d get assigned
> > correctly with the domain name for mail_location = if this method of
> > running dovecot/deliver is used?
>
> You can either try it or perhaps ask on the Postfix forum.
>

Maybe it's related to -d vs -a in dovecot/deliver.  Postfix was sending the
full u...@domain to dovecot/deliver, and the %d should have been filled in
from that by dovecot/deliver.  But I was using -a and that may be wrong.
I'll try with -d instead.  Now I get a new error I didn't get before:

Error: Can't connect to auth server at /var/run/dovecot//auth-master:
Permission denied

It's not really clear how it is that worked before.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 15:25, Jerry  wrote:

> From my 'master.cf' file:
>
> dovecot   unix  -   n   n   -   -   pipe
>  flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/deliver
> -f ${sender} -d ${us...@${nexthop}
>
> From 'main.cf' file: (snippet)
>
> virtual_gid_maps = static:1002
> virtual_minimum_uid = 100
> virtual_transport = dovecot
> virtual_uid_maps = static:1002
>

I want to give this approach a try.  But I can't find anything in the docs
on what ${nexthop} means.  I'm not doing any hopping.  I don't know what to
put in here.  Is this just the domain?  Would that be ${domain} that I need
to use?

I just gave it a shot.  Nothing is happening.  Postfix comes up.  Email into
port 25 goes in.  But nothing shows up in a mailbox and the log files are
not created.  It's as if deliver doesn't even get run.

Oops ... just found that the mail is showing up in /var/mail/${USER} ...
totally wrong place like Postfix is ignoring this and not running
dovecot/deliver at all.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 17:11, Romer Ventura  wrote:

> try using -d ${recipient}, but change the format of the username in
> dovecot.conf
>

What does "change the format of the username" mean?



> What i did was to set the mail attribute for each user in AD, then perform
> a query for it and have dovecot group users by domain, this way i can have
> us...@example.net and us...@example.com


Sorry, now I'm just not following this at all.  I don't know what mail
attribute apply here, and I don't know what "in AD" means.

I have the following in in my dovecot-postfix.conf file:

mail_location =
maildir:/home/mail/dnamesum=%12MLd/dname=%Ld/unamesum=%12MLn/uname=%Ln/mail

And this was working until I switched to the "virtual_transport = dovecot"
method ... although %d was coming up empty (and %12MLd was the md5 of
empty).  Ultimately my intention is to have:

mail_location = maildir:/home/mail/%2MLd/%Ld/%2MLn/%Ln/mail

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 17:23, Romer Ventura  wrote:

> man pipe
>
>  ${nexthop}
> This macro expands to the next-hop hostname.
>
> This information is modified by the h flag for case
> folding.
>

But what is next hop?  I don't have any next hop that I'm aware of.  These
are local domains being kept distinct.

Re: [Dovecot] %d is empty in mail_location

[Phil Howard]

On Tue, May 11, 2010 at 04:00, Steffen Kaiser  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On Mon, 10 May 2010, Phil Howard wrote:
>
>  u...@domain address.  The problem is that %d and %Ld are coming up as
>> empty,
>> and %12MLd is giving me the first 12 hex characters of an md5 of an empty
>> content.  It's losing the domain name somewhere.  It's in the mail headers
>> and in the -a option.  So what else is needed?
>>
>
> Do you re-write the "user" attribute in the passdb?
>

I do whatever this means:

auth default {
  mechanisms = plain login

  passdb passwd-file {
args = username_format=...@%ld /etc/mailauth/ALL.deny
deny = yes
  }
  passdb passwd-file {
args = username_format=%Ln /etc/mailauth/%Ld.deny
deny = yes
  }
  passdb passwd-file {
args = scheme=crypt username_format=...@%ld
/etc/mailauth/ALL.passwd
  }
  passdb passwd-file {
args = scheme=crypt username_format=%Ln /etc/mailauth/%Ld.passwd
  }
  userdb passwd-file {
args = username_format=...@%ld /etc/mailauth/ALL.passwd
  }
  userdb passwd-file {
args = username_format=%Ln /etc/mailauth/%Ld.passwd
  }

The intention of the above is that these  passwd-file format files have only
the username part of the full email address being logged in as, and a
separate file be there for each domain.  So if I login to IMAP as
p...@example.com, then I would be authenticated by accessing file
"/etc/mailauth/example.com.passwd" and searching for user "phil" in that
file.

I would not expect the formatting of what username I search for in these
files to cause the %d variable to lose its content.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Mon, May 10, 2010 at 18:16, Jerry  wrote:

> Please post the output of "dovecot -n" and "postconf -n". Better,
> provide output from the postfinger tool. This can be found at
> http://ftp.wl0.org/SOURCES/postfinger.
>

I have redacted external IP addresses and domain names.

from dovecot -n:

# 1.1.11: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.31-19-server x86_64 Ubuntu 9.10
base_dir: /var/run/dovecot/
log_path: /var/log/dovecot/error.log
info_log_path: /var/log/dovecot/info.log
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap pop3 imaps pop3s
listen: 172.30.0.24, [fc00::18], [${MYIPV6}::18], 127.0.0.1, [::1]
ssl_cert_file: /etc/ssl/certs/ssl-mail.pem
ssl_key_file: /etc/ssl/private/ssl-mail.key
ssl_parameters_regenerate: 24
ssl_cipher_list:
ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM
disable_plaintext_auth: no
login_dir: /var/run/dovecot//login
login_executable(default): /usr/lib/dovecot/imap-login
login_executable(imap): /usr/lib/dovecot/imap-login
login_executable(pop3): /usr/lib/dovecot/pop3-login
login_greeting: AUTHORIZED USERS ONLY -- unauthorized access strictly
prohibited
login_greeting_capability(default): yes
login_greeting_capability(imap): yes
login_greeting_capability(pop3): no
mail_chroot: /home/mail
mail_max_userip_connections(default): 10
mail_max_userip_connections(imap): 10
mail_max_userip_connections(pop3): 3
verbose_proctitle: yes
mail_privileged_group: mail
mail_uid: vmail
mail_gid: vmail
mail_location:
maildir:/home/mail/dnamesum=%12MLd/dname=%Ld/unamesum=%12MLn/uname=%Ln/mail
mail_executable(default): /usr/lib/dovecot/imap
mail_executable(imap): /usr/lib/dovecot/imap
mail_executable(pop3): /usr/lib/dovecot/pop3
mail_process_size: 768
mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
imap_client_workarounds(default): outlook-idle delay-newmail
imap_client_workarounds(imap): outlook-idle delay-newmail
imap_client_workarounds(pop3):
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
auth default:
  mechanisms: plain login
  username_format: %...@ld
  passdb:
driver: passwd-file
args: username_format=%Ln /etc/mailauth/%Ld.deny
deny: yes
  passdb:
driver: passwd-file
args: scheme=crypt username_format=%Ln /etc/mailauth/%Ld.passwd
  userdb:
driver: passwd-file
args: username_format=%Ln /etc/mailauth/%Ld.passwd
  socket:
type: listen
client:
  path: /var/spool/postfix/private/dovecot-auth
  mode: 432
  user: postfix
  group: postfix
master:
  path: /var/run/dovecot/auth-master
  mode: 384
  user: vmail
  group: vmail

from postconf -n:

command_directory = /usr/sbin
config_directory = /etc/postfix
daemon_directory = /usr/lib/postfix
data_directory = /var/lib/postfix
default_destination_concurrency_limit = 2
default_privs = nobody
in_flow_delay = 1s
inet_interfaces = 172.30.0.25
inet_protocols = ipv4, ipv6
local_destination_concurrency_limit = 2
mail_owner = postfix
mydestination = 17.DOMAIN.NAMES.REDACTED
mydomain = 1.DOMAIN.NAME.REDACTED
myhostname = mail.1.DOMAIN.NAME.REDACTED
mynetworks = ${MYIPV4}.80/28, 127.0.0.0/8, 172.16.0.0/16, 172.20.0.0/16,
172.30.0.0/16, [::1]/128, [fc00::]/48, [${MYIPV6}::]/48, [fe80::]/48
myorigin = $mydomain
proxy_interfaces = ${MYIPV4}.90
queue_directory = /var/spool/postfix
recipient_delimiter = -
relay_domains = $mydestination
smtp_bind_address = 172.30.0.25
smtp_bind_address6 = fc00::25, ${MYIPV6}::25
smtpd_banner = $myhostname ESMTP
smtpd_sasl_path = private/dovecot-auth
smtpd_sasl_type = dovecot
soft_bounce = yes
unknown_local_recipient_reject_code = 450
virtual_gid_maps = static:252
virtual_minimum_uid = 100
virtual_transport = dovecot
virtual_uid_maps = static:252

from postfinger:

postfinger - postfix configuration on Tue May 11 09:29:55 EDT 2010
version: 1.30

Warning: postfinger output may show private configuration information,
such as ip addresses and/or domain names which you do not want to show
to the public.  If this is the case it is your responsibility to modify
the output to hide this private information.  [Remove this warning with
the --nowarn option.]

--System Parameters--
mail_version = 2.6.5
hostname = marconi
uname = Linux marconi 2.6.31-19-server #56-Ubuntu SMP Thu Jan 28 03:40:48
UTC 2010 x86_64 GNU/Linux

--Packaging information--
looks like this postfix comes from deb package: postfix-2.6.5-3

--main.cf non-default parameters--
default_destination_concurrency_limit = 2
inet_interfaces = 172.30.0.25
inet_protocols = ipv4, ipv6
mydestination = 17.DOMAIN.NAMES.REDACTED
mydomain = 1.DOMAIN.NAME.REDACTED
myhostname = mail.1.DOMAIN.NAME.REDACTED
mynetworks = ${MYIPV4}.80/28, 127.0.0.0/8, 172.16.0.0/16, 172.20.0.0/16,
172.30.0.0/16, [::1]/128, [fc00::]/48, [${MYIPV6}::]/48, [fe80::]/48
myorigin = $mydomain
proxy_interfaces = ${MYIPV4}.90
recipient_delimiter = -
s

Re: [Dovecot] %d is empty in mail_location

[Phil Howard]

So what I am wondering, right now, is where the value for %d ... as used in
mail_location ... comes from when running in dovecot/deliver.  Apparently it
is not getting anything through that means.  Not know where it gets it from,
I don't know where to look to see what is misconfigured.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Tue, May 11, 2010 at 12:59, Gerard Seibert wrote:

> I have to admit that I am somewhat confused. You have "postfix" listed
> as user/group in the dovecot.conf file, yet you have "vmail" listed as
> the user in 'master.cf". That doesn't look right.
>

I'm not sure which way things are supposed to be, and I've been trying
changes, some of which worked (which confirmed I didn't understand it to
begin with).  There are too many different usernames (although I can
understand the need to have a certain number of distinct isolation users) to
keep straight.  Which ones go where.  There being a lack of a thorough
document to configure BOTH postfix and dovecot together makes this harder.


> Do you actually have a user with uid 252?
>

dovecot:x:250:250:Dovecot mail server,,,:/usr/lib/dovecot:/bin/false
postfix:x:251:251::/var/spool/postfix:/bin/false
vmail:x:252:252::/tmp:/bin/false


I am assuming that you are not using "mydestination =
> 17.DOMAIN.NAMES.REDACTED" as a virtual delivery address. Virtual
> domains must not be listed in "my destination". I could not find a
> "virtual_mailbox_domains"
>
> virtual_mailbox_domains ($virtual_mailbox_maps)
>  Postfix is final destination for the specified list
>  of   domains;  mail  is  delivered  via  the  $vir-
>  tual_transport mail delivery transport.
>

All 17 (and more to come) domains are destined for this MACHINE.  And I have
gotten email delivered through Dovecot already, which was addressed to one
of these machines.

I do think Postfix has always had a somewhat different notion of what
"virtual user" is than what I'm doing.  If it does make the distinction
between Postfix itself as a destination (defined by "mydestination") and
Dovecot as a different destination, then the obvious question now is, what
do I assign the list of domains (which includes the domain this machine's
hostname is in) that go to Dovecot (which, BTW, for now, is all of them)?
For temporary reference for discussion purpose, until someone tells me the
real variable name to use, I'll call it "dovecotdestination =".  So what is
that real variable name?

I've done virtual users in Postfix before ... this way.  And this way is the
only one that worked.  What was described as virtual users in Postfix docs
did not work.  When I discussed this with Postfix people on the list a long
time ago when I set that up (elsewhere), they said that it was not really
"virtual users" in the sense that Postfix thinks of virtual users.  I had
"local" delivery going by other means.  For THIS setup, I am trying to make
local deliver go to Dovecot.  And that is successful.  And the deliver
program is getting what I think it should be getting (the domain name is in
the message headers and on the command line).


Personally, I think that you should take this up on the Postfix forum.
> Post the output of the "postfinger" and "dovecot -n" and I think that
> you will be able to get your problem solved. It looks to me like you
> have "virtual" configured incorrectly. It certainly is not configured
> like I have it on my system. I use purely "vitual" users and have
> "mydestination = " in main.cf on my system.
>

I'm not seeing how this is a Postfix issue, yet.  The mail is getting to
Dovecot, and it includes the domain name in the addresses.  But Dovecot is
not filling in the %d variable.  I don't see how that is a Postfix issue.

FYI, I am on the Postfix list, already.  Maybe what is needed is a list just
for combination users, those using Dovecot and Postfix together?

Jerry
> dovecot.u...@seibercom.net
>

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Tue, May 11, 2010 at 12:59, Gerard Seibert wrote:

>  Post back if you get this fixed.
>

Bsically, what I need to know from THIS list is exactly what conditions the
dovecot/deliver program needs in order to properly fill in the %d variable
for mail_location.  Once I know that ... know exactly where the value for %d
comes from (e.g. does it come from parsed headers ... or from a specific
command line option), then I'll know how to make Postfix do it ... or know
what question to ask on the Postfix list.  As of now, I don't know what to
ask them.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Tue, May 11, 2010 at 12:59, Gerard Seibert wrote:

>  Virtual documentaion: http://www.postfix.org/virtual.8.html
>

This seems to be a delivery agent of its own.  I don't want Postfix to do
the delivery.  I want Dovecot to do the delivery so it can create the
additional cache/index files (whatever they were ... Dovecot documentation
encourages this).  So that means handing it off to the
/usr/lib/dovecot/deliver program.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Tue, May 11, 2010 at 14:38, Bradley Giesbrecht <
bradley.giesbre...@gmail.com> wrote:

>
> On May 11, 2010, at 11:26 AM, Phil Howard wrote:
>
>  On Tue, May 11, 2010 at 12:59, Gerard Seibert > >wrote:
>>
>>  Virtual documentaion: http://www.postfix.org/virtual.8.html
>>>
>>>
>> This seems to be a delivery agent of its own.  I don't want Postfix to do
>> the delivery.  I want Dovecot to do the delivery so it can create the
>> additional cache/index files (whatever they were ... Dovecot documentation
>> encourages this).  So that means handing it off to the
>> /usr/lib/dovecot/deliver program.
>>
>
> Basically postfix just needs to know that a username/email address is local
> and how to deliver.
>

And it did seem to do that already.  Mail was sent to dovecot/deliver.  It
included the domain name.  But deliver just didn't construct the
mail_location correctly due to %d being empty.  The resulting path with the
empty space where the domain name should have been was used to actually
deliver the mail.  I read that file and the domain name was also in the
headers.  The domain was there, but %d didn't get it.



> If you are using virtual users in main.cf this works for me.
> virtual_transport= dovecot


> In master.cf this works for me.
>
> dovecot  unix-nn--pipe
>flags=DRhu user=_vmail:_vmail argv=/opt/local/libexec/dovecot/deliver -d
> ${recipient}
>

I tried it, but effectively, nothing happened.  Maybe the other virtual_*
stuff also needs to be configured.  I've used that virtual_* stuff before
many years ago without success.  At the time, from what I remember, the
concept of "virtual" the way they were using it just wasn't the same as my
idea of "virtual".  The way I read the Dovecot docs, virtual for Dovecot
seemed to be the same.  Now I don't know.  I do know I have run across at
least 4 different concepts called "virtual email users".


Now dovecot needs to know where to deliver to. I use a database backend so
> postfix and dovecot can look the information up in the same place. They just
> need queries to return the values they require. In dovecot docs look for
> userdb and passworddb.
>

I'm using "passwd-file" to authenticate, and mail_location = to compose a
pattern of where each maildir will be found.  I won't be using a backend
database (that's the last thing I want to do).



> 777 you log file till you figure out which users need to write to it. I
> have _vmail as my user and group name and my dovecot-deliver.log has
> _vmail:_vmail for owner and group and dovecot.log has root:_vmail for owner
> and group.
>

I got the log file working.  I had to tell Postfix to run dovecot/deliver as
user:group vmail:vmail and that did it.  It WAS running dovecot/deliver as
some user whose name just happened to match (even though the mail didn't
belong to the person who had that system account).

I'm looking over the Postfix virtual_* stuff again.  Maybe there's new stuff
since I last did Postfix about 6 years ago or so.

Summary of what I want to accomplish:

There are many domains and many users in each domain.  Where the user part
of a domain happens to be the same as the user part of another domain, that
is NOT to be considered the same mainbox at all.  All mail to a set of
domains (currently all the domains) is to be delivered to maildir format
mailboxes via Dovecot.  IMAP users will login as "u...@domain".  The
userdb/authdb is in passwd-file format, with different files for each
domain, and user names w/o domain being the index.  If necessary, I can
change the format of that to one big passwd-file format with
u...@domainindex.  The mailboxes will be located in
/home/mail/XX/domain/YY/user/home/mail where XX will be 2 hex digits from
the MD5 of the domain, and YY will be 2 hex digits from the MD5 of the
username.  The part of the path before the final "mail" directory is the
"home" for the "user", and the last "mail" subdirectory is in maildir
format.  Domains and users are to be translated to lower case before
composing that path and before taking theur MD5 hash for XX and YY.  The
delivery into that path is to be done by Dovecot's deliver program so it
builds whatever indexes and stuff are needed to make IMAP access faster.
Also running POP3 is a plus, but not required.  The same userdb/passdb is to
also be used for submission of mail, via Dovecot's SASL support exported
back to Postfix.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Tue, May 11, 2010 at 19:25, Noel Butler  wrote:

>  > And it did seem to do that already.  Mail was sent to dovecot/deliver.
>  It
> > included the domain name.  But deliver just didn't construct the
> > mail_location correctly due to %d being empty.  The resulting path with
> the
> > empty space where the domain name should have been was used to actually
> > deliver the mail.  I read that file and the domain name was also in the
> > headers.  The domain was there, but %d didn't get it.
> >
> >
>
> interesting...
>
> %d is derived from the right hand side of a username, dovecot's deliver
> couldn't care less about verifying the domain, since that is the MTA's
> job.
>

No doubt.  However, the big question is WHICH particular instance of
u...@domain does it derive domain from?  There is more than one inside the
headers.  There are also options -a and -d and maybe it gets it from ONE of
those.  Or maybe it looks around more than once source for an address to
derive the RHS from.  I doubt it would verify any more so than whether it
should deliver.  But it did deliver, so clearly it believed it could.  That
big question can also be in the form of "where should the domain be provided
that it so far was not provided in?".


> I tried it, but effectively, nothing happened.  Maybe the other virtual_*
> > stuff also needs to be configured.  I've used that virtual_* stuff before
>
>
>
> it certainly does
>

That's a different mode of operation of Postfix that I have had troubles
with in the past.  The big one I remember having (of more than one) was that
it treated all the domains as equivalent.  That is, b...@example.com and
b...@example.net were the same.  OTOH, that may have been due to mishandling
of, or by, the NON-Dovecot delivery agent I was using back then.

So I'll try this with Dovecot deliver.  Been out of the office for a couple
days, so I hope I'll have some time today to give it a shot.



>  > I'm using "passwd-file" to authenticate, and mail_location = to compose
> a
> > pattern of where each maildir will be found.  I won't be using a backend
> > database (that's the last thing I want to do).
> >
>
>
> why not? it simplifies virtual users, you're trying to use a method
> primarily designed for system accounts, as demonstrated over the past
> several days you are only giving yourself pain for no reason.
>

I don't see how one database lookup method vs. another database lookup
method has anything to do with whether email users are virtual or not.  The
actual DATA that comes back from the lookup might.  But the method itself
should be transparent to the mail delivery decisions.  In another thread,
CDB was asked for, for a future Dovecot.  How do you feel about CDB?  Does
using CDB make users virtual or system?

Re: [Dovecot] %d is empty in mail_location

[Phil Howard]

On Wed, May 12, 2010 at 04:20, Steffen Kaiser  wrote:

>
> actually it looks like, nobody uses passwd-file like you do :)
>

This aspect can be changed, if needed.  If needed, one big file with
u...@domain in the first colon-separated field would be doable, too.



> The doc at http://wiki.dovecot.org/AuthDatabase/PasswdFile would lead to
> your assumption of operation, that is:
>
> + you auth using u...@domain
> + look up the user in the domain-specific file without domain
> + but you don't overwrite "user", hence, further processing should have the
> domainin %d
>

Yes, the user will need to provide the full u...@domain they want to login
as.  Just user alone is insufficient in some cases, and even if user fred is
only in one domain today, it might also be in another domain, tomorrow.  So
all users will need to login with u...@domain.

The initial scheme was looking for user by itself in a passwd-file format
file identified with domain name.  But this can be changed to looking for
u...@domain (which seems pointless unless it is one big file).

Overwrite?

It looks like my understanding of %n and %u was swapped.  But that shouldn't
affect %d.  I can't imagine that by using %u it would cause %d to be empty
(as in "oh, domain is already used, can't use it again").

I'd be using CDB format if it were available.  So far, passwd-file is the
only plain file format Dovecot supports.  Postfix supports CDB and others,
but not passwd-file.  I wish there was common ground there.  That or the
Berkeley-DB format.

Yet another "format" I suggested many years ago is not supported anywhere
(except in some code I wrote unrelated to email).  That "format" is where
the key is a file name in a specified directory, and the contents of that
file would be the value yielded.  It's a wasteful format in filesystems like
ext2/3, but in reiserfs it works well as long as tail packing is not
disabled.


Could you verify that the domain gets stripped by setting mail_debug,
> auth_verbose and auth_debug?
>

Gonna work on that, today.


> This page http://wiki.dovecot.org/VirtualUsers catches my eyes:
>
> "In the above examples users are expected to log in as "u...@domain".
> Their mail is kept in their home directory at
> /home///Maildir.
>
> The usernames in the passwd and shadow files are expected to contain only
> the user part, no domain. This is because the path itself already contained
> %d to specify the domain. If you want the files to contain full 
> u...@domainnames, you can use %0.d instead of %d."
>
> This is exactly what you want, IMO.
>

I believe so.  But I'm still not so sure Postfix's idea of "virtual users"
is the same.  I need to try it both ways.  I do know in the past, virtual_*
in Postfix was treating all domains as one and didn't work right for what I
was doing (didn't involve Dovecot way back then).


>
> These pages also describes your idea:
>
> http://neranjara.org/article/title/How_to_configure_PostFix_and_Dovecot_for_Virtual_Users_with_out_a_Database_
> - -and-
>
> http://serverfault.com/questions/80590/how-do-i-persuade-dovecot-and-postfix-to-use-multiple-passwd-files-for-user-accou
>
>
They do seem to be using %d in the same kind of way I'm expecting to.  I'll
look these over to see if any differences in how other parts are configured
can make sure %d has the recipient's domain.

Re: [Dovecot] %d is empty in mail_location

[Phil Howard]

On Wed, May 12, 2010 at 04:20, Steffen Kaiser  wrote:

> Could you verify that the domain gets stripped by setting mail_debug,
> auth_verbose and auth_debug?
>

Where would I see the results of this?

Re: [Dovecot] dovecot 2.0 vs. SELinux

[Phil Howard]

On Fri, May 14, 2010 at 08:10, Michal Hlavinka  wrote:

> Does anyone know what happens here? Why dovecot tries to use regular file
> dovecot.conf as socket? There is probably some magic in it (from my pov)
> which
> I don't  understand.
>

I don't see that happening with mine.  That sure doesn't make sense.  So I
can only guess some mangled configuration.  But if it hasn't even read the
configuration, yet, even stranger.  What other actions on the config file
does strace show happened?  E.g. did it also open it for reading or not?
Before or after?

Re: [Dovecot] %d is empty in mail_location

[Phil Howard]

On Fri, May 14, 2010 at 12:47, Thomas Leuxner  wrote:

> Am 14.05.2010 um 17:05 schrieb Phil Howard:
>
> > On Wed, May 12, 2010 at 04:20, Steffen Kaiser <
> skdove...@smail.inf.fh-brs.de
> >> wrote:
> >
> >> Could you verify that the domain gets stripped by setting mail_debug,
> >> auth_verbose and auth_debug?
> >>
> >
> > Where would I see the results of this?
>
> In whatever logging facility you set up. It will simply raise verbosity.
>

Nothing new is showing up when I test sending some mail through (which does
get delivered to that path constructed with an empty string for %).

marconi/root/x1 /root 53# cd /var/log/dovecot
marconi/root/x1 /var/log/dovecot 54# ls -l
total 8
-rw--- 1 root root  120 2010-05-14 11:26 error.log
-rw--- 1 root root 1130 2010-05-14 11:22 info.log
marconi/root/x1 /var/log/dovecot 55# cat error.log
dovecot: 2010-05-14 11:21:39 Warning: Killed with signal 15
dovecot: 2010-05-14 11:26:52 Warning: Killed with signal 15
marconi/root/x1 /var/log/dovecot 56# cat info.log
dovecot: 2010-05-14 11:00:27 Info: dovecot v1.1.11 starting up (core dumps
disabled)
dovecot: 2010-05-14 11:00:28 Info: auth(default): new auth connection:
pid=3233
dovecot: 2010-05-14 11:00:28 Info: auth(default): new auth connection:
pid=3234
dovecot: 2010-05-14 11:00:28 Info: auth(default): new auth connection:
pid=3236
dovecot: 2010-05-14 11:00:28 Info: auth(default): new auth connection:
pid=3235
dovecot: 2010-05-14 11:00:28 Info: auth(default): new auth connection:
pid=3237
dovecot: 2010-05-14 11:00:28 Info: auth(default): new auth connection:
pid=3238
dovecot: 2010-05-14 11:22:17 Info: dovecot v1.1.11 starting up (core dumps
disabled)
dovecot: 2010-05-14 11:22:18 Info: auth(default): new auth connection:
pid=3328
dovecot: 2010-05-14 11:22:18 Info: auth(default): new auth connection:
pid=3327
dovecot: 2010-05-14 11:22:18 Info: auth(default): new auth connection:
pid=3329
dovecot: 2010-05-14 11:22:18 Info: auth(default): new auth connection:
pid=3326
dovecot: 2010-05-14 11:22:18 Info: auth(default): new auth connection:
pid=3330
dovecot: 2010-05-14 11:22:18 Info: auth(default): new auth connection:
pid=3331
marconi/root/x1 /var/log/dovecot 57#

Re: [Dovecot] %d is empty in mail_location

[Phil Howard]

On Fri, May 14, 2010 at 13:30, Thomas Leuxner  wrote:

> Am 14.05.2010 um 19:25 schrieb Phil Howard:
>
> > Nothing new is showing up when I test sending some mail through (which
> does get delivered to that path constructed with an empty string for %).
> >
>
> Which version are you now testing on? Recent posts were set around the
> beta. Is this for the 2.x beta or this rather old build you're showing here?
>

It is the version packaged in Ubuntu 9.10 ...

marconi/root/x0 /root 51# dpkg -l | egrep 'dovecot|postfix'
ii  dovecot-antispam  1.1+20090218.git.g28075fa-2   a
Dovecot plugin that helps train spam filte
ii  dovecot-common1:1.1.11-0ubuntu11
secure mail server that supports mbox and ma
ii  dovecot-dev   1:1.1.11-0ubuntu11
header files for the dovecot mail server
ii  dovecot-imapd 1:1.1.11-0ubuntu11
secure IMAP server that supports mbox and ma
ii  dovecot-pop3d 1:1.1.11-0ubuntu11
secure POP3 server that supports mbox and ma
ii  dovecot-postfix   1:1.1.11-0ubuntu11full
mail server stack provided by Ubuntu se
ii  postfix   2.6.5-3
High-performance mail transport agent
ii  postfix-cdb   2.6.5-3   CDB
map support for Postfix
ii  postfix-doc   2.6.5-3
Documentation for Postfix
ii  postfix-pcre  2.6.5-3   PCRE
map support for Postfix
ii  postfix-pgsql 2.6.5-3
PostgreSQL map support for Postfix
marconi/root/x0 /root 52#

Re: [Dovecot] %d is empty in mail_location

[Phil Howard]

On Fri, May 14, 2010 at 14:00, Thomas Leuxner  wrote:

> Am 14.05.2010 um 19:51 schrieb Phil Howard:
>
> > It is the version packaged in Ubuntu 9.10 …
>
> This one is ancient. Also it has configuration syntax changes compared to
> the latest production release. Anyway the debug parameters should work,
> given they are in the right conf file...
>

I've been thinking that as soon as I get this working, I will plan a
transition over to a newer server with Slackware instead of Ubuntu.
Slackware is easier about letting you run packages you build from source.

Re: [Dovecot] %d is empty in mail_location

[Phil Howard]

On Fri, May 14, 2010 at 14:06, Thomas Leuxner  wrote:

> Am 14.05.2010 um 20:03 schrieb Phil Howard:
>
> > I've been thinking that as soon as I get this working, I will plan a
> transition over to a newer server with Slackware instead of Ubuntu.
>  Slackware is easier about letting you run packages you build from source.
>
> Probably OT, but did you consider using Stephan Bosch's builds for that?
>
> http://xi.rename-it.nl/debian/pool/testing-auto/dovecot-1.2/
>
> No, I haven't.  Due to some management policy, that might not even be an
option.  I've got a couple other ideas for config changes on 1.11 that I
still need to try.

[Dovecot] [SOLVED] %d is empty in mail_location

[Phil Howard]

On Mon, May 10, 2010 at 15:21, Phil Howard  wrote:

> I have this in dovecot-postfix.conf:
>
> mail_location =
> maildir:/home/mail/dnamesum=%12MLd/dname=%Ld/unamesum=%12MLn/uname=%Ln/mail
>
> Yes, it is excessive, but that's just for testing.  The pattern I really
> want is less clear for debugging.  In postfix/main.cf I have:
>
> mailbox_command = /usr/lib/dovecot/deliver -c
> /etc/dovecot/dovecot-postfix.conf -a "${RECIPIENT}"
>
> I verified through strace that -a "${RECIPIENT}" is in fact getting a full
> u...@domain address.  The problem is that %d and %Ld are coming up as
> empty, and %12MLd is giving me the first 12 hex characters of an md5 of an
> empty content.  It's losing the domain name somewhere.  It's in the mail
> headers and in the -a option.  So what else is needed?
>

There was a typo in an earlier config item:

auth_username_format = %...@ld

Maybe that was what someone was screaming at me about when I looked at
username_format.  I didn't see this until I was trying to figure out why the
passwd file wasn't being read.  I ran strace, and saw that it was trying to
access a file with "ld" in it.  Even then it took a while because the domain
had been reformed as "Ld" and subsequently lower cased later.

I found this while trying out a subset of a couple suggested configurations
with some virtual_* settings in Postfix, and using dovecot/deliver via
transport.

I will now clean up the mess and hope I don't break it doing that.  Then
slam it with email and see what happens next.

Re: [Dovecot] [SOLVED] %d is empty in mail_location

[Phil Howard]

On Fri, May 14, 2010 at 15:51, Alex Baule  wrote:

> Add in your auth,conf configuration:
>
> auth_default_realm = [your domain]
>

Which domain goes there?  I have many.

Re: [Dovecot] dovecot/deliver ... Can't open log file /var/log/dovecot/error.log: Permission denied

[Phil Howard]

On Fri, May 14, 2010 at 17:48, Noel Butler  wrote:

> CDB, oh dear god, you want to go back in time?
> CDB is no better than any other flatfile based system, it was horrible
> with qmail and it'll be horrible with anything else above a couple
> thousand users, you clearly dont add/del users all the time, rebuilding
> its DB can take some time (I've seen some take 3 minutes, tuff luck if
> your clients want to add a few users,... so using that is something you
> cant afford to do as a SP.
>

I see no problem with CDB.  I designed my own variation of that a couple
decades ago.  Mine isn't quite as fast as CDB, but it was along the basic
idea.  It was very useful in its time.  Had CDB not come along, I'd probably
have tried to add mine into things like Postfix and Dovecot at some point.



> MySQL makes it such a dream, even with customers adding aliases and so
> on, its a simple instruction to mysql via the web portal from them, and
> using replication means every front end has its own local copy, and able
> to fallback to the master if for some reason it becomes  unavailable
> (never seen that in all the years been using it tho, but its nice
> insurance)
>

MySQL (or PostgreSQL, etc) has its place.  And for things like CRM with a
lot of different aspects, that is the way to go.  But even then, I would
(and have for other purposes) just export the data out of the SQL database
and build a fast index like CDB.  However long it takes to build CDB is NOT
downtime; it's just lag from data entry to activation.  And there are ways
to work around that if the lag is an issue, such as having a CDB first,
followed by another lookup that may have the most recent data.  For example,
when the list of new users arrives, add them to a Berkeley DB that is
queried next when the CDB has no match.  Thus they work even while the CDB
(think of it has a static cache) is being rebuilt.  One thing I would NOT do
is have mail servers hitting the CRM database (or its replicas) directly.
It's not a performance issue; it's a security issue.  The larger the
operation, the more important this is.


its your network (I hope for your sake).. its up to you how efficient it
> is.
>

CDB is very fast.

Re: [Dovecot] [SOLVED] %d is empty in mail_location

[Phil Howard]

On Sat, May 15, 2010 at 10:10, Charles Marcus wrote:

> On 2010-05-14 3:52 PM, Phil Howard wrote:
> > On Fri, May 14, 2010 at 15:51, Alex Baule  wrote:
> >> Add in your auth,conf configuration:
> >>
> >> auth_default_realm = [your domain]
>
> > Which domain goes there?  I have many.
>
> Whichever one you want to be the 'default' if the user neglects to add a
> domain to their client username field.
>

I don't want a default.  I want them to get an authentication failure.  If
that failure could be informative and tell them that the domain is missing,
fine (but other aspects like "no such user" vs. "wrong password" vs. "no
such domain" I want to be left unexplained, and just be a simple
"authentication failed").

Re: [Dovecot] [SOLVED] %d is empty in mail_location

[Phil Howard]

On Sat, May 15, 2010 at 10:22, Charles Marcus wrote:

> On 2010-05-14 3:37 PM, Phil Howard wrote:
> > There was a typo in an earlier config item:
> >
> > auth_username_format = %...@ld
>
> I looked back in this thread and don't see any post where you provided
> output of dovecot -n...
>
> For future reference, I think this would have been easily seen in the
> beginning had you done so. One of the things it will do is tell you when
> there is a syntax error and make typos more evident...
>

I know I posted it, but it might have been in one of the other threads.  I
overlooked that syntax error several times, myself, so I'm in no position to
blame anyone else for that (nor was I expecting someone to directly find a
syntax error ... posting the configuration was expected to be more of a help
to see how I was doing things in general ... when it was asked for).  Sure,
extra eyes might have caught it had I posted the config in every thread.
But I'm not thinking that a mailing list should be for that.

Re: [Dovecot] [SOLVED] %d is empty in mail_location

[Phil Howard]

On Mon, May 17, 2010 at 10:56, Charles Marcus wrote:

> On 2010-05-17 9:34 AM, Phil Howard wrote:
> > On Sat, May 15, 2010 at 10:10, Charles Marcus wrote:
> >> On 2010-05-14 3:52 PM, Phil Howard wrote:
> >>> On Fri, May 14, 2010 at 15:51, Alex Baule 
> wrote:
> >>>> auth_default_realm = [your domain]
>
> >>> Which domain goes there?  I have many.
>
> >> Whichever one you want to be the 'default' if the user neglects to add a
> >> domain to their client username field.
>
> > I don't want a default.  I want them to get an authentication failure.
>
> So... remove the setting...?
>

I currently have it commented out, and will remove that line in the config
file completely when I go back to clean everything.

[Dovecot] looking for IMAP testing tool

[Phil Howard]

I'm looking for an IMAP testing tool, suitable to use with Dovecot IMAP.  It
needs to support TLS, STARTTLS, and login/authentication.  It needs to be
able run from command line, shell scripts, and even do so under cron jobs
(e.g. a way to supply the password to use w/o a terminal prompt).  Typical
interactive mail clients just don't cut it (even the text mode ones).  One
reason is I need to do the tests on a number of machines, under a number of
user and domain names, and with a variety of parameters or destinations.
This is for a suite of regression tests I am putting together intended to
verify that configuration changes do not break things (or unbreak things
that are supposed to not work).

Anyone ever heard of such a tool?  Open source would be preferred (better
yet, my favorite programming languages: C, Pike, Python).

Re: [Dovecot] looking for IMAP testing tool

[Phil Howard]

On Tue, May 18, 2010 at 10:17, Steffen Kaiser  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
>
> On Tue, 18 May 2010, Phil Howard wrote:
>
>  Anyone ever heard of such a tool?  Open source would be preferred (better
>>
>
> http://search.cpan.org/search?mode=dist&query=imap
>

Those all looked like libraries/modules.  Any complete commands?  Writing
Perl code is not an option for me.

Re: [Dovecot] looking for IMAP testing tool

[Phil Howard]

On Tue, May 18, 2010 at 11:42, Hugo Monteiro wrote:

>  Replying myself in this one. Should have looked a bit further into it.
>
>
> http://imapwiki.org/ImapTest


What I'm looking for is not something that just tries commands.  What I want
to do is actually try to pick up mail and store it in some directory.  My
script would then check to see if it got the mail it was expecting, or got a
failure it was expecting (for a scenario that should fail).  Sorry for not
being clear on that.

Re: [Dovecot] looking for IMAP testing tool

[Phil Howard]

On Tue, May 18, 2010 at 13:17, Mark Moseley  wrote:

> I haven't used it much but it looked useful:
> http://bc-bd.org/blog/imapfoo/
>

Hmmm.  That (inserting mail into folders) can have it uses.  But I'm really
looking for something to do, in a very basic way, what is expected of an
IMAP client, which is to fetch and read mail.  Something that can login and
list messages to stdout, or login and fetch a specified message to stdout,
would be what I need (and that login might be by clear port 143, STARTTLS on
port 143, or plain SSL/TLS on port 993).

Re: [Dovecot] looking for IMAP testing tool

[Phil Howard]

On Tue, May 18, 2010 at 15:50, Timo Sirainen  wrote:

> imaptest actually allows something similar to that. It has possibility to
> "expect" kind of scripts where it sends some commands and expects something
> specific in return. There's a tests/ directory that tests a lot of imap
> commands replies. I don't remember if it currently supports matching
> multi-line replies, and if it does it would be a bit difficult to write
> those for larger emails.
>
> Anyway, it doesn't support SSL/TLS. You could run it through stunnel
> though.
>

What I need is a program that already has all the logic to do IMAP as a
client already in it.  This isn't about testing IMAP logic per-se.  It's
about making sure mail is going through OK, and logins that should fail will
fail, and mail deliveries that should fail will fail (for example mail from
a computer listed as blocked should never show up in the mailbox designated
to test that, along with mail that has keywords specifically marked as "is
always spam").  It's testing at a higher level than seeing if a given IMAP
command gives the expected response (which is more of a diagnostic tool than
the monitoring tool I need).

Re: [Dovecot] looking for IMAP testing tool

[Phil Howard]

On Wed, May 19, 2010 at 05:24, Brian Candler  wrote:

> Then you're testing the whole environment: you'll need to deliver mail
> either by making SMTP connections or by invoking your LDA (e.g.  sendmail)
> and piping the mail in - with some way of forcing it to look "spammy" or
> "not spammy" - to check the blocking.  Then you'll use IMAP to retrieve
> them.  This is clearly more than testing just IMAP; rather, you're testing
> the whole mail server platform and its configs.
>

Yes, that is what I want to be testing.  So I need a way to send mail via
SMTP (including TLS and login authentication) as well as picking it up via
IMAP.  But I chose to only ask for the IMAP piece of it here (and the SMTP
piece of it on the Postfix mailing list ... to which Wietse suggested
"expect" and "openssl s_client" which I think I can handle using "pexpect"
in Python).  I know enough SMTP to do that end of things.  I don't know the
IMAP protocol at all, so something already built would help.  But I only
need a subset which is to just pick up all email from the inbox of a given
u...@domain, deleting them from the server while depositing them in a
designated directory.  Each different class of test would be using a
different mailbox.  Some tests will, when things work as expected, get no
mail at all (basically a test for rejecting mail).


I don't know of any ready-made test framework for what you want, and I
> suspect it would end up looking much like a programming language by the
> time
> you were able to configure all the different tests for processing different
> flavours of incoming mail.
>

An IMAP library might be doable (though not in Perl since I don't know that
language and don't have the time to learn it), but the basic "just pick up
and delete all mail" would be sufficient.

Re: [Dovecot] looking for IMAP testing tool

[Phil Howard]

On Wed, May 19, 2010 at 07:52, William Blunn  wrote:

> It sounds like you want a sort of "toolbox" of ready-made and tested
> components, such as an IMAP client, but with rich programmatic interfaces so
> that you only need to write a little bit of "glue" code to make it do
> exactly what you want.
>

I don't really need a lot.  I just need to pick up all mail in the inbox of
a given u...@domain I login with, deposit each in a given directory, and
delete them from the server.  The only options I think I need are:  what
u...@domain to login as, whether to use clear, STARTTLS, or a basic TLS/SSL
protocol, and what port number (143 or 993).  I don't have any interest in
other folders.  I just want to see if mail was delivered and do so from a
separate machine than the mail server itself (or see if IMAP is even up).

Re: [Dovecot] looking for IMAP testing tool

[Phil Howard]

On Wed, May 19, 2010 at 10:04, Brian Candler  wrote:

> If you can do SMTP, you can do IMAP. This should get you started:
>
> a login f...@bar.com xyzzy
> a select inbox   -- or "a examine inbox" for read-only
> a fetch 1:15 (rfc822)
> a store 1:15 +flags (\Deleted)
> a expunge
> a logout
>
> Also useful:
>
> a namespace-- folder separator chars
> a list "" "*"  -- list folder hierarchy
>
> Flick through RFC3501 for anything else you need.
>

I might have to do that sometime.  But from what I've seen of IMAP it is
more complex than SMTP.  POP3 was (though not greatly so).  Still, I don't
feel I'd want to implement a POP3 tool (and don't need one).



>  >An IMAP library might be doable (though not in Perl since I don't
> know
> >that language and don't have the time to learn it), but the basic
> "just
> >pick up and delete all mail" would be sufficient.
>
> As suggested by someone else, you can use 'fetchmail' to do that. Normally
> it delivers using SMTP, but with appropriate flags I believe it can pipe
> all
> the retrieved mail to stdout.  And if these are all separate mailboxes,
> POP3
> will do for your purposes anyway.
>

Or maybe just hack its source code?

Re: [Dovecot] looking for IMAP testing tool

[Phil Howard]

On Wed, May 19, 2010 at 12:49, Mailing List  wrote:

> Hi Phil,
>
> I've got an hack piece of regression perl code that we might be a good
> starting point. I'd love to turn it into a better piece of hack code
> that we could offer up. It does SSL/TLS.
>
> Right now it is pretty basic and forks 1K-10K simultaneous IMAP
> login/logout connections. Its goal is to find the sink point for the
> IMAP server and/or test client  so that tuning exceeds current usage.
>
> Let me know if you are interesting in looking at it and cleaning it up.
>
> ---Jack
>

The best I could do with something in Perl is rewrite it into something
else.  I'm still kinda weak on Python (don't know all the cool tricks, yet),
and Pike isn't found universally, so things would need to be in C for me to
do it and it to be widely usable.  And that's for later when I get some free
time.