Re: [vchkpw] The future of valias & other topics

[John Simpson]

On 2007-01-05, at 1334, Rick Widmer wrote:

email clients I know of have handled smtp auth for years now, and the
best thing we can do for the average user is to let smtp after pop die
off.  Drastic improvements in the mainstream code will send the wrong
message.

I wrote the paragraph above before reading the 'Further thoughts...'
thread on Jan. third.  If you want to do massive changes they will get

and letting smtp after pop die?

I think the best thing to do with the localrelay patch is to put it into the contrib directory where people who need it can find it easily. The main reason I think this is because smtp after pop is obsolete. All the added, but is it really better than recommending smtp auth for everyone

i think the idea of an "onauth" hook, similar to how "onchange" works, will minimize the amount of code changes needed for vpopmail, while opening the door for somebody (like joshua) to write a relay- after-pop3/imap system as external scripts if he likes, and also opening the door for others to add their own custom functionality. and it also removes any specific relay-after-pop3 functionality from vpopmail itself, while not destroying that functionality for people who may not be able to stop using it for political reasons. and it also means that vpopmail itself will never have to worry about relay-after-pop3 issues again- they can be referred to whoever wrote the external scripts that they will be using.

I did not see any objections on changing to .vpopmail* files in user but not domain directories. If anyone does object, now is the time. Files

in ~vppopmail/domains/user_name/ are executed by vdelivermail and will
be renamed.  Files in ~vpopmail/domains/ will stay .qmail and are
executed by qmail-local.

i agree- any files which are currently processed by qmail-local should retain the ".qmail-*" names, and others (i.e. the per-mailbox files) should use the ".vpopmail" names EXCLUSIVELY, so as to make it obvious to anybody poking through the directory structure that those files are processed by vpopmail rather than by qmail itself. as for making .vpopmail* files fully compatible with .qmail* files, that could also be a good thing- however the interface (in terms of which environment variables will be passed to scripts run from the .vpopmail file, what values will be contained in those variables, and how the return values will be interpreted) should be not only documented, but made to be as compatible as possible with what qmail- local would pass to a script in the .qmail file of a system account mailbox. this way if somebody has to transition a domain from being "system accounts" to being managed by vpopmail, there should be no changes to the files or scripts (as long as they are using the environment variables properly, as opposed to hard-coding path names into the script- which we all know users do.)

Adding an order field to aliases... I think we should "just say no!" to

ordered aliases.  With 5.4.18 all the basic operations on an alias are
available.  We just declare that you can not count on the order of
delivery of an alias, and call alias handling complete.

i disagree. very strongly.

I realize that sometimes order is important.  I propose to provide
another interface similar to aliases that are always ordered, called
scripts, or more formally mail scripts.  Vdelivermail already supports

share the same code, if not the same table.

alias handling from a database so scripts and aliases should be able to

so you're going to design this new thing which is LIKE an alias, but it supports sequencing... and it can use the same code and probably the same database tables... but it's NOT an alias, simply because you give it some other name. explain to me how your "scripts" are any different from "aliases with a sequence field", or "a .qmail file". either way, the functional result is the same: a set of delivery targets, guaranteed to be processed in a certain order. seriously- i don't see any difference between what you call "scripts" and what vpopmail currently calls an "alias", but with a sequence field attached to each line. in fact, for the cdb back-end, these are called ".qmail files". the only thing missing is a way to store the sequencing in the database back ends, and the code to use and maintain those sequence fields. for vdelivermail, the only change in the code involves a single "ORDER BY" clause added to the proper SELECT query. and for the maintenance programs, it's one extra parameter passed to (1) the command, (2) the vpopmail.c stub function, and (3) the various back-end functions... and the extra code added to each back- end function. the hard parts are going to be coordinating the changes to all of the back-end functions at once, and then testing all of it. if somebody has the time, it won't be difficult for one person (myself, if needed) to write the changes for all of the back-end libraries and add them all to the CVS repository. finding testers will be interesting though... i'm sure there are no shortage of cdb and mysql users, but what about the others? not just finding users, but finding people who are able to TEST the new code, without directly impacting their existing services? i could probably set up a test box with postgresql on it, but the others (oracle, sybase, and LDAP) i wouldn't be able to touch.

There are good arguments that scripts are best implemented as .qmail or

.vpopmail files but vdelivermail already supports all the delivery
methods from a database, and it will continue to do so.

actually, this i agree with. as long as vdelivermail can be taught to add an ORDER BY clause to the SELECT query it uses to fetch the individual "lines" from the alias "file" in the database, there should be no difference whether the instructions come from a text file or a database.

On the other hand, I plan to do all I can to steer people towards using

files for mail scripts.

since i don't see any difference between your "scripts" and my "add a sequence field to the database table", i read this as "i plan to do all i can to steer people towards storing aliases as files." and that's certainly an option. however, i'm sure there are users now who rely on being able to edit the contents of an alias by modifying an SQL table, even if they can't currently control the order in which the instructions within the alias will be processed by the delivery program. i don't think it's right to take away their current partial functionality and force them to find an entirely new solution (a way to edit text files, or to start using vpopmaild.)

Once the script functions are added I plan to add a template system that

allows the system admin (or anyone with admin and expert abilities) to
manage the templates.

that's fine (actually it's a great idea) but i think it's external to the core of vpopmail. what it all comes down to is that an "alias" is nothing more than an ordered set of instructions, stored somewhere- either in a .qmail file, a .vpopmail file, or a database table. whether those instructions are generated by a series of manual vpopmail API calls, an administrator with a text editor or the mysql command line program, or your macro processing script- it doesn't matter, as long as the correct information ends up in the "alias" (file or database table) in the right order. i do, however, think the sequencing support needs to be in there first, before we start worrying about templates.

Finally, my response to various questions from the list...

there is a possible race condition- if process A is doing a delivery,
looks for a ".vpopmail" file, and doesn't find one, then process B
takes over and renames a ".qmail" file to ".vpopmail", then process A
takes over and looks for a ".qmail" file, then process A will think
that no file with either name exists.

and the open close together.

The reader process needs to keep the stat that verifies the file exists

ahhhh... you say "close together". but there's no way you can guarantee that there won't be a context switch between the two operations, so as long as they ARE two separate operations, a potential race condition exists. (curse/thank my CS instructors for drilling the idea of recognizing and avoiding race conditions into my head at such a young and tender age...) the easiest "safe" way around this would be to assert an flock() on the directory (or on a specific filename within the directory) which contains the .qmail and/or .vpopmail files, before searching for either filename... and only release the lock after reading the contents of whichever file you're going to use (if any) into memory... and also ensuring that every piece of code in the vpopmail suite (including any external management scripts which may have been written by others) which touches these files, participates in (and honours) the flock() scheme.

The open needs to handle the case where the file suddenly disappears,

which you would detect HOW? by doing another stat()?

and if both processes are doing this "stat, sleep, stat" routine at roughly the same time, what's to prevent the same race condition from happening during the second round of stat() calls? i hate to say it, but the more i think about it, the more i think it's not such a good idea to try and rename files "on the fly" like this. (keep reading, the next bit explains the other reason why.)

i think a safer way to do it would be to have a script which finds and renames the files as needed, and tell the users to only run the script while qmail-send is stopped. if they're upgrading the vpopmail binaries

anyway, they will probably be in some kind of maintenance window to
start with, and stopping qmail-send won't be a major deal.

I agree we should include or point people to the script, but if they don't run it, I still want to change files to .vpopmail, unless they go out of their way to disable it. I think Charles Cazabon's objection to .qmail files that are executed by vdelivermail is a strong enough reason to do so. I want to eliminate as many of his objections to vpopmail as I can.

i respect charles and his opinion about .qmail and .vpopmail files... but making charles happy is not a primary goal of vpopmail, and it's certainly not an excuse to unsafely force this change on all of the existing vpopmail systems out there. i think it makes more sense to explain the situation to the administrators, provide them with a tool to manually rename the files en masse (and identify any problem cases where both .qmail and .vpopmail files already exist) and TELL them that it should be done- but the final decision about whether and when to do the change should be left in the hands of the administrator of each system. of course i would also remove the "either/or" filename logic from the NEXT version of vpopmail, so that if they haven't renamed the files, they become broken and it's their own fault.

i had one incident where the domain's admin didn't realize that this

...

one alias was "special"- he accidentally added a new employee after the

as a user with expert rights and creating a custom script.  The system
will protect your script from other administrators within the account.

what is there before the delete.


the program then maybe forwards to another script.  That would allow
non-expert users to manage the address lists without touching the part
that checks for keywords.

In my vision, that would be a mail script that you create by logging in The only thing non-experts can do with an expert edited script is delete the whole thing. They will be warned that they can not edit or recreate Maybe you should write an expert script that forwards to one alias, runs

what you call a "script", i see as the same thing that we've all been calling an "alias with sequences", and wishing we had, for the past year. so i'm not sure there is such a thing as a "custom script", since any user can use the API and add lines which call programs (and once sequencing is working, any user will be able to use the API and specify, when they add a user, whether the user should be added before or after a given program line.)

besides, i'm not sure i understand the point of sorting the entries,

there are certainly no performance reasons, and if you're  trying to

files work the way they are, why worry about sorting them  at all?

either within a .(q|vpop)mail file, or within the qmail control files. encourage people to use the API instead of manually editing the files (and potentially introducing errors) then i would think as long as the

Completeness.  Domains are sorted.  Users are sorted.  The list of
aliases within a domain is sorted.  The list of addresses within an
alias should be sorted too.  All that is to allow vpopmaild to handle
breaking result data into pages without having to load and sort the
entire dataset each time a web based interface requests a new page.

i'm sorry, but i totally disagree with you on this one.

i can SORTA see having a list of domains within the server, a list of users within a domain, or a list of aliases within a domain, and keeping those lists in a pre-sorted format... and while i think it's a waste of time and CPU cycles, it doesn't really hurt anything- because these entities (domains, users, and aliases) are all things which exist independently of each other. there is no NEED for them to be maintained in a certain order within their parent sets, therefore it's harmless if you want to sort them for cosmetic reasons. however, the set of lines within an alias is a different beast entirely. there IS a need for them to be stored in a specific sequence, because when they are being used to control a delivery, they need to be processed in a certain order. you stated above "I realize that sometimes order is important", but now you're saying that you want to sort the entries for purely cosmetic reasons- which means that you DON'T think that the existing order is important. in a way you're contradicting yourself. either the sequence IS important (and therefore must be preserved) or it's NOT important (and you can feel free to sort cosmetically if you like.) to me it looks like this all started as code patches to make vpopmaild able to handle "list_" commands which may return multiple items- basically, being able to serve up "page 4 of 19" from some list. honestly, i think those kinds of issues belong with the client program which calls vpopmaild, rather than with vpopmaild itself. sure, it makes it easier to write a web interface if the server back- end can sort and paginate the list for you... but the sorting and pagination are functions of the program which is doing the presenting- in this case, the web interface itself. how much easier would vpopmaild be to write and maintain if it didn't have to deal with these kinds of issues- if it could just deal with the raw data, in the order it finds it? look at how many changes have been made to other parts of vpopmail over the past few months, just to keep things like rcpthosts and virtualdomains and users/assign in some cosmetically sorted order... and for what? to save some PHP programmer from having to call a sort() function? there is a line between "data" and "presentation"... and i think it's been drawn in the wrong place.

3) Most importantly, to me the idea of storing aliases in the DB is just

that - it's for aliases.  If you have a program that you want to pipe
the message through, or send a copy of your mail offsite, I'd much
rather have that be via .qmail file and not as a database line.

what exactly do we mean when we say "alias"?

i don't recognize this paragraph, although it's written in a style similar to my own so i may have written it... regardless, it brings another point to mind- one which is probably the heart of this whole issue. is an "alias" limited to only containing email addresses? or does an "alias" correspond to a .qmail file? didn't you say that one of the goals is to make vdelivermail be able to handle anything that qmail- local itself handles? IF by "alias" you mean JUST a set of email addresses, then yes, you could sort the list in any order you like, without doing any damage. however, if "alias" is a term corresponding to a .qmail file, then an "alias" needs to have the same properties as a .qmail file- it contains a set of lines which are read and processed by the delivery agent (qmail-local or vdelivermail) IN A CERTAIN ORDER. i think somebody needs to make a decision about what the word "alias" really means. if it corresponds to a .qmail file, then we DO need a way to store sequences in the databases, and we DO need ways to maintain those sequence values using the command line tools, and using vpopmaild, and using qmailadmin. and 80% of this discussion goes away- we can stop trying to figure out whether or not to make certain changes, and we can figure out how to do them in such a way as to provide the functions people need while changing as little of the API, command line tools, and vpopmaild, as possible. and if somebody's going to make the executive decision that "alias" is just a bag of email addresses, and that sequence is not important... then storing my own aliases in any database back-end becomes impossible (no great loss for me personally, i'm happy with the cdb files) and people who do use database back-ends remain limited in the kinds of aliases they can set up.

That is an example of the argument for not using the database for
scripts.  I'm on the fence about it.  A lot of work has gone into
vdelivermail to allow it to do aliases from a database, and there has
been considerable work on extending that to scripts.  I would be happy
to just say that scripts should be stored in the filesystem, but it
seems a lot of database users want to keep everything in the database.

Let the argument commence...

i see two options:

it CAN be done either way... but then remember that to me, a "script" and an "alias with sequences" are the same thing, so my "either way" may not be exactly what you're thinking- so i'll spell it out. (1) add a sequence field to the database, and write the code necessary to support it. this will result in the users of the database back-ends GAINING the same functionality that users of the cdb back-end already have- namely, the ability to sort the lines within a given alias, in a certain order. (2) change all of the database back-ends to store their aliases in the filesystem. while this allows these users to control the sequence in which the delivery instructions will be processed, it TAKES AWAY their current ability to maintain the contents of their aliases by doing SQL queries, and FORCES them to have to edit a file on the filesystem in order to maintain their aliases. the second option would certainly be easier to write (only one set of maintenance functions involved.) however, if there are users who NEED to have the alias lines stored in a database, and they are willing to adjust their own systems to deal with a sequence field, then there is no reason (other than time constraints) that we can't store everything in a database. i've already written pseudo-code for the framework of each maintenance operation, the only thing preventing me from turning that into real working code at this point is free time, and the lack of a consensus about how it will be handled (i.e. i don't want to write code which won't actually be used.) if there are users out there who NEED to have the alias lines stored in their database back-ends, we CAN make that happen. granted, it means more coding and more testing, which means it'll probably take a little longer to finish, but if it makes the program usable to more people, isn't that the important thing?

I'm OK if you want to try to extend the functionality of Vpopmail to do more stuff, just please don't break the current way of doing things.

I don't think I'm breaking any existing functionality.  If you see
something that my plans would break, please let me know.  Backwards
compatibility is very important.

add_alias [EMAIL PROTECTED] target
        add a new line "target" to the end of the indicated alias (current)

add_alias [EMAIL PROTECTED] @2 target
        any existing lines with sequence 2 or higher, increment sequence
        add "target" with sequence 2

remove_alias [EMAIL PROTECTED] target
        find and delete the line matching "target" from the alias (current)

remove_alias [EMAIL PROTECTED] @2
        delete the line with sequence 2
        any existing lines with sequence higher than 2, decrement sequence



----------------------------------------------------------------
| John M. Simpson    ---   KG4ZOW   ---    Programmer At Large |
| http://www.jms1.net/                         <[EMAIL PROTECTED]> |
----------------------------------------------------------------
| http://video.google.com/videoplay?docid=-4312730277175242198 |
----------------------------------------------------------------

unfortunately, the current design of vpopmail was broken to begin with. the original design of the alias-related commands should have included support for sequencing from the start. it never did. whatever scripts or web interface you're using now will have to be modified, unless you're happy with only being able to add new lines to the end of an existing alias. the closest i can see doing would be to add an OPTIONAL flag to the "valias -i" command, which allows you to specify the sequence in which to add the new target... and if that option is not present, then the new target would be added to the end of the alias- which, if you're using the cdb back-end, is what happens now anyway. the same would go for the "valias -d" command... you may continue to use the current syntax (which deletes the first line matching the given target) or you would be able to use an option to say "delete line 3" instead. similar changes will be needed for the "add_alias" and "remove_alias" commands in vpopmaild. since "@" is never a valid character at the beginning of an alias line, i propose the following: and again, i've already written pseudo-code for all of the back-end functions to support aliases. all we need is for somebody to say "yes", and probably a week for me to squeeze out some spare time and turn that pseudo-code into actual working code... and it could be done. the only reason i can see for not wanting to do it is because it does involve writing and testing more code. however, i think we as developers need to remember that the point of writing a program is to make our users' lives easier, not to make our own programming jobs easier.

PGP.sig
Description: This is a digitally signed message part