I actually wrote these patches about a year and a half ago, but never got
around to joining the list and submitting them. They've been working in
production on 5.4.3 at a previous employer of mine since ~10/04, and I
just updated them to patch against 5.4.13. They were written one on top
of the other in the order presented, but #1 is independant of #2-3. I
believe that at least the first one should be integrated into the mainline
of vpopmail, and the latter two may be relevant for that but need some
work (see below).
Patch #1: Local Relay Patch
This patch came about from a problem we had with a high-volume vpopmail
environment where it was attempting to rebuild the tcp.smtp.cdb file so
often that it was causing performance problems. This path was intended to
eliminate the overhead of updating the CDB when the IP address was already
listed permanently in the tcp.smtp file. Since the IP address is already
listed, updating the CDB file is irrlelevant - if they're allowed they're
already in there, and if they're denied they'll never make it to the auth
stage. This checks for static relay matches on the remote IP address and
returns true on a match, then exits the update function cleanly.
Patch #2: Bigdir Backfill Patch - MYSQL Only
This patch came from another problem we had on the same environment -
every year we would add and delete ~800 accounts, but the new ones were
always added to the end. This resulted in a fragmented number of users in
different hash directories, as there was never any way to fill-in the
spaces that were left by deleted accounts. To make things worse, because
the increment check was based on the total number of accounts % 100, it
was very possible (and likely!) to get more than 100 users in a single
directory. For example:
Directory a has 100 users, and directory b has 98 users. When you add
another user, it sees that 198%100 = 98, so it adds a user to b, which now
has 99 users. Now delete 10 users from a, and then add 10 users; where do
they go? Logically, the first one should go in b, which
would then have 100 users and cause the next 9 to go in c. However, since
(90+99=189)%100 = 89, it will put them all into b, incrementing b to 109.
And there would still be room for 1 more in b before it moved on to c,
assuming no accounts were deleted from a.
This patch changes the way a directory's contents are counted, by checking
the vpopmail database and counting the number of users in that directory.
It also excludes has directories from its count, so it just lists the
users. We were using a MySQL backend so that's what I coded the patch for
- I haven't looked into other backends, but I'd guess that it shouldn't be
too hard to port to another backend for someone who's familiar with that
method. I call it the backfill patch because you can reset the
level_index information and it will walk the hash dirs filling in where
there are available spots as you add more users.
Patch #3: Dir Limits Patch - MYSQL ONLY
This patch builds on the previous patch and as such requires it. It
changes the vdir_type struct definition to include and int field named
the_dir, as well as adding a field to the dir_control table. The default
value is 100, to correspond to the original MAX_USERS_PER_DIR, but it
allows you to in/decrease that number on a per-domain basis to fit your
needs. Newly created vpopmail DBs will automatically be crated properly,
but existng DBs will need to be altered to work - see the file
README.dirlimits included in the patch. Again, this one is for mysql
only, but should be convertable to most any auth scheme with ease.
Please let me know if you {like|dislike|don't understand|other} these
patches, and if you have any questions/suggestions for them.
Thanks,
Josh
--
Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
- Layman's translation of the Laws of Thermodynamics
[EMAIL PROTECTED]
diff -urN ../clean/vpopmail-5.4.13/vpopmail.c vpopmail-5.4.13/vpopmail.c
--- ../clean/vpopmail-5.4.13/vpopmail.c 2005-07-19 20:10:22.000000000 -0400
+++ vpopmail-5.4.13/vpopmail.c 2006-01-05 09:39:46.000000000 -0500
@@ -2465,6 +2465,9 @@
*/
int open_smtp_relay()
{
+ if (check_static_relay()) {
+ return(0);
+ }
#ifdef USE_SQL
/* store the user's ip address into the sql relay table */
if (vopen_smtp_relay()) {
@@ -2643,6 +2646,44 @@
/************************************************************************/
+#ifdef POP_AUTH_OPEN_RELAY
+/* check_static_relay() looks to see if the remote IP address is in the
+ * TCP_RULES file, and if so returns true. Used to not update the relay
+ * table with IP addresses that are already covered by another rule.
+ */
+
+int check_static_relay()
+{
+ FILE *fs;
+ char tmpbuf1[MAX_BUFF];
+ int found = 0;
+ char *ipaddr, *p;
+
+ /* get the remote IP address as a string */
+ ipaddr = get_remote_ip();
+
+ /* open the tcp.smtp file and read in the static rules - these addresses
+ * are handled by tcprules and not the relay table */
+ fs = fopen(TCP_FILE, "r");
+ if ( fs != NULL ) {
+ /* read each entry and compare it to the remote IP address */
+ while (( fgets(tmpbuf1, sizeof(tmpbuf1), fs ) != NULL ) && (!found)){
+ if ( p = strchr(tmpbuf1, ':') ) {
+ *p = '\0';
+ if ( !(strncmp(ipaddr, tmpbuf1, strlen(tmpbuf1))) ) {
+ found = 1;
+ }
+ }
+ }
+ fclose(fs);
+ }
+
+ return(found);
+}
+#endif /* POP_AUTH_OPEN_RELAY */
+
+/************************************************************************/
+
int vfd_copy(int to, int from)
{
if (to == from) return 0;diff -urN ../01-TCD/vpopmail-5.4.13/bigdir.c vpopmail-5.4.13/bigdir.c
--- ../01-TCD/vpopmail-5.4.13/bigdir.c 2003-10-20 14:59:56.000000000 -0400
+++ vpopmail-5.4.13/bigdir.c 2006-01-05 14:01:43.000000000 -0500
@@ -55,9 +55,9 @@
return(0);
}
-char *next_big_dir(uid_t uid, gid_t gid)
+char *next_big_dir(char *domain, uid_t uid, gid_t gid)
{
- inc_dir_control(&vdir);
+ inc_dir_control(&vdir, domain);
if ( vdir.the_dir[0] != 0 ) {
r_mkdir(vdir.the_dir, uid, gid);
}
@@ -93,16 +93,26 @@
return(dirlist[i]);
}
-int inc_dir_control(vdir_type *vdir)
+int inc_dir_control(vdir_type *vdir, char *domain)
{
+ int user_count, i, rc;
+
++vdir->cur_users;
- if ( vdir->cur_users%MAX_USERS_PER_LEVEL == 0 ) {
- if ( strlen(vdir->the_dir) == 0 ) {
- vdir->the_dir[0] = dirlist[vdir->level_start[0]];
- vdir->the_dir[1] = 0;
- return(0);
- }
+ if ( (strlen(vdir->the_dir) == 0 ) && (vdir->cur_users >= MAX_USERS_PER_LEVEL) ) {
+ vdir->the_dir[0] = dirlist[vdir->level_start[0]];
+ vdir->the_dir[1] = 0;
+ return(0);
+ }
+ if (dirlist[vdir->level_index[vdir->level_cur]] != vdir->the_dir[vdir->level_mod[vdir->level_cur]]) {
+ vdir->level_cur = ((strlen(vdir->the_dir) < vdir->level_mod[1]) ? 0 : ((strlen(vdir->the_dir) < vdir->level_mod[2]) ? 1 : 2 ) );
+ for(i=vdir->level_start[vdir->level_cur];i<vdir->level_end[vdir->level_cur]+1 && (dirlist[i] != vdir->the_dir[vdir->level_mod[vdir->level_cur]]);i++);
+ vdir->level_index[vdir->level_cur] = i;
+ for(i=vdir->level_cur+1;i<vdir->level_max;i++)
+ vdir->level_index[i] = 0;
+ }
+ rc = vcount_dir_users(&user_count, domain, vdir->the_dir);
+ while ( user_count >= MAX_USERS_PER_LEVEL ) {
if ( vdir->level_index[vdir->level_cur] ==
vdir->level_end[vdir->level_cur] ) {
switch (vdir->level_cur) {
@@ -122,6 +132,7 @@
}
}
inc_dir( vdir, vdir->level_cur );
+ rc = vcount_dir_users(&user_count, domain, vdir->the_dir);
}
return(0);
}
diff -urN ../01-TCD/vpopmail-5.4.13/vauth.h vpopmail-5.4.13/vauth.h
--- ../01-TCD/vpopmail-5.4.13/vauth.h 2005-03-10 00:22:38.000000000 -0500
+++ vpopmail-5.4.13/vauth.h 2006-01-05 14:01:43.000000000 -0500
@@ -95,16 +95,17 @@
int open_big_dir(char *domain, uid_t uid, gid_t gid);
int close_big_dir(char *domain, uid_t uid, gid_t gid);
-char *next_big_dir(uid_t uid, gid_t gid);
+char *next_big_dir(char *domain, uid_t uid, gid_t gid);
char *inc_dir(vdir_type *, int);
char next_char(char, int, int);
-int inc_dir_control(vdir_type *);
+int inc_dir_control(vdir_type *, char *);
int dec_dir_control(char *domain, uid_t uid, gid_t gid);
void print_control();
int vread_dir_control(vdir_type *vdir, char *domain, uid_t uid, gid_t gid );
int vwrite_dir_control(vdir_type *vdir, char *domain, uid_t uid, gid_t gid);
int vdel_dir_control(char *domain);
+int vcount_dir_users(int *user_count, char *domain, char *dir);
char *valias_select( char *alias, char *domain );
char *valias_select_next();
diff -urN ../01-TCD/vpopmail-5.4.13/vconvert.c vpopmail-5.4.13/vconvert.c
--- ../01-TCD/vpopmail-5.4.13/vconvert.c 2004-08-24 13:17:39.000000000 -0400
+++ vpopmail-5.4.13/vconvert.c 2006-01-05 14:01:43.000000000 -0500
@@ -248,7 +248,7 @@
}
vauth_setpw(pw, domain);
#ifdef USERS_BIG_DIR
- next_big_dir (uid, gid); /* increment user count */
+ next_big_dir (domain, uid, gid); /* increment user count */
#endif
}
fclose(fs);
diff -urN ../01-TCD/vpopmail-5.4.13/vmysql.c vpopmail-5.4.13/vmysql.c
--- ../01-TCD/vpopmail-5.4.13/vmysql.c 2004-12-16 10:57:34.000000000 -0500
+++ vpopmail-5.4.13/vmysql.c 2006-01-05 14:01:43.000000000 -0500
@@ -1122,6 +1122,31 @@
return(0);
}
+int vcount_dir_users(int *user_count, char *domain, char *dir)
+{
+ *user_count = -1;
+
+ snprintf( SqlBufRead, SQL_BUF_SIZE, "select count(*) from `vpopmail` where ((`pw_dir` like binary \"/var/vpopmail/domains/%s/%s/%%\") and (`pw_dir` not like binary \"/var/vpopmail/domains/%s/%s/%%/%%\"))", domain, dir, domain, dir);
+
+ if (mysql_query(&mysql_read,SqlBufRead)) {
+ fprintf(stderr, "vcount_dir_users: SQL Query Failed - check your database!\n");
+ exit(1);
+ }
+
+ if (!(res_read = mysql_store_result(&mysql_read))) {
+ fprintf(stderr, "vcount_dir_users: store result failed 8\n");
+ exit(1);
+ }
+
+ if ((row = mysql_fetch_row(res_read)) != NULL) {
+ *user_count = atoi(row[0]);
+ }
+
+ mysql_free_result(res_read);
+
+ return(0);
+}
+
#ifdef ENABLE_AUTH_LOGGING
int vset_lastauth(char *user, char *domain, char *remoteip )
{
diff -urN ../01-TCD/vpopmail-5.4.13/vpopmail.c vpopmail-5.4.13/vpopmail.c
--- ../01-TCD/vpopmail-5.4.13/vpopmail.c 2006-01-05 09:39:46.000000000 -0500
+++ vpopmail-5.4.13/vpopmail.c 2006-01-05 14:01:43.000000000 -0500
@@ -159,7 +159,7 @@
* Depending on how many domains we have, it may need to be hashed
*/
open_big_dir(dir_control_for_uid, uid, gid);
- domain_hash = next_big_dir(uid, gid);
+ domain_hash = next_big_dir(domain, uid, gid);
close_big_dir(dir_control_for_uid, uid, gid);
if ( strlen(domain_hash) > 0 ) {
@@ -1731,7 +1731,7 @@
#ifdef USERS_BIG_DIR
/* go into a user hash dir if required */
open_big_dir(domain, uid, gid);
- user_hash = next_big_dir(uid, gid);
+ user_hash = next_big_dir(domain, uid, gid);
close_big_dir(domain, uid, gid);
chdir(user_hash);
#endifdiff -urN ../02-TCD/vpopmail-5.4.13/bigdir.c vpopmail-5.4.13/bigdir.c
--- ../02-TCD/vpopmail-5.4.13/bigdir.c 2006-01-05 14:01:43.000000000 -0500
+++ vpopmail-5.4.13/bigdir.c 2006-01-05 14:08:19.000000000 -0500
@@ -98,7 +98,7 @@
int user_count, i, rc;
++vdir->cur_users;
- if ( (strlen(vdir->the_dir) == 0 ) && (vdir->cur_users >= MAX_USERS_PER_LEVEL) ) {
+ if ( (strlen(vdir->the_dir) == 0 ) && (vdir->cur_users >= vdir->dir_limit) ) {
vdir->the_dir[0] = dirlist[vdir->level_start[0]];
vdir->the_dir[1] = 0;
return(0);
@@ -112,7 +112,7 @@
vdir->level_index[i] = 0;
}
rc = vcount_dir_users(&user_count, domain, vdir->the_dir);
- while ( user_count >= MAX_USERS_PER_LEVEL ) {
+ while ( user_count >= vdir->dir_limit ) {
if ( vdir->level_index[vdir->level_cur] ==
vdir->level_end[vdir->level_cur] ) {
switch (vdir->level_cur) {
diff -urN ../02-TCD/vpopmail-5.4.13/README.dirlimits vpopmail-5.4.13/README.dirlimits
--- ../02-TCD/vpopmail-5.4.13/README.dirlimits 1969-12-31 19:00:00.000000000 -0500
+++ vpopmail-5.4.13/README.dirlimits 2006-01-05 14:28:34.000000000 -0500
@@ -0,0 +1,27 @@
+The dirlimits patch allows you to specify the number of users the vpopmail
+will put at any level of the directory tree before incrementing to the next
+directory. The default is 100, which is what will populate the table when
+the domain is created. If you want to change it, just edit the value in
+the mysql table (this patch currently only supports mysql - anyone who
+wants to add other back-end support is welcome to copy the idea).
+
+The dirlimits patch to vpopmail requires that you change the structure
+of any existing dir_control table to match the new schema. If you are
+starting fresh and have no vpopmail tables yet, you can safely ignore
+this. If you have an existing dir_control table, just execute the following
+line at a mysql command prompt with admin privileges on the vpopmail DB:
+
+ ALTER TABLE `dir_control` ADD `dir_limit` INT( 11 );
+
+Once this column has been added, set the value of the field to whatever
+your desired limit is (default 100) using the following:
+
+ UPDATE `dir_control` SET `dir_limit` = '100';
+
+That's it - you now can change this on a per-domain basis as needed.
+
+If you have questions about this patch, send email with 'dirlimits' somewhere
+in the subject so I know it's specifically for me and not just list traffic.
+
+Joshua Megerman
[EMAIL PROTECTED]
diff -urN ../02-TCD/vpopmail-5.4.13/vauth.h vpopmail-5.4.13/vauth.h
--- ../02-TCD/vpopmail-5.4.13/vauth.h 2006-01-05 14:01:43.000000000 -0500
+++ vpopmail-5.4.13/vauth.h 2006-01-05 14:08:52.000000000 -0500
@@ -89,6 +89,7 @@
int level_index[MAX_DIR_LEVELS]; /* current spot in dir list */
long unsigned cur_users;
char the_dir[MAX_DIR_NAME];
+ int dir_limit;
} vdir_type;
#define MAX_DIR_LIST 62
diff -urN ../02-TCD/vpopmail-5.4.13/vmysql.c vpopmail-5.4.13/vmysql.c
--- ../02-TCD/vpopmail-5.4.13/vmysql.c 2006-01-05 14:01:43.000000000 -0500
+++ vpopmail-5.4.13/vmysql.c 2006-01-05 14:11:41.000000000 -0500
@@ -1018,6 +1018,11 @@
vdir->level_index[2] = atoi(row[14]);
strncpy(vdir->the_dir, row[15], MAX_DIR_NAME);
+
+ vdir->dir_limit = atoi(row[16]);
+ if (vdir->dir_limit == 0) {
+ vdir->dir_limit = MAX_USERS_PER_LEVEL;
+ }
}
mysql_free_result(res_read);
@@ -1050,19 +1055,19 @@
level_start0, level_start1, level_start2, \
level_end0, level_end1, level_end2, \
level_mod0, level_mod1, level_mod2, \
-level_index0, level_index1, level_index2, the_dir ) values ( \
+level_index0, level_index1, level_index2, the_dir, dir_limit ) values ( \
'%s', %lu, %d, %d, \
%d, %d, %d, \
%d, %d, %d, \
%d, %d, %d, \
%d, %d, %d, \
-'%s')\n",
+'%s', %d)\n",
domain, vdir->cur_users, vdir->level_cur, vdir->level_max,
vdir->level_start[0], vdir->level_start[1], vdir->level_start[2],
vdir->level_end[0], vdir->level_end[1], vdir->level_end[2],
vdir->level_mod[0], vdir->level_mod[1], vdir->level_mod[2],
vdir->level_index[0], vdir->level_index[1], vdir->level_index[2],
- vdir->the_dir);
+ vdir->the_dir, vdir->dir_limit);
if (mysql_query(&mysql_update,SqlBufUpdate)) {
vcreate_dir_control(domain);
@@ -1086,15 +1091,15 @@
level_start0, level_start1, level_start2, \
level_end0, level_end1, level_end2, \
level_mod0, level_mod1, level_mod2, \
-level_index0, level_index1, level_index2, the_dir ) values ( \
+level_index0, level_index1, level_index2, the_dir, dir_limit ) values ( \
'%s', 0, \
0, %d, \
0, 0, 0, \
%d, %d, %d, \
0, 2, 4, \
0, 0, 0, \
-'')\n",
- domain, MAX_DIR_LEVELS, MAX_DIR_LIST-1, MAX_DIR_LIST-1, MAX_DIR_LIST-1);
+'', %d)\n",
+ domain, MAX_DIR_LEVELS, MAX_DIR_LIST-1, MAX_DIR_LIST-1, MAX_DIR_LIST-1, MAX_USERS_PER_LEVEL);
if (mysql_query(&mysql_update,SqlBufUpdate)) {
fprintf(stderr, "vmysql: sql error[d]: %s\n", mysql_error(&mysql_update));
diff -urN ../02-TCD/vpopmail-5.4.13/vmysql.h vpopmail-5.4.13/vmysql.h
--- ../02-TCD/vpopmail-5.4.13/vmysql.h 2004-12-16 10:57:35.000000000 -0500
+++ vpopmail-5.4.13/vmysql.h 2006-01-05 14:12:29.000000000 -0500
@@ -248,14 +248,14 @@
level_end0 int, level_end1 int, level_end2 int, \
level_mod0 int, level_mod1 int, level_mod2 int, \
level_index0 int , level_index1 int, level_index2 int, the_dir char(160), \
-primary key (domain) "
+dir_limit int, primary key (domain) "
#define DIR_CONTROL_SELECT "cur_users, \
level_cur, level_max, \
level_start0, level_start1, level_start2, \
level_end0, level_end1, level_end2, \
level_mod0, level_mod1, level_mod2, \
-level_index0, level_index1, level_index2, the_dir"
+level_index0, level_index1, level_index2, the_dir, dir_limit"
#define VALIAS_TABLE_LAYOUT "alias char(32) not null, \
domain char(64) not null, \
