[Dovecot] Force rebuild v1.0.1
Haven't seen a thread discussing forcing rebuilding of indexes / cache, so the question is, if these files in a users home directory were deleted... -rw--- 1 bb admin 80 May 17 15:05 dovecot-keywords -rw--- 1 bb admin 85449 Jul 2 06:15 dovecot-uidlist -rw--- 1 bb admin6832 Jul 2 06:15 dovecot.index -rw--- 1 bb admin 752640 Jul 2 06:19 dovecot.index.cache -rw--- 1 bb admin 95416 Jul 2 06:20 dovecot.index.log -rw--- 1 bb admin 131104 Jun 27 21:32 dovecot.index.log.2 ...would they be rebuilt by dovecot? Thank you, B. Bodger
Re: [Dovecot] mbox vs maildir
I did the ext3 -> ext4 switch on two of our proxyservers a few months ago. Then we forgot (!) about that test and the boxes just kept running and running and running ... Interesting... have you noticed any differences in performance? No. But at least it didn't explode in my face :) Heh.. yeah, thats always a GoodThing(TM) -- Best regards, Charle
[Dovecot] [Patch] Samba's proposed "ntlm_auth + winbind" support for dovecot-auth
The Samba team recommends to use their "ntlm_auth" command line helper for "NTLM" and "GSS-SPNEGO" authentication. This helper interacts with the Samba's winbind daemon, and this way can authenticate users against NT or Active Directory windows domain. Currently Dovecot can do "NTLM" authentication too, but just "locally" (against a local or sql database etc.). I've made a patch (attached), which adds "ntlm_auth" (or "winbind") support for Dovecot. The idea is to add two new authentication mechanisms: "mech_winbind_ntlm" and "mech_winbind_spnego". Both are coded in one additional file, "mech-winbind.c". An option "auth_ntlm_use_winbind" specifies whether to use the current implementation of ntlm, or do it by the "ntlm_auth" helper. "GSS-SPNEGO" always performed by the helper. Normally, "ntlm_auth" is invoked once, for all further requests. Such a way, "ntlm_auth helper from the Samba package, interacting with the Samba's winbind daemon", is used now by Squid, Apache and AFAIK some other applications. It is "strongly recommended" by the Samba team, and was already proposed even in this maillist 3 year ago (see f.e. http://www.dovecot.org/list/dovecot/2004-September/004775.html ). I hope there are no any serious performance issues for such a "complex way" -- f.e. with our web proxy, using this way, ~200 users do not feel any actual delays etc. I've successfully tested this patch with NTLM against AD domain. This patch can considerably improve the situation of "Email client on Windows desktop under Windows domain, but imap/pop at UNIX server". Currently, windows users have to specify their "login/password" for email accounts manually. There is an "SPA" (Secure Password Authentication) alternative for them, where just the desktop's login is used transparently, but it cannot be used now, because Dovecot cannot perform NTLM against, say, Active Directory domain. Since "dovecot-auth" daemon can be utilized by MTA as well (Postfix and other), the support of "ntlm_auth" in Dovecot can satisfy both SMTP and IMAP servers at UNIX side, and solve the issue completely. Questions and requests: - I try to code things most close to used style, i.e. using Dovecot's memory-management and io-pipe routines etc., but could someone look at it and check whether I've missed something or not? - Perhaps some names (of routines, modules) could be chosen better? - Maybe some other options should be implemented, i.e. "auth_winbind_helper_ntlm" and "auth_winbind_helper_spnego" to specify the helper's cmdline exactly ? - Currently I strip domain part of the username returned, i.e. from "DOMAIN\user" just to "user". Maybe better add some option "auth_winbind_strip_domain" for this? Certainly, it will be fine if someone else check it more, especially for "GSS-SPNEGO" which I cannot test for a while. Regards, Dmitry Butskoy http://www.fedoraproject.org/wiki/DmitryButskoy diff -Nru dovecot-1.0.1/src/auth/Makefile.am dovecot-1.0.1-OK/src/auth/Makefile.am --- dovecot-1.0.1/src/auth/Makefile.am 2007-05-19 15:14:04.0 +0400 +++ dovecot-1.0.1-OK/src/auth/Makefile.am 2007-06-29 17:34:33.0 +0400 @@ -56,6 +56,7 @@ mech-cram-md5.c \ mech-digest-md5.c \ mech-ntlm.c \ + mech-winbind.c \ mech-gssapi.c \ mech-rpa.c \ mech-apop.c \ diff -Nru dovecot-1.0.1/src/auth/Makefile.in dovecot-1.0.1-OK/src/auth/Makefile.in --- dovecot-1.0.1/src/auth/Makefile.in 2007-06-14 16:02:13.0 +0400 +++ dovecot-1.0.1-OK/src/auth/Makefile.in 2007-06-29 17:35:06.0 +0400 @@ -78,6 +78,7 @@ mech.$(OBJEXT) mech-anonymous.$(OBJEXT) mech-plain.$(OBJEXT) \ mech-login.$(OBJEXT) mech-cram-md5.$(OBJEXT) \ mech-digest-md5.$(OBJEXT) mech-ntlm.$(OBJEXT) \ + mech-winbind.$(OBJEXT) \ mech-gssapi.$(OBJEXT) mech-rpa.$(OBJEXT) mech-apop.$(OBJEXT) \ passdb.$(OBJEXT) passdb-blocking.$(OBJEXT) \ passdb-bsdauth.$(OBJEXT) passdb-cache.$(OBJEXT) \ @@ -325,6 +326,7 @@ mech-cram-md5.c \ mech-digest-md5.c \ mech-ntlm.c \ + mech-winbind.c \ mech-gssapi.c \ mech-rpa.c \ mech-apop.c \ @@ -494,6 +496,7 @@ @AMDEP_TRUE@@am__include@ @[EMAIL PROTECTED]/$(DEPDIR)/[EMAIL PROTECTED]@ @AMDEP_TRUE@@am__include@ @[EMAIL PROTECTED]/$(DEPDIR)/[EMAIL PROTECTED]@ @AMDEP_TRUE@@am__include@ @[EMAIL PROTECTED]/$(DEPDIR)/[EMAIL PROTECTED]@ [EMAIL PROTECTED]@@am__include@ @[EMAIL PROTECTED]/$(DEPDIR)/[EMAIL PROTECTED]@ @AMDEP_TRUE@@am__include@ @[EMAIL PROTECTED]/$(DEPDIR)/[EMAIL PROTECTED]@ @AMDEP_TRUE@@am__include@ @[EMAIL PROTECTED]/$(DEPDIR)/[EMAIL PROTECTED]@ @AMDEP_TRUE@@am__include@ @[EMAIL PROTECTED]/$(DEPDIR)/[EMAIL PROTECTED]@ diff -Nru dovecot-1.0.1/src/auth/mech-winbind.c dovecot-1.0.1-OK/src/auth/mech-winbind.c --- dovecot-1.0.1/src/auth/mech-winbind.c 1970-01-01 03:00:00.0 +0300 +++ dovecot-1.0.1-OK/src/auth/mech-winbind.c 2007-06-29 17:34:33.0 +0400 @@ -0,0 +1,269 @@ +/* + * NTLM and Negotiate authentication mechanisms, + * using Samba winbind daemon + * + * Copyrig
Re: [Dovecot] v1.1 max connections per user
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sun, 1 Jul 2007, Jürgen Herz wrote: If you think about it, also take NATed networks into account. So it would need a fairly high default and should have a remark in the comment to make admins aware. Yep! A limit per IP is not enforcable in my mind. Perhaps if you can exclude some IPs. Donnu, but I wouldn't enable a connection limit by default. Let the admin choose a correct value after "QuickStart" went well. Bye, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBRojwTC9SORjhbDpvAQL9RQf+Jc4ej0FjBlYxEtLf927iR6JFkG542SIy ustG06SoxXrJCVRJGNFO4u3yZLDcfH3eRF/agSkgxWRykVoD3B3BpEP2ivNDAOYR vKgVgdBH2eXbIq0a7jK0DPlTAyq9RUqtu8SX48lLB3ysnAzUfY20uBcMAlJUCPwR XcLiGkbLpygdtqXgvLi4+1W0zL2xFBZOXO+MVQ3dGADXQqy4Sgahx4hm6zvAIGEe Gy1LfFeFOhluh3eeyMwh8NBj8jTPTxcKhMNUeRltK7VXfs8/eGjAZ/56uNWLS7l+ tYDPmeU5shVSs+ucNinrk+b83ctYyTOU2kTVYbxGnjW0wSzK9o1n5g== =ltiB -END PGP SIGNATURE-
Re: [Dovecot] v1.1 max connections per user
On Mon, 2 Jul 2007 14:32:11 +0200 (CEST) Steffen Kaiser <[EMAIL PROTECTED]> wrote: > > Donnu, but I wouldn't enable a connection limit by default. Let the > admin choose a correct value after "QuickStart" went well. > Agreed. A per IP limit that defaults to something other than UNLIMITED and which might sneak into the system/config under the radar with 1.1 would happily break things here (all connections come from the proxy). A per user (login id) one might still affect things in an unexpected manner if uncaught in the upgrade, but at least w/o a such a totally devastating effect as with a per IP one. My 2 yen, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
[Dovecot] [Fwd: Bounce action notification]
Anyone have ideas why this happened? Nothing special in logs, all [EMAIL PROTECTED] deliveries went OK. Mailman's bounce log is unhelpful. Why does it say it tried to deliver to dovecot@dovecot.org and when it failed it disabled my account? Why doesn't it even bother saying which remote host gave an error? Grepping mail.log for 'This user' shows nothing. --- Begin Message --- This is a Mailman mailing list bounce action notice: List: dovecot Member: [EMAIL PROTECTED] Action: Subscription disabled. Reason: Excessive or fatal bounces. The triggering bounce notice is attached below. Questions? Contact the Mailman site administrator at [EMAIL PROTECTED] --- Begin Message --- Unable to deliver message to the following address(es) dovecot@dovecot.org Remote host said: 554 delivery error: This user doesn't have an account --- Original message follows. On Wed, 27 Jun 2007 23:15:32 +0300 Timo Sirainen <[EMAIL PROTECTED]> wrote: > On Thu, 2007-06-21 at 16:49 +0900, Christian Balzer wrote: > > > You could try > > > http://dovecot.org/patches/debug/mempool-accounting.diff and send > > > USR1 signal to dovecot-auth after a while. It logs how much memory > > > is used by all existing memory pools. Each auth request has its own > > > pool, so if it's really leaking them it's probably logging a lot of > > > lines. If not, then the leak is elsewhere. > > > > > I grabbed the Debian package source on a test machine (not gonna chance > > anything on the production servers), applied the patch, did add > > --enable-debug to the debian/rules file (and got the #define DEBUG > > in config.h), created the binary packages, installed, configured, > > started them, tested a few logins and... nothing gets logged > > in mail.* if I send a USR1 to dovecot-auth. Anything I'm missing? > > Bug, fixed: http://hg.dovecot.org/dovecot-1.0/rev/a098e94cd318 > Thanks, that fixed the silence of the auth-sheep. This is the output after start-up: --- Jul 2 13:59:54 engtest03 dovecot: auth(default): pool auth request handler: 104 / 4080 bytes Jul 2 13:59:54 engtest03 last message repeated 19 times Jul 2 13:59:54 engtest03 dovecot: auth(default): pool passwd_file: 56 / 10224 bytes Jul 2 13:59:54 engtest03 dovecot: auth(default): pool Environment: 224 / 2032 bytes Jul 2 13:59:54 engtest03 dovecot: auth(default): pool ldap_connection: 576 / 1008 bytes Jul 2 13:59:54 engtest03 dovecot: auth(default): pool auth: 1520 / 2032 bytes --- Used memory of dovecot-auth after 1 login was 3148KB(RSS). This is after a good trashing with rabid (from the postal package), with just 2 users though, using POP3 logins: --- Jul 2 14:12:30 engtest03 dovecot: auth(default): pool auth request handler: 104 / 4080 bytes Jul 2 14:12:30 engtest03 last message repeated 128 times Jul 2 14:12:30 engtest03 dovecot: auth(default): pool passwd_file: 56 / 10224 bytes Jul 2 14:12:30 engtest03 dovecot: auth(default): pool Environment: 224 / 2032 bytes Jul 2 14:12:30 engtest03 dovecot: auth(default): pool ldap_connection: 576 / 1008 bytes Jul 2 14:12:30 engtest03 dovecot: auth(default): pool auth: 1520 / 2032 bytes --- Note that the amount of auth request handler pools have grown to 128. After another short round of rabid the handler pools grew to 137 and the size of dovecot-auth to 5100KB. The number of handler pools never fell, nor did the memory footprint, obviously. :-p At about 800k logins/day/node here it's obvious now why dovecot-auth explodes after less than a week with max size of 512MB. > > But no matter, it is clearly leaking just as bad as 0.99 and I venture > > that his is the largest installation with LDAP as authentication > > backend. I wonder if this leak would be avoided by having LDAP lookups > > performed by worker processes as with SQL. > > Then you'd only have multiple leaking worker processes. > Yes, I realize that. But I presume since these are designed to die off and be recreated on the fly the repercussions would be much better. ;) Of course now it looks like this is not LDAP related after all. > > > The same as 0.99. You could also kill -HUP dovecot when dovecot-auth > > > is nearing the limit. That makes it a bit nicer, although not > > > perfectly safe either (should fix this some day..). > > > > > If that leak can't be found I would very much appreciate a solution > > that at least avoids failed and/or delayed logins. > > That would require that login processes don't fail logins if connection > to dovecot-auth drops, but instead wait until they can connect back to > it and try again. And maybe another alternative would be to just > disconnect the client instead of giving login failure. > Anything that fixes this one way or the other would be nice. ^_^ Oh and HUP'ing the master is not an option here, I guess the system load triggers a race condition in dovecot because several times when doing so I got this: --- Jun 22 15:08:58 mb11 dovecot: listen(143) failed: Interrupted system c
Re: [Dovecot] Leaky dovecot-auth ?
On Mon, 2007-07-02 at 15:20 +0900, Christian Balzer wrote: > Jul 2 14:12:30 engtest03 dovecot: auth(default): pool auth request handler: > 104 / 4080 bytes > Jul 2 14:12:30 engtest03 last message repeated 128 times Auth request handler is created for each imap-login connection. So if you have 128 imap-login processes this isn't a leak. Hmm. Does this help: http://hg.dovecot.org/dovecot-1.0/rev/50c79521e8f5 > Oh and HUP'ing the master is not an option here, I guess the system load > triggers a race condition in dovecot because several times when doing so > I got this: > --- > Jun 22 15:08:58 mb11 dovecot: listen(143) failed: Interrupted system call Did you use killall? I think this happens only with it. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] [Fwd: Bounce action notification]
Timo Sirainen wrote: Unable to deliver message to the following address(es) dovecot@dovecot.org Remote host said: 554 delivery error: This user doesn't have an account That isn't a terribly helpful error message, since it doesn't include the original e-mail message, with headers, so that you could see what Mailman thought the original Mail-From: address was (which is what is failing here)... John -- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
Re: [Dovecot] v1.1 max connections per user
On Sun, 2007-07-01 at 01:43 +0300, Timo Sirainen wrote: > #mail_max_user_connections = 10 Changed: protocol imap { # Maximum number of IMAP connections allowed for a user from each IP address. # NOTE: The username is compared compared case-sensitively. #mail_max_userip_connections = 10 } protocol pop3 { # Maximum number of POP3 connections allowed for a user from each IP address. # NOTE: The username is compared compared case-sensitively. #mail_max_userip_connections = 3 } signature.asc Description: This is a digitally signed message part
[Dovecot] passdb/userdb restart
Hello Timo, I've noticed that when slapd which I'm using as the passdb and the prefetched userdb is restarted I have to restart dovecot. As a matter of fact, if I don't, I get the following error : -- Jul 2 16:38:39 dovecot: imap-login: Internal login failure: user=, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=xxx.xxx.xxx.xxx, TLS Jul 2 16:38:46 dovecot: auth(default): prefetch(xxx,xxx.xxx.xxx.xxx): passdb didn't return userdb entries Jul 2 16:38:46 dovecot: auth(default): userdb(xxx,xxx.xxx.xxx.xxx): user not found from userdb -- I'm using dovecot-1.0.1 on FreeBSD 6.2-STABLE with openldap-server-2.3.35_1 (bdb backend). Would some cache-related parameters allow to fix that or will restarting dovecot be always mandatory after an ldap restart ? Thanks. -- Thomas Hummel | Institut Pasteur <[EMAIL PROTECTED]> | Pôle informatique - systèmes et réseau
Re: [Dovecot] [Fwd: Bounce action notification]
On Mon, Jul 02, 2007 at 10:51:17AM -0400, John Peacock wrote: > That isn't a terribly helpful error message, since it doesn't include the > original e-mail message, with headers, so that you could see what Mailman > thought the original Mail-From: address was (which is what is failing > here)... Actually, it does include the bounce with headers. The bounce seems to have originated from 81.3.115.182, which reverse-resolves to canville-182.adsl.newnet.co.uk. First guess is, something is seriously broken in that host. Another guess is that Mailman just scanned the bounce for subscriber addresses and found [EMAIL PROTECTED], given that the bounce wasn't formatted in any standard way. -- Antti-Juhani Kaijanaho, Jyväskylä http://antti-juhani.kaijanaho.fi/newblog/ http://www.flickr.com/photos/antti-juhani/ signature.asc Description: Digital signature
Re: [Dovecot] Leaky dovecot-auth ?
On Mon, 02 Jul 2007 17:37:05 +0300 Timo Sirainen <[EMAIL PROTECTED]> wrote: > On Mon, 2007-07-02 at 15:20 +0900, Christian Balzer wrote: > > Jul 2 14:12:30 engtest03 dovecot: auth(default): pool auth request > > handler: 104 / 4080 bytes Jul 2 14:12:30 engtest03 last message > > repeated 128 times > > Auth request handler is created for each imap-login connection. So if > you have 128 imap-login processes this isn't a leak. > At that point in time only POP3 was tried, since this is by far the most used protocol here and rabid defaults to it anyway. But there were plenty of pop3-login processes indeed. Enough to make up that number combined with the IMAP ones. Which is interesting, as this does NOT happen on the production servers, I guess rabid can dish out even more stress than my users (and cause these login processes to be left hanging around). But that's not the issue anyway, with identical pool outputs the local DB incarnation retains its size (I got an internal IMAP server with 1.0.0 and PAM and a few dozen intense users which also shows no signs of a growing dovecot-auth) while the LDAP DB one keeps growing with nothing to show for in that pool debug output. > Hmm. Does this help: http://hg.dovecot.org/dovecot-1.0/rev/50c79521e8f5 > Will try that tomorrow if I can. > > Oh and HUP'ing the master is not an option here, I guess the system > > load triggers a race condition in dovecot because several times when > > doing so I got this: > > --- > > Jun 22 15:08:58 mb11 dovecot: listen(143) failed: Interrupted system > > call > > Did you use killall? I think this happens only with it. > Nope, this is a Debian/Linux show and I did HUP just the master process. It only happened some of the times on the (then) busiest node, but it clearly is a race condition of sorts. Set up a test environment with about 30-50 logins/second and I'm sure you can reproduce it. ;) Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] [Patch] Samba's proposed "ntlm_auth + winbind" support for dovecot-auth
On Mon, 2007-07-02 at 16:19 +0400, Dmitry Butskoy wrote: > - I try to code things most close to used style, i.e. using Dovecot's > memory-management and io-pipe routines etc., but could someone look at > it and check whether I've missed something or not? close() and dup2() errors should be reported, although they're unlikely. ostream is a bit pointless there since you're doing only a single write. You could just use write_full(). > - Maybe some other options should be implemented, i.e. > "auth_winbind_helper_ntlm" and "auth_winbind_helper_spnego" to specify > the helper's cmdline exactly ? Yes. > - Currently I strip domain part of the username returned, i.e. from > "DOMAIN\user" just to "user". Maybe better add some option > "auth_winbind_strip_domain" for this? What if you changed it to [EMAIL PROTECTED] Then you could use Dovecot's standard %n or %u variables. There's one thing I'd want changed: make it non-blocking. Both input and output are currently blocking, so dovecot-auth is stuck while waiting for ntlm_auth to reply. I want to avoid this whenever possible (I don't ever want to see "authentication just gets stuck, why??" mails. "ntlm_auth timed out" message in log is much nicer). I guess ntlm_auth can handle only a single session at a time? So this would pretty much require that you either implement some kind of a queue or execute multiple ntlm_auths. Or maybe both. Using auth worker processes would probably be best. Unfortunately that currently works only for passdbs and userdbs, not for mechanisms. I guess I could try changing this for v1.1, unless you want to try? :) signature.asc Description: This is a digitally signed message part
Re: [Dovecot] [Fwd: Bounce action notification]
Antti-Juhani Kaijanaho wrote: Actually, it does include the bounce with headers. The bounce seems to have originated from 81.3.115.182, which reverse-resolves to canville-182.adsl.newnet.co.uk. Duh, of course you are right! I glanced at that block and thought that was the dovecot.org server itself generating the bounce. I concur that the above listed server is the cause of the bounce message; I've seen this before with badly written homemade spam filters... John -- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
Re: [Dovecot] [Fwd: Bounce action notification]
On Mon, 2007-07-02 at 18:10 +0300, Antti-Juhani Kaijanaho wrote: > On Mon, Jul 02, 2007 at 10:51:17AM -0400, John Peacock wrote: > > That isn't a terribly helpful error message, since it doesn't include the > > original e-mail message, with headers, so that you could see what Mailman > > thought the original Mail-From: address was (which is what is failing > > here)... > > Actually, it does include the bounce with headers. The bounce seems to have > originated from 81.3.115.182, which reverse-resolves to > canville-182.adsl.newnet.co.uk. Right, thanks. I didn't notice that. Banned the IP, hopefully it's not dynamic. :) > First guess is, something is seriously broken in that host. Another guess is > that Mailman just scanned the bounce for subscriber addresses and found > [EMAIL PROTECTED], given that the bounce wasn't formatted in any standard way. I wish Mailman supported some kind of hashes in bounce addresses. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Debian packages
Hi, > I'd love some up-to-date packages for Dovecot on Etch. I need to get > some practice in on my personal server before I migrate work... from > 0.99.14 (Still going strong, Timo :) You can get my 1.0.1 packages from http://www.mertinkat.net/dovecot/ Built for Debian etch/i386. Hope everything runs fine with the installation -- if not tell me! Greets, Moritz
Re: [Dovecot] Leaky dovecot-auth ?
On Tue, 2007-07-03 at 00:17 +0900, Christian Balzer wrote: > > > Jun 22 15:08:58 mb11 dovecot: listen(143) failed: Interrupted system > > > call > > > > Did you use killall? I think this happens only with it. Not even with it actually. I was probably thinking about something else. > Nope, this is a Debian/Linux show and I did HUP just the master process. > It only happened some of the times on the (then) busiest node, but it > clearly is a race condition of sorts. Set up a test environment with > about 30-50 logins/second and I'm sure you can reproduce it. ;) Not even with 1100 logins/sec. :) And in the code I'm already catching EINTR. Only if listen() fails 10 times with EINTR it exits with that error. And between those 10 listen() calls is a 1 second sleep. But of course if the process is getting signals all the time the sleep also fails with EINTR. But a single HUP signal shouldn't interrupt more than a single syscall, so I've really no idea what's happening. signature.asc Description: This is a digitally signed message part
[Dovecot] Force rebuild v1.0.1
Haven't seen a thread discussing forcing rebuilding of indexes / cache, so the question is, if these files in a users home directory were deleted... -rw--- 1 bb admin 80 May 17 15:05 dovecot-keywords -rw--- 1 bb admin 85449 Jul 2 06:15 dovecot-uidlist -rw--- 1 bb admin6832 Jul 2 06:15 dovecot.index -rw--- 1 bb admin 752640 Jul 2 06:19 dovecot.index.cache -rw--- 1 bb admin 95416 Jul 2 06:20 dovecot.index.log -rw--- 1 bb admin 131104 Jun 27 21:32 dovecot.index.log.2 ...would they be rebuilt by dovecot? Thank you, B. Bodger
[Dovecot] Patch for Dovecot 1.0.1 to support cross realm GSSAPI authentication
Please find enclosed two patch files (one for configure.in and one for src/mech/auth-gssapi.c) that fixes a problem in the GSSAPI code that prevented Dovecot to successfully authenticate when using cross-realm Kerberos credentials. Unfortunately I think it'll only work on Solaris (it uses the undocummented call __gss_userok() - however I think that there have been proposals on making that into an official one called gss_userok() - so perhaps some operating systems already have it? It basically calls the underlying krb5_userok() function that does the right stuff). Our setup is like this: 1. Windows XP PC that authenticates against a normal Windows 2003 AD server in the AD realm "AD.IFM.LIU.SE" 2. Solaris 10 server that authenticates against a normal Kerberos KDC in the Kerberos realm "IFM.LIU.SE" 3. Cross-realm trust between AD.IFM.LIU.SE and IFM.LIU.SE has been configured (varios steps needed - both on the Unix side and on the Windows sides (both on the AD servers and on the client). 4. On the Solaris server the /etc/krb5/krb5.conf files has been configured like this: [ libdefaults] default_realm = IFM.LIU.SE [realms] IFM.LIU.SE = { kdc = as-master.ifm.liu.se kdc = as-slave.ifm.liu.se admin_server = as-master.ifm.liu.se auth_to_local_realm = AD.IFM.LIU.SE } AD.IFM.LIU.SE = { kdc = ad-master.ad.ifm.liu.se kdc = ad-slave.ad.ifm.liu.se admin_server = ad-master.ifm.liu.se kpasswd_protocol = SET_CHANGE } 5. I use Thunderbird 2 on the PC to connect to the Dovecot server with the AD-aquired credentials and with the patch above it works correctly. Without the patch then Dovecot will fail since the code that does the GSSAPI authentication fails at gss_compare_names(). (that code only works if the credentials used are from the local realm) (The cross-realm setup also works with Quest Putty so you can do passwordless SSH logins from your AD-connected PC to the Unix servers, and get a delegated AD ticket so that Secure NFS also works). Storing of delegated tickets is something that my patch above doesn't solve. It shouldn't be that hard to implement though - you basically just have to call gss_store_cred() at the right place in Dovecot (when you've switched to the right userid, and preferable after having called some PAM setup functions) - can be useful if you ever want to run Dovecot and access a Secure NFS protected remote filesystem... - Peter --- dovecot-1.0.1/configure.in Thu Jun 14 14:01:28 2007 +++ dovecot-1.0.1-ifm/configure.in Mon Jul 2 17:45:47 2007 @@ -1484,10 +1484,15 @@ # have gssapi.h old_CFLAGS=$CFLAGS CFLAGS="$CFLAGS `krb5-config --cflags gssapi`" + old_LIBS=$LIBS + LIBS="$LIBS `krb5-config --libs gssapi`" AC_CHECK_HEADER([gssapi/gssapi.h], [ AC_DEFINE(HAVE_GSSAPI_GSSAPI_H,, GSSAPI headers in gssapi/gssapi.h) have_gssapi=yes ]) + AC_CHECK_HEADER([gssapi/gssapi_ext.h], [ +AC_DEFINE(HAVE_GSSAPI_GSSAPI_EXT_H,, GSSAPI headers in gssapi/gssapi_ext.h) + ]) AC_CHECK_HEADER([gssapi.h], [ AC_DEFINE(HAVE_GSSAPI_H,, GSSAPI headers in gssapi.h) have_gssapi=yes @@ -1494,8 +1499,13 @@ ]) if test $have_gssapi = yes; then AC_DEFINE(HAVE_GSSAPI,, Build with GSSAPI support) +AC_CHECK_LIB(gss, __gss_userok, [ + AC_DEFINE(HAVE___GSS_USEROK,, + Define if you have __gss_userok()) +]) fi CFLAGS=$old_CFLAGS + LIBS=$old_LIBS fi fi fi --- dovecot-1.0.1/src/auth/mech-gssapi.c Sat May 19 13:14:04 2007 +++ dovecot-1.0.1-ifm/src/auth/mech-gssapi.c Mon Jul 2 17:51:23 2007 @@ -29,6 +29,10 @@ # include #endif +#ifdef HAVE_GSSAPI_GSSAPI_EXT_H +# include +#endif + /* Non-zero flags defined in RFC */ enum sasl_gssapi_qop { SASL_GSSAPI_QOP_UNSPECIFIED = 0x00, @@ -163,8 +167,13 @@ name_buf.length = len; major_status = gss_import_name(&minor_status, &name_buf, +#if 0 /* Added 070702 Peter Eriksson <[EMAIL PROTECTED]> - ref cyrus-sasl Solaris 8/9 has problems with NO_OID */ + GSS_C_NT_USER_NAME, +#else GSS_C_NO_OID, +#endif &name); + if (GSS_ERROR(major_status)) { auth_request_log_gss_error(request, major_status, GSS_C_GSS_CODE, "gss_import_name"); @@ -174,6 +183,7 @@ return name; } + static void gssapi_sec_context(struct gssapi_auth_request *request, gss_buffer_desc inbuf) { @@ -273,7 +283,9 @@ OM_uint32 major_status, minor_status; gss_buffer_desc outbuf; int equal_authn_authz = 0; + char *name; + major_status = gss_unwrap(&minor_status, request->gss_ctx, &inbuf, &outbuf, NULL, NULL); @@ -292,6 +304,44 @@ return; } +#ifdef HAVE___GSS_USEROK + /* __gss_userok() correctly handles cross-realm authentication, whereas the original code + does not... */ + + name = p_strndup(request->auth_request
Re: [Dovecot] [Patch] Samba's proposed "ntlm_auth + winbind" support for dovecot-auth
Timo Sirainen wrote: - Currently I strip domain part of the username returned, i.e. from "DOMAIN\user" just to "user". Maybe better add some option "auth_winbind_strip_domain" for this? What if you changed it to [EMAIL PROTECTED] Then you could use Dovecot's standard %n or %u variables. AFAIK "[EMAIL PROTECTED]" should have an actual form of "[EMAIL PROTECTED]", but "ntlm_auth" returns "EXAMPLE\user" in such a case, not "EXAMPLE.COM\user". At least for NTLM (against AD). OTOH the spnego could retrurn the actual "[EMAIL PROTECTED]", but it is still unknown to me :) There's one thing I'd want changed: make it non-blocking. Both input and output are currently blocking, so dovecot-auth is stuck while waiting for ntlm_auth to reply. I want to avoid this whenever possible (I don't ever want to see "authentication just gets stuck, why??" mails. "ntlm_auth timed out" message in log is much nicer). I guess ntlm_auth can handle only a single session at a time? No. Normally ntlm_auth invoked at once for ALL authentication sessions. It is a child process which lives all the time while the dovecot-auth is alive. There's one thing I'd want changed: make it non-blocking Still actual after the clarification above?.. So this would pretty much require that you either implement some kind of a queue Currently I prefer to use blocking io, which provides such "a queue" de-facto. At least for initial implementation. (I hope "ntlm_auth" is fast enough). But some timeout surely could be useful. How can it be implemented? or execute multiple ntlm_auths. Does the use of worker for, say, PAM userdb affects us here too? Or maybe both. Using auth worker processes would probably be best. Unfortunately that currently works only for passdbs and userdbs, not for mechanisms. I guess I could try changing this for v1.1, unless you want to try? :) It seems that I prefer you try it... :) Does "blocking io" + "worker" look like the best way? Regards Dmitry Butskoy
Re: [Dovecot] Patch for Dovecot 1.0.1 to support cross realm GSSAPI authentication
Oh, another "hack" that's needed to build Dovecot on Solaris 10 with the native GSS library is a very simple, dummy, "krb5-config" script. Please find it also enclosed (probably not needed when Solaris 10 update 4 is released). - Peter Peter Eriksson wrote: Please find enclosed two patch files (one for configure.in and one for src/mech/auth-gssapi.c) that fixes a problem in the GSSAPI code that - Peter #!/bin/ksh while [ "$1" != "" ]; do case "$1" in --version) echo "1.4" ;; --libs) case "$2" in gssapi) echo "-lgss" shift ;; esac ;; --cflags) case "$2" in gssapi) echo "" shift ;; esac ;; esac shift done exit 0 begin:vcard fn:Peter Eriksson n:Eriksson;Peter org;quoted-printable:Link=C3=B6ping University;Physics Department adr;quoted-printable:;;;Link=C3=B6ping;;SE-58183;Sweden email;internet:[EMAIL PROTECTED] title:Computer Systems Manager tel;work:+46-13-282786 tel;fax:+46-13-137568 tel;home:+46-13-55255 tel;cell:+46-705-182786 x-mozilla-html:FALSE url:http://people.ifm.liu.se/peter version:2.1 end:vcard
Re: [Dovecot] [Patch] Samba's proposed "ntlm_auth + winbind" support for dovecot-auth
On Mon, 2007-07-02 at 20:19 +0400, Dmitry Butskoy wrote: > Timo Sirainen wrote: > >> - Currently I strip domain part of the username returned, i.e. from > >> "DOMAIN\user" just to "user". Maybe better add some option > >> "auth_winbind_strip_domain" for this? > >> > > > > What if you changed it to [EMAIL PROTECTED] Then you could use Dovecot's > > standard %n or %u variables. > > > > AFAIK "[EMAIL PROTECTED]" should have an actual form of "[EMAIL PROTECTED]", > but "ntlm_auth" returns "EXAMPLE\user" in such a case, not > "EXAMPLE.COM\user". At least for NTLM (against AD). > OTOH the spnego could retrurn the actual "[EMAIL PROTECTED]", but it is > still unknown to me :) I don't think it's a problem even if it returned only [EMAIL PROTECTED] %d then just expands to example. At least I think it's better than not having the domain at all. > > There's one thing I'd want changed: make it non-blocking. Both input and > > output are currently blocking, so dovecot-auth is stuck while waiting > > for ntlm_auth to reply. I want to avoid this whenever possible (I don't > > ever want to see "authentication just gets stuck, why??" mails. > > "ntlm_auth timed out" message in log is much nicer). > > > > I guess ntlm_auth can handle only a single session at a time? > > No. I mean simultaneously. Like you can't send multiple "begin/continue authentication" commands to it and then just wait until one of them finishes? > > There's one thing I'd want changed: make it non-blocking > > Still actual after the clarification above?.. Yep. > > So this > > would pretty much require that you either implement some kind of a queue > > > > Currently I prefer to use blocking io, which provides such "a queue" > de-facto. At least for initial implementation. (I hope "ntlm_auth" is > fast enough). I think the problem isn't ntlm_auth itself, but that it also has to talk to AD. So any network problems there could leave it hanging. > But some timeout surely could be useful. How can it be implemented? By using non-blocking I/O :) > > or execute multiple ntlm_auths. > > Does the use of worker for, say, PAM userdb affects us here too? PAM is passdb. But anyway it shouldn't affect. > > Or maybe both. Using auth worker > > processes would probably be best. Unfortunately that currently works > > only for passdbs and userdbs, not for mechanisms. I guess I could try > > changing this for v1.1, unless you want to try? :) > > > > It seems that I prefer you try it... :) > > Does "blocking io" + "worker" look like the best way? I think so. Or some kind of a generic queue maybe.. Hmm. Doesn't GSSAPI have the exact same problem? I think it does. Maybe I could figure out something for them both. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Patch for Dovecot 1.0.1 to support cross realm GSSAPI authentication
On Mon, 2007-07-02 at 18:08 +0200, Peter Eriksson wrote: > > + AC_CHECK_LIB(gss, __gss_userok, [ > + AC_DEFINE(HAVE___GSS_USEROK,, > + Define if you have > __gss_userok()) > + ]) Instead of changing $LIBS, I think you can use: AC_CHECK_LIB(gss, __gss_userok, [ ... ],, `krb5-config --libs gssapi`) > +#if 0 /* Added 070702 Peter Eriksson <[EMAIL PROTECTED]> - ref cyrus-sasl > Solaris 8/9 has problems with NO_OID */ > + GSS_C_NT_USER_NAME, > +#else >GSS_C_NO_OID, > +#endif Is this a common problem? Should it be a setting? Although with your change this code is never even executed, right? > + name = p_strndup(request->auth_request.pool, > +(unsigned char *)outbuf.value + 4, > +outbuf.length - 4); > + > + if (!name) { > + auth_request_log_error(&request->auth_request, > "gssapi", > + "Invalid response size"); > + auth_request_fail(&request->auth_request); > + return; > + } name is never NULL. > + free(name); And you shouldn't free() it either. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] [Patch] Samba's proposed "ntlm_auth + winbind" support for dovecot-auth
Timo Sirainen wrote: On Mon, 2007-07-02 at 20:19 +0400, Dmitry Butskoy wrote: Timo Sirainen wrote: - Currently I strip domain part of the username returned, i.e. from "DOMAIN\user" just to "user". Maybe better add some option "auth_winbind_strip_domain" for this? What if you changed it to [EMAIL PROTECTED] Then you could use Dovecot's standard %n or %u variables. AFAIK "[EMAIL PROTECTED]" should have an actual form of "[EMAIL PROTECTED]", but "ntlm_auth" returns "EXAMPLE\user" in such a case, not "EXAMPLE.COM\user". At least for NTLM (against AD). OTOH the spnego could retrurn the actual "[EMAIL PROTECTED]", but it is still unknown to me :) I don't think it's a problem even if it returned only [EMAIL PROTECTED] %d then just expands to example. At least I think it's better than not having the domain at all. I propose to have an option. For example, winbind already has "winbind use default domain" option, which cause (sometimes :) ) it to return usernames without domain part. It is useful at UNIX machines, where the actual UNIX username is just "user", not "DOMAIN\user" . What happens when dovecot receives "DOMAIN\user" or "[EMAIL PROTECTED]" form? How it determine then the local UNIX username? There's one thing I'd want changed: make it non-blocking. Both input and output are currently blocking, so dovecot-auth is stuck while waiting for ntlm_auth to reply. I want to avoid this whenever possible (I don't ever want to see "authentication just gets stuck, why??" mails. "ntlm_auth timed out" message in log is much nicer). I guess ntlm_auth can handle only a single session at a time? No. I mean simultaneously. Like you can't send multiple "begin/continue authentication" commands to it and then just wait until one of them finishes? Oh, yep. You're right, only one at a time. So this would pretty much require that you either implement some kind of a queue Currently I prefer to use blocking io, which provides such "a queue" de-facto. At least for initial implementation. (I hope "ntlm_auth" is fast enough). I think the problem isn't ntlm_auth itself, but that it also has to talk to AD. So any network problems there could leave it hanging. OTOH winbind daemon can "cache" something now... Hmm.. Apache seems to use blocking-io. I'll look later for Squid. or execute multiple ntlm_auths. Does the use of worker for, say, PAM userdb affects us here too? PAM is passdb. But anyway it shouldn't affect. I mean that when we use worker for some userdb, we actually have several dovecot-auth processes, hence several ntlm_auth as well. Am I right? ~buc
Re: [Dovecot] Force rebuild v1.0.1
On Mon, 2007-07-02 at 06:23 -0400, Bruce Bodger wrote: > Haven't seen a thread discussing forcing rebuilding of indexes / > cache, so the question is, if these files in a users home directory > were deleted... > > -rw--- 1 bb admin 80 May 17 15:05 dovecot-keywords > -rw--- 1 bb admin 85449 Jul 2 06:15 dovecot-uidlist > -rw--- 1 bb admin6832 Jul 2 06:15 dovecot.index > -rw--- 1 bb admin 752640 Jul 2 06:19 dovecot.index.cache > -rw--- 1 bb admin 95416 Jul 2 06:20 dovecot.index.log > -rw--- 1 bb admin 131104 Jun 27 21:32 dovecot.index.log.2 > > ...would they be rebuilt by dovecot? Sure. Although you shouldn't delete dovecot-uidlist and dovecot-keywords, they're not indexes. http://wiki.dovecot.org/MailLocation/Maildir signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Patch for Dovecot 1.0.1 to support cross realm GSSAPI authentication
Timo Sirainen skrev: On Mon, 2007-07-02 at 18:08 +0200, Peter Eriksson wrote: + AC_CHECK_LIB(gss, __gss_userok, [ + AC_DEFINE(HAVE___GSS_USEROK,, + Define if you have __gss_userok()) + ]) Instead of changing $LIBS, I think you can use: AC_CHECK_LIB(gss, __gss_userok, [ ... ],, `krb5-config --libs gssapi`) Ok, I'll try that instead (I used the changing stuff since similar stuff was done earlier (with CFLAGS) :-) Btw. I think that the Heimdal implementation of GSSAPI has a similar function but called gss_userok() (without the leading two "_" characters) that does something similar - and users of Heimdal around that can verify or deny that? +#if 0 /* Added 070702 Peter Eriksson <[EMAIL PROTECTED]> - ref cyrus-sasl Solaris 8/9 has problems with NO_OID */ + GSS_C_NT_USER_NAME, +#else GSS_C_NO_OID, +#endif Is this a common problem? Should it be a setting? Although with your change this code is never even executed, right? Dunno actually - When I was reading the Cyrus-SASL GSSAPI-code I saw that they have a configure-check for GSS_C_NT_USER_NAME in their code base and a comment that Solaris 8/9 doesn't like GSS_C_NO_OID in that call so at first I thought that was the reason things didn't work - however it wasn't the problem I later figured out... Anyway, as you can see I disabled that change (and with the _gcc_userid() code it's not used anyway). + name = p_strndup(request->auth_request.pool, +(unsigned char *)outbuf.value + 4, +outbuf.length - 4); + + if (!name) { + auth_request_log_error(&request->auth_request, "gssapi", + "Invalid response size"); + auth_request_fail(&request->auth_request); + return; + } name is never NULL. Ok, I'll remove those checks. + free(name); And you shouldn't free() it either. And I'll remove the free() calls too... :-) I've uploaded new versions of the patches to: ftp://ftp.ifm.liu.se/pub/unix/dovecot/ I've also put a Solaris 10 SMF manifest there (dovecot.xml). Btw, is it intentional that the first line in configure.in starts with an "x"? - Peter
Re: [Dovecot] Force rebuild v1.0.1
On Jul 2, 2007, at 1:25 PM, Timo Sirainen wrote: On Mon, 2007-07-02 at 06:23 -0400, Bruce Bodger wrote: Haven't seen a thread discussing forcing rebuilding of indexes / cache, so the question is, if these files in a users home directory were deleted... -rw--- 1 bb admin 80 May 17 15:05 dovecot-keywords -rw--- 1 bb admin 85449 Jul 2 06:15 dovecot-uidlist -rw--- 1 bb admin6832 Jul 2 06:15 dovecot.index -rw--- 1 bb admin 752640 Jul 2 06:19 dovecot.index.cache -rw--- 1 bb admin 95416 Jul 2 06:20 dovecot.index.log -rw--- 1 bb admin 131104 Jun 27 21:32 dovecot.index.log.2 ...would they be rebuilt by dovecot? Sure. Although you shouldn't delete dovecot-uidlist and dovecot-keywords, they're not indexes. http://wiki.dovecot.org/MailLocation/Maildir Thank you, Timo. One more thing: would you explain the existence of this file: -rw--- 1 bb admin 131104 Jun 27 21:32 dovecot.index.log.2 I can understand dovecot-keywords having the old creation date but dovecot.index.log.2 being old makes me think it's no longer needed. Under what conditions is it created and should it be kept around? Thanks, B. Bodger
Re: [Dovecot] Patch for Dovecot 1.0.1 to support cross realm GSSAPI authentication
On Mon, 2007-07-02 at 19:55 +0200, Peter Eriksson wrote: > Btw. I think that the Heimdal implementation of GSSAPI has a similar > function but called gss_userok() (without the leading two "_" > characters) that does something similar - and users of Heimdal around > that can verify or deny that? I don't see anything like that in Debian's heimdal-dev package (0.7.2.dfsg.1-10). I do see gss_userok() in GNU's GSSLib though: http://www.gnu.org/software/gss/manual/html_node/Extended-GSS-API.html > ftp://ftp.ifm.liu.se/pub/unix/dovecot/ Committed with small changes: http://hg.dovecot.org/dovecot/rev/dfdedb187b26 ..and.. http://hg.dovecot.org/dovecot/rev/159929f53161 > Btw, is it intentional that the first line in configure.in starts with > an "x"? No. Thanks, removed. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Force rebuild v1.0.1
On Mon, 2007-07-02 at 14:15 -0400, Bruce Bodger wrote: > Thank you, Timo. One more thing: would you explain the existence of > this file: > > -rw--- 1 bb admin 131104 Jun 27 21:32 dovecot.index.log.2 > > I can understand dovecot-keywords having the old creation date but > dovecot.index.log.2 being old makes me think it's no longer needed. > Under what conditions is it created and should it be kept around? When .log gets full, it's rotated to .log.2. If there are multiple connections open to the same mailbox, it's useful for next few minutes after the rotation. After that it's not very useful. I've been thinking about getting it deleted automatically at some point, but it's not done yet. signature.asc Description: This is a digitally signed message part
[Dovecot] userdb nss
I remembered people complaining how glibc ignores NSS errors with getpwent(), so I thought I'd see how easy it is to use NSS directly. Unless I misunderstood something, this should work: http://hg.dovecot.org/dovecot/rev/c9b49ed99d28 Tested only with "files" backend. Also it won't work with Solaris, it seems to have completely different NSS modules. Don't know about BSDs. Can someone test how it handles LDAP/NIS failures (or if it even works with them)? signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Semi-static userdb...?
On Tue, 2007-05-15 at 12:24 +0300, Timo Sirainen wrote: > On 15.5.2007, at 5.16, John Robinson wrote: > > >> One possibility would be to set "uid_file=/vmail/%d gid_file=/ > >> vmail/%d". > >> I guess that would be good. Added to TODO, but I'm not sure when I > >> get > >> around to implementing it. > > > > Something like the attached? > > Otherwise it's OK, but I'd want it to work with all userdbs. Looks > like the code doesn't currently support doing that in any easy way. > With passdbs it'd have been easy to use auth_request_set_field(). I > guess I'll add a similar auth_request_add_userdb_field() for CVS HEAD. Added uidgid_file to v1.1 tree: http://hg.dovecot.org/dovecot/rev/f7cdede45a88 signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Leaky dovecot-auth ?
On Mon, 02 Jul 2007 17:37:05 +0300 Timo Sirainen <[EMAIL PROTECTED]> wrote: > > Hmm. Does this help: http://hg.dovecot.org/dovecot-1.0/rev/50c79521e8f5 > We have a winner! Auth process grows to the same size as with a local DB and stays there. Now I just have to get this into a security maintained Debian package... (looks around for the official package masters and backports maintainers) Am I correct in assuming that this code did not change since 0.99, read that the leak I saw for the last 4 years was the same thing? ;) Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] Leaky dovecot-auth ?
On Tue, 2007-07-03 at 11:08 +0900, Christian Balzer wrote: > On Mon, 02 Jul 2007 17:37:05 +0300 Timo Sirainen <[EMAIL PROTECTED]> wrote: > > > > Hmm. Does this help: http://hg.dovecot.org/dovecot-1.0/rev/50c79521e8f5 > > > We have a winner! > Auth process grows to the same size as with a local DB and stays there. > > Now I just have to get this into a security maintained Debian package... > (looks around for the official package masters and backports maintainers) > > Am I correct in assuming that this code did not change since 0.99, read > that the leak I saw for the last 4 years was the same thing? ;) Yea, I had never known about ber_free() before. Also I hadn't noticed auth process growing in my LDAP stress tests either, so I thought the problem was something else. signature.asc Description: This is a digitally signed message part