Hi folks! I'm glad that so many people are finally involved in our "file locking discussion". However, we had lots of discussions about this topic in the past and we have no results, yet.
That's why I want to try to bring more "structure" into the discussion. (Please complain if you don't agree :-) And someone else mentioned that we should write a little assey about the possible decision we'll draw soon, so that newcomers are informed why we have chosen a specific locking mechanism. Thus, I started to write such a document. It's attached below. Please note, that this is release 0.00001 of the document so feel free to tell me _any_ comments/critics/suggestions. I'll try to update this document often and introduce all new arguments that will show up here. Perhaps we can come to a result this time, and have the documentation ready when we have a decision. (Please do not quote the whole document if you are referring to a single section to reduce the traffic on this list.) -------------------------------------------------------------------------- File Locking in the Debian system ================================= Table of Contents ----------------- 0. References 1. Current situation 2. Locking mechanisms 3. Debian specific ideas 4. Open issues 5. Frequently Asked Questions Chapter 0: References --------------------- - The open(2) describes "dot-locking" and a NFS safe locking mechanism. - http://www.swb.de/personal/okir/lockd.html explains rpc.lockd - several discussions on debian-devel (starting from Mar 97) - Lars' "publib" library (see publib Debian package) - Karl's "LockFile.pm"--a Perl implementation of publib's lockfile (see http://inetarena.com/~karlheg/Public) Chapter 1: Current situation ---------------------------- 1.1 Debian Policy Manual The current Debian Poliy Manual (2.1.3.3) only mentions file locking for MUAs (mail-user-agents) and MTAs (mail-transport-agents). In section 4.3 "Mail processing on Debian systems" it says: Mailboxes are locked using the username.lock lockfile convention, rather than fcntl, flock or lockf. Note, that this is obviously not enough information for the maintainers to adopt their packages. This results in incompatible locking mechanism. Thus, sometimes problem occur on Debian systems with mail locking when using NFS. 1.2 Where do we need locking? We need file locking in several places. For example, the incoming mail boxes in /var/spool/mail/*, mail folders that are accessed by "procmail" and "pine", etc. Chapter 2: Locking mechanisms ----------------------------- 2.1 fcntl, flock, lockf ----------------------- The Linux kernel provides some system calls to do file locking. However, neither of these functions is working over NFS. 2.2 dot-locking --------------- To perform dot-locking, create the file <mailbox-name>.lock using open(name, O_CREAT|O_EXCL, 0644). Create with O_EXCL will fail if the file exists. If you can't create it, sleep for a while and try again. To unlock, unlink() the file. This technique is not NFS safe, since the NFS "open" function is not atomic. The man page open(2) reads: O_EXCL When used with O_CREAT, if the file already exists it is an error and the open will fail. O_EXCL is broken on NFS file systems, programs which rely on it for performing locking tasks will contain a race condition. 2.3 rcp.lockd ------------- Please refer to http://www.swb.de/personal/okir/lockd.html for a more detailed description. The idea is simple: Each "server host" runs a rcp.lockd daemon. When a process wants to lock a file, it contacts that daemon and asks for locking the file. This technique is obviously NFS safe. However, there is no Linux implementation of rpc.lockd, yet. 2.4 SUN's lockd --------------- (Is this the same as "rpc.lockd"?) Rumors say, that "lockd" can cause dead-locks when unlocking a file. And there is no Linux port of this daemon, yet. 2.5 NFS safe locking through "link" ----------------------------------- The man page open(2) reads: The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile and use stat(2) on the unique file to check if its link count has increased to 2. Do not use the return value of the link() call. This method is currently implemented in Lars' "publib" and has been ported to a Perl module by Karl. There is no documentation available for "publib" yet, but this could easily be fixed :-) 2.6 Other implementations ------------------------- A few packages install binaries that handle the locking of files: `lockfile' --> procmail `shlock' --> innd `newslock' --> mgetty-fax If we should choose supporting one of these locking mechanisms, we would have to change these binaries into shared libraries, for efficiency. Chapter 3: Debian specific ideas -------------------------------- 3.1 Creating a library When we have chosen a good locking mechanism we should implement it in a shared library so that the maintainers do not have to hand-code this themselves. (Since these techniques are a bit "hairy" that would likely introduce new bugs.) We'll also provide "modules" for Perl, Python, and other popular programming languages. In addition, we'll provide a "binary" that can be called by shell scripts or other languages to lock/unlock files. 3.2 Creating an upstream library Since there is no standard way of file locking yet, we should try to build a new "upstream library". That is, we don't want to create a Debian specific library, but one that can be used by the other distributions as well. Perhaps this library can become the new "standard" in the future. Perhaps, we could merge this into the new "glibc 2.0". That way, it will surely become the new standard! 3.3 Policy Manual The Policy Manual should document the "Right Way of File Locking" in pseudo-code. In addition, there will be a short manual (actually, this document!) describing the different locking techniques that were discussed. Chapter 4: Open issues ---------------------- Who releases a lock when the process, who locked the file, has died? Chapter 5: Frequently Asked Questions ------------------------------------- Q: What's wrong with using dot-locking in conjunction with fcntl/flock? A: Provided that everybody enables the two locks in the same order and disables them in the opposite order, there is no problem in using two different locking mechanisms. However, we wouldn't get any advantages from using two mechanisms at the same time. Q: Why can't we simply fix the Linux kernel to support NFS safe locking? A: A lot of people use Debian in a network with other UNICES. As such, we'll have to choose a locking mechanism which is implemented at "application level" so that the involved programs (mail daemons, mail clients, etc.) on other platforms can be fixed to apply to our mechanism, as well. -------------------------------------------------------------------------- -- Christian Schwarz Do you know [EMAIL PROTECTED], [EMAIL PROTECTED], Debian GNU/Linux? [EMAIL PROTECTED], [EMAIL PROTECTED] Visit PGP-fp: 8F 61 EB 6D CF 23 CA D7 34 05 14 5C C8 DC 22 BA http://www.debian.org http://fatman.mathematik.tu-muenchen.de/~schwarz/ -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
