Re: why are there /bin and /usr/bin...

[Ted Ts'o]

On Mon, Aug 16, 2010 at 09:01:42PM +0200, Bernhard R. Link wrote:
> * Perry E. Metzger  [100816 20:21]:
> > The most reasonable argument against altering such things is that
> > after decades, people are used to the whole /usr thing and the fight
> > to change it isn't worthwhile. That I will agree with -- see the
> > emotional reactions people get when you suggest their preferred layout
> > is an "onion".
> 
> Accusion people of irrational behaviour almost always results in
> irrational behviour. Either they were irrational already before or
> making false insulting accusations. So I should better not tell you
> that accusing people of irrational behaviour is quite irrational...

There is a rational reason for doing this at least for servers.
Having a small root partition can be a huge advantage because it
minimizes the chances that it will get corrupted.  Making the root
read-only is even better from that perspective (but generally requires
more work).  What I like to do for servers that absolutely, positively
can't go down, and for which I don't have redudant servers (mainly
because I'm too poor :-) is to have a root partition which is small
(say, half a gig) and then mirror it onto another partition on a
separate spindle, and set up grub so I can boot off of either root
partition.  (If the BIOS has a way for me to specify via a serial
console booting off of the 2nd hard drive, even better; then I can
have a duplicate grub setup on the 2nd hard drive as well.)

I used to do this for desktops as well, but these days, a rescue CD is
easy enough to use.

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100819005027.ga16...@thunk.org

Re: Bug#605009: serious performance regression with ext4

[Ted Ts'o]

On Fri, Nov 26, 2010 at 03:53:27PM +0100, Raphael Hertzog wrote:
> Just to sum up what dpkg --unpack does in 1.15.8.6:
> 1/ set the package status as half-installed/reinst-required
> 2/ extract all the new files as *.dpkg-new
> 3/ for all the unpacked files: fsync(foo.dpkg-new) followed by
>rename(foo.dpkg-new, foo)

What are you doing?

1) Suppose package contains files "a", "b", and "c".  Which are you
doing?

a)  extract a.dpkg-new ; fsync(a.dpkg-new); rename(a.dpkg-new, a);
extract b.dpkg-new ; fsync(b.dpkg-new); rename(b.dpkg-new, b);
extract c.dpkg-new ; fsync(c.dpkg-new); rename(c.dpkg-new, c);

or

b)  extract a.dpkg-new ; fsync(a.dpkg-new);
extract b.dpkg-new ; fsync(b.dpkg-new);
extract c.dpkg-new ; fsync(c.dpkg-new);
rename(a.dpkg-new, a);
rename(b.dpkg-new, b);
rename(c.dpkg-new, c);

or

c)  extract(a.dpkg-new);
extract(b.dpkg-new);
extract(c.dpkg-new);
fsync(a.dpkg-new);
fsync(b.dpkg-new);
fsync(c.dpkg-new);
rename(a.dpkg-new, a);
rename(b.dpkg-new, b);
rename(c.dpkg-new, c);


(c) will perform the best for most file systems, including ext4.  As a
further optimization, if "b" and "c" does not exist, of course it
would be better to extract into "b" and "c" directly, and skip the
rename, i.e.:

d)  extract(a.dpkg-new);
extract(b); # assuming the file "b" does not yet exist
extract(c); # assuming the file "c" does not yet exist
fsync(a.dpkg-new);
fsync(b);
fsync(c);
rename(a.dpkg-new, a);

... and then set the package status as unpacked.

I am guessing you are doing (a) today --- am I right?  (c) or (d)
would be best.

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20101126215254.gj2...@thunk.org

Re: Bug#605009: serious performance regression with ext4

[Ted Ts'o]

I did some experimenting, and I figured out what was going on.  You're
right, (c) doesn't quite work, because delayed allocation meant that
the writeout didn't take place until the fsync() for each file
happened.  I didn't see this at first; my apologies.

However, this *does* work:

extract(a);
sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WRITE); 
extract(b.dpkg-new);
sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WRITE); 
extract(c.dpkg-new);
sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WRITE); 

sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 

fdatasync(a);
fdatasync(b.dpkg-new);
fdatasync(c.dpkg-new);

rename(b.dpkg-new, b);
rename(c.dpkg-new, c);

This assumes that files b and c existed beforehand, but a is a new file.

What's going on here?  sync_file_range() is a Linux specific system
call that has been around for a while.  It allows program to control
when writeback happens in a very low-level fashion.  The first set of
sync_file_range() system calls causes the system to start writing back
each file once it has finished being extracted.  It doesn't actually
wait for the write to finish; it just starts the writeback.

The second series of sync_file_range() calls, with the operation
SYNC_FILE_RANGE_WAIT_BEFORE, will block until the previously initiated
writeback has completed.  This basically ensures that the delayed
allocation has been resolved; that is, the data blocks have been
allocated and written, and the inode updated (in memory), but not
necessarily pushed out to disk.

The fdatasync() call will actually force the inode to disk.  In the
case of the ext4 file system, the first fdatasync() will actually push
all of the inodes to disk, and all of the subsequent fdatasync() calls
are in fact no-ops (assuming that files 'a', 'b', and 'c' are all on
the same file system).  But what it means is that it minimizes the
number of (heavyweight) jbd2 commits to a minimum.

It uses a linux-specific system call --- sync_file_range --- but the
result should be faster performance across the board for all file
systems.  So I don't consider this an ext4-specific hack, although it
probably does makes things faster for ext4 more than any other file
system.

I've attached the program I used to test and prove this mechanism, as
well as the kernel tracepoint script I used to debug why (c) wasn't
working, which might be of interest to folks on debian-kernel.
Basically it's a demonstration of how cool ftrace is.  :-)

But using this program on a file system composed of a 5400rpm laptop
drive running LVM and LUKS, I get:

mass-sync-tester -d:dpkg current: time:  0.83/ 0.01/ 0.00

versus

mass-sync-tester -n:dpkg fixed: time:  0.07/ 0.00/ 0.01

   - Ted

/*
 * Mass sync tester
 */

#define _GNU_SOURCE

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

void write_file(const char *name, int sync, int sync_range)
{
	int	fd, i, ret;
	char	buf[1024];

	fd = open(name, O_WRONLY|O_TRUNC|O_CREAT, 0666);
	if (fd < 0) {
		fprintf(stderr, "open(%s) in write_file: %s\n",
			name, strerror(errno));
		exit(1);
	}
	memset(buf, 0, sizeof(buf));
	for (i=0; i < 16; i++) {
		ret = write(fd, buf, sizeof(buf));
		if (ret < 0) {
			fprintf(stderr, "writing %s: %s\n",
name, strerror(errno));
			exit(1);
		}
	}
	if (sync) {
		ret = fsync(fd);
		if (ret < 0) {
			fprintf(stderr, "fsyncing %s in write_file: %s\n",
name, strerror(errno));
			exit(1);
		}
	}
	if (sync_range) {
		ret = sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WRITE);
		if (ret < 0) {
			fprintf(stderr, "sync_file_range %s in write_file: %s\n",
name, strerror(errno));
			exit(1);
		}
	}
	ret = close(fd);
	if (ret < 0) {
		fprintf(stderr, "closing %s in write_file: %s\n",
			name, strerror(errno));
		exit(1);
	}
}

void rename_file(const char *src, const char *dest)
{
	int ret;

	ret = rename(src, dest);
	if (ret) {
		fprintf(stderr, "renaming %s to %s: %s\n", src, dest,
			strerror(errno));
		exit(1);
	}
}

void sync_file(const char *name)
{
	int	fd, i, ret;

	fd = open(name, O_RDONLY|O_NOATIME, 0666);
	if (fd < 0) {
		fprintf(stderr, "open(%s) in sync_file: %s\n",
			name, strerror(errno));
		exit(1);
	}
	ret = fsync(fd);
	if (ret < 0) {
		fprintf(stderr, "fsyncing %s in sync_file: %s\n",
			name, strerror(errno));
		exit(1);
	}
	ret = close(fd);
	if (ret < 0) {
		fprintf(stderr, "closing %s in sync_file: %s\n",
			name, strerror(errno));
		exit(1);
	}
}

void datasync_file(const char *name)
{
	int	fd, i, ret;

	fd = open(name, O_RDONLY|O_NOATIME, 0666);
	if (fd < 0) {
		fprintf(stderr, "open(%s) in datasync_file: %s\n",
			name, strerror(errno));
		exit(1);
	}
	ret = fdatasync(fd);
	if (ret < 0) {
		fprintf(stderr, "fsyncing %s in datasync_file: %s\n",
			name, strerror(errno));
		exi

Re: Bug#605009: serious performance regression with ext4

[Ted Ts'o]

On Mon, Nov 29, 2010 at 02:58:16PM +, Ian Jackson wrote:
> 
> This is the standard way that ordinary files for which reliability was
> important have been updated on Unix for decades.  fsync is for files
> which need synchronisation with things external to the computer (or at
> least, external to the volume) - eg, email at final dot.

This is simply not true.  And I'm speaking as someone who has been
doing Unix/Linux kernel development since the BSD 4.3 days.  (Well,
BSD 4.3+Tahoe days, to be precise.)

fsync() has always been the only guarantee that files would be on
disk.  In fact the way BSD worked, there was no guarantee that
rename() would provide any kind of file synchronization primitive;
that's actually something new.  No, in the old days, if you really
cared about a file, you would fsync() it.  Period.  End of paragraph.

It was just that in those days, the main things people cared about
where either source/text files (so the editors of the day would do the
right thing) or e-mail (and no, just for the final delivery; for all
MTA's).

The problem that caused people to get this wrong idea was because (a)
back then Unix machines tended to be more reliable, because they were
run by professionals in machine rooms, very often with UPS's.  Also,
(b) people weren't loading craptastic video drivers with buggy
proprietary kernel modules; they may have used proprietary drivers,
but kernels weren't changing all the time, and there was a lot more
careful testing of drivers before they were unloosed onto the world.

Finally (c), ext3 had as an accident of how it provided protection
against old file blocks showing up newly allocated files (something
which BSD 4.3 did __not__ protect against, by the way), had the
behaviour that renaming over a file __usually__ (but not always)
provided atomic guarantees.

(c) was especially unfortunate, because it never applied to all Linux
file systems, just to ext3, and because the fact that it did this was
also responsible for disastrous desktop performance when you had the
combination of large number of streaming writes (i.e., bittorrent,
video ripping/copying, etc.) going on in the background combined with
foreground GUI applications that were fsync-happy() --- i.e., firefox.

Lots of users have complained about the desktop performance problem,
but the reality is we can't really solve that without also taking away
the magic that made (c) happen.  Whether you solve it by using
data=writeback and stick with ext3, or switch to ext4, or switch to
XFS, or switch to btrfs --- all of these will solve the desktop
performance problem, but they also leave you vulnerable to file loss
in the case of system crashes and applications that don't use
fsync()/fdatasync().

Hence the fact that all file system developers, whether they were
btrfs developers or XFS developers or ext4 developers, made the joke
at the file system developers summit two years ago, that what the
application programmers really wanted was O_PONY, with the magic pixie
dust.   Unfortunately:

http://www.linuxformat.com/files/nopony.jpg

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20101129151812.gu2...@thunk.org

Re: Safe File Update (atomic)

[Ted Ts'o]

On Fri, Dec 31, 2010 at 09:51:50AM -0200, Henrique de Moraes Holschuh wrote:
> On Fri, 31 Dec 2010, Olaf van der Spek wrote:
> > Ah, hehe. BTW, care to respond to the mail I send to you?
> 
> There is nothing more I can add to this thread.  You want O_ATOMIC.  It
> cannot be implemented for all use cases of the POSIX API, so it will not
> be implemented by the kernel.  That's all there is to it, AFAIK.
> 
> You could ask for a new (non-POSIX?) API that does not ask of a
> POSIX-like filesystem something it cannot provide (i.e. don't ask for
> something that requires inode->path reverse mappings).  You could ask
> for syscalls to copy inodes, etc.  You could ask for whatever is needed
> to do a (open+write+close) that is atomic if the target already exists.
> Maybe one of those has a better chance than O_ATOMIC.

The O_ATOMIC open flag is highly problematic, and it's not fully
specified.  What if the system is under a huge amount of memory
pressure, and the badly behaved application program does:

fd = open("file", O_ATOMIC | O_TRUNC);
write(fd, buf, 2*1024*1024*1024); // write 2 gigs, heh, heh heh

write(fd, buf2, 1024);
close(fd);

What happens if another program opens "file" for reading during the
one day sleep period?  Does it get the the old contents of "file"?
The partially written, incomplete new version of "file"?  What happens
if the file is currently mmap'ed, as Henrique has asked?

What if another program opens the file O_ATOMIC during the one day
sleep period, so the file is in the middle of getting updated by two
different processes using O_ATOMIC?

How exactly do the semantics for O_ATOMIC work?

And given at the momment ***zero*** file systems implement O_ATOMIC,
what should an application do as a fallback?  And given that it is
highly unlikely this could ever be implemented for various file
systems including NFS, I'll observe this won't really reduce
application complexity, since you'll always need to have a fallback
for file systems and kernels that don't support O_ATOMIC.

And what are the use cases where this really makes sense?  Will people
really code to this interface, knowing that it only works on Linux
(there are other operating systems, out there, like FreeBSD and
Solaris and AIX, you know, and some application programmers _do_ care
about portability), and the only benefits are (a) a marginal
performance boost for insane people who like to write vast number of
2-4 byte files without any need for atomic updates across a large
number of these small files, and (b) the ability to keep the the file
owner unchanged when someone other than the owner updates said file
(how important is this _really_; what is the use case where this
really matters?).

And of course, Olaf isn't actually offerring to implement this
hypothetical O_ATOMIC.  Oh, no!  He's just petulently demanding it,
even though he can't give us any concrete use cases where this would
actually be a huge win over a userspace "safe-write" library that
properly uses fsync() and rename().

If someone were to pay me a huge amount of money, and told me what was
the file size range where such a thing would be used, and what sort of
application would need it, and what kind of update frequency it should
be optimized for, and other semantic details about parallel O_ATOMIC
updates, what happens to users who are in the middle of reading the
file, what are the implications for quota, etc., it's certainly
something I can entertain.  But at the moment, it's a vague
specification (not even a solution) looking for a problem.

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110102070922.ga6...@thunk.org

Re: Safe File Update (atomic)

[Ted Ts'o]

On Sun, Jan 02, 2011 at 04:14:15PM +0100, Olaf van der Spek wrote:
> 
> Last time you ignored my response, but let's try again.
> The implementation would be comparable to using a temp file, so
> there's no need to keep 2 g in memory.
> Write the 2 g to disk, wait one day, append the 1 k, fsync, update inode.

Write the 2g to disk *where*?  Some random assigned blocks?  And using
*what* to keep track of the where to find all of the metadata blocks?
That information is normally stored in the inode, but you don't want
to touch it.  So we need to store it someplace, and you haven't
specified where.  Some alternate universe?  Another inode, which is
only tied to that file descriptor?  That's *possible*, but it's (a)
not at all trivial, and (b) won't work for all file systems.  It
definitely won't work for FAT based file systems, so your blithe, "oh,
just emulate it in the kernel" is rather laughable.

If you think it's so easy, *you* go implement it.

> > How exactly do the semantics for O_ATOMIC work?
> >
> > And given at the momment ***zero*** file systems implement O_ATOMIC,
> > what should an application do as a fallback?  And given that it is
> 
> Fallback could be implement in the kernel or in userland. Using rename
> as a fallback sounds reasonable. Implementations could switch to
> O_ATOMIC when available.

Using rename as a fallback means exposing random temp file names into
the directory.  Which could conflict with files that the userspace
might want to create.  It could be done, but again, it's an awful lot
of complexity to shove into the kernel.

> > highly unlikely this could ever be implemented for various file
> > systems including NFS, I'll observe this won't really reduce
> > application complexity, since you'll always need to have a fallback
> > for file systems and kernels that don't support O_ATOMIC.
> 
> I don't see a reason why this couldn't be implemented by NFS.

Try it; it should become obvious fairly quickly.  Or just go read the
NFS protocol specifications.

> As you've said yourself, a lot of apps don't get this right. Why not?
> Because the safe way is much more complex than the unsafe way. APIs
> should be easy to use right and hard to misuse. With O_ATOMIC, I feel
> this is the case. Without, it's the opposite and the consequences are
> obvious. There shouldn't be a tradeoff between safety and potential
> problems.

Application programmers have in the past been unwilling to change
their applications.  If they are willing to change their applications,
they can just as easily use a userspace library, or use fsync() and
rename() properly.  If they aren't willing to change their programs
and recompile (and the last time we've been around this block, they
weren't; they just blamed the file system), asking them to use
O_ATOMIC probably won't work, given the portiability issues.

> > And of course, Olaf isn't actually offerring to implement this
> > hypothetical O_ATOMIC.  Oh, no!  He's just petulently demanding it,
> > even though he can't give us any concrete use cases where this would
> > actually be a huge win over a userspace "safe-write" library that
> > properly uses fsync() and rename().
> 
> Not true. I've asked (you) for just such a lib, but I'm still waiting
> for an answer.

Pay someone enough money, and they'll write you the library.  Whining
about it petulently and expecting someone else to write it is probably
not going to work.

Quite frankly, if you're competent enough to use it, you should be
able to write such a library yourself.  If you aren't going to be
using it yourself, they why are you wasting everyone's time on this?

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110103032549.gc11...@thunk.org

Re: Safe File Update (atomic)

[Ted Ts'o]

On Sun, Jan 02, 2011 at 03:14:41PM -0200, Henrique de Moraes Holschuh wrote:
> 
> 1. Create unlinked file fd (benefits from kernel support, but doesn't
> require it).  If a filesystem cannot support this or the boundary conditions
> are unaceptable, fail.  Needs to know the destination name to do the unliked
> create on the right fs and directory (otherwise attempts to link the file
> later would have to fail if the fs is different).

This is possible.  It would be specific only to file systems that
support inodes (i.e., ix-nay for NFS, FAT, etc.).  Some file systems
would want to know a likely directory where the file would be linked
so for their inode and block allocation policies can optimize the
inode and block placement.

> 2. fd works as any normal fd to an unlinked regular file.
> 
> 3. create a link() that can do unlink+link atomically.  Maybe this already
> exists, otherwise needs kernel support.
> 
> The behaviour of (3) should allow synchrous wait of a fsync() and a sync of
> the metadata of the parent dir.  It doesn't matter much if it does
> everything, or just calling fsync(), or creating a fclose() variant that
> does it.

OK, so this is where things get trickly.  The first is you are asking
for the ability to take a file descriptor and link it into some
directory.  The inode associated with the fd might or might not be
already linked to some other directory, and it might or might not be
owned by the user trying to do the link.  The latter could get
problematical if quota is enabled, since it does open up a new
potential security exposure.

A user might pass a file descriptor to another process in a different
security domain, and that process could create a link to some
directory which the original user doesn't have access to.  The user
would no longer be able to delete file and drop quota, and the process
would retain permanent access to the file, which it might not
otherwise have if the inode was protected by a parent directory's
permissions.  It's for the same reason that we can't just implement
open-by-inode-number; even if you use the inode's permissions and
ACL's to do a access check, this allows someone to bypass security
controls based on the containing directory's permissions.  It might
not be a security exposure, but for some scenarios (i.e., a mode 600
~/Private directory that contains world-readable files), it changes
accessibility of some files.

We could control for this by only allowing the link to happen if the
user executing this new system call owns the inode being linked, so
this particular problem is addressable.

The larger problem is this doesn't solve give you any performance
benefits over simply creating a temporary file, fsync'ing it, and then
doing the rename.  And it doesn't solve the problem that userspace is
responsible for copying over the extended attributes and ACL
information.  So in exchange for doing something non-portable which is
Linux specific, and won't work on FAT, NFS, and other non-inode based
file systems at all, and which requires special file-system
modifications for inode-based file systems --- the only real benefit
you get is that the temp file gets cleaned up automatically if you
crash before the link/unlink new magical system call is completed.

Is it worth it?   I'm not at all convinced.

Can this be fixed?  Well, I suppose we could have this magical
link/unlink system call also magically copy over the xattr and acl's.

And if you don't care about when things happen, you could have the
kernel fork off a kernel thread, which does the fsync, followed by the
magic ACL and xattr copying, and once all of this completes, it could
do the magic link/unlink.

So we could bundle all of this into a system call.  *Theoretically*.
But then someone else will say that they want to know when this magic
link/unlink system call actually completes.  Others might say that
they don't care about the fsync happening right away, but would rather
wait some arbitary time, and let the system writeback algorithsm write
back the file *whenever*, but only when the file is written back
*whenever*, should the rest of the magical link/unlink happen.

So now we have an explosion of complexity, with all sorts of different
variants.  And there's also the problem where if you don't do don't
make the system call synchronous (where it does an fsync() and waits
for it to complete), you'll lose the ability to report errors back to
userspace.

Which gets me back to the question of use cases.  When are we going to
be using this monster?  For many use cases, where the original reason
why we said people were doing it wrong because they weren't doing
things right, the risk was losing data.  But if you don't do things
synchronously, and use fsync(), you'll also end up risking losing data
because you won't know about write failures --- specifically, your
program may have long exited by the time the write failure is noticed
by the kernel.  But if you use make the system call synchronous, no

Re: Safe File Update (atomic)

[Ted Ts'o]

On Mon, Jan 03, 2011 at 09:49:40AM -0200, Henrique de Moraes Holschuh wrote:
> 
> > 1) You care about data loss in the case of power failure, but not in
> > the case of hard drive or storage failure, *AND* you are writing tons
> > and tons of tiny 3-4 byte files and so you are worried about
> > performance because you're doing something insane with large number of
> > small files.
> 
> That usage pattern cannot be made both safe and fast outside of a full-blown
> ACID database, so lets skip it.

Agreed.

> 
> > 2) You are specifically worried about the case where you are replacing
> > the contents of a file that is owned by different uid than the user
> > doing the data file update, in a safe way where you don't want a
> > partially written file to replace the old, complete file, *AND* you
> > care about the file's ownership after the data update.
> 
> I am not sure about the file ownership, but this is the useful usecase IMO.

But if you don't care about file ownership, then you can do the write
a temp file, fsync, and rename trick.  If it's about "ease of use", as
you suggest, a userspace library solves that problem.  It's *only* if
you care about the file ownership remaining the same that (2) comes
into play.

> > 3) You care about the temp file used by the userspace library, or
> > application which is doing the write temp file, fsync(), rename()
> > scheme, being automatically deleted in case of a system crash or a
> > process getting sent an uncatchable signal and getting terminated.
> 
> This is always useful, as well.

 and (3) is the recovery after a power failure/crash scenario

If you don't care about the file ownership issue, then recovering
after a powerfailure/crash is the last remaining case --- and you
could solve this by creating a file with an mktemp-style name in a
mode 1777 directory, where the contents of the file contains the temp
file name to be deleted by an init.d script.  This could be done in
the userspace library, and if you crash after the rename, but before
you have a chance to delete the file containing the
temp-filename-to-be-deleted, that's not a problem, since the init.d
file will find no file with that name to be deleted, and then
continue.

Hence, all of these problems can be solved in userspace, with a
userspace library, with the exception of the file ownership issue,
which you've admitted may not be all that critical.

> > Is it worth it?  I'd say no; and suggest that someone who really cares
> > should create a userspace application helper library first, since
> > you'll need it as a fallback for the cases listed above where this
> > scheme won't work.  (Even if you do the fallback in the kernel, you'll
> > still need userspace fallback for non-Linux systems, and for when the
> > application is run on an older Linux kernel that doesn't have all of
> > this O_ATOMIC or link/unlink magic).
> 
> That's what I suggested, as well.

Then we're in agreement.  :-)

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/2011010319.gg11...@thunk.org

Re: Safe File Update (atomic)

[Ted Ts'o]

On Mon, Jan 03, 2011 at 12:26:29PM +0100, Olaf van der Spek wrote:
> 
> Given that the issue has come up before so often, I expected there to
> be a FAQ about it.

Your asking the question over (and over... and over...)  doesn't make
it an FAQ.  :-)

Aside from your asking over and over, it hasn't come up that often,
actually.  The right answer has been known for decades, and it's is
very simple; write a temp file, copy over the xattr's and ACL's if you
care (in many cases, such as an application's private state files, it
won't care, so it can skip this step --- it's only the more generic
file editors that would need to worry about such things --- but when's
the last time anyone has really worried about xattr's on a .c file?),
fsync(), and rename().

This is *not* hard.   People who get it wrong are just being lazy.

In the special case of dpkg, where they are writing a moderate number
of large files, and they care about syncing the files without causing
journal commits, the use of sync_file_range() on the files followed by
a series of fdatasync() calls has solved their issues as far as I know.

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110103144335.gd6...@thunk.org

Re: Safe File Update (atomic)

[Ted Ts'o]

On Wed, Jan 05, 2011 at 01:05:03AM +0100, Olaf van der Spek wrote:
> 
> Why is it that you ignore all my responses to technical questions you asked?
> 

In general, because they are either (a) not well-formed, or (b) you
are asking me to prove a negative.  Getting people to believe that you
can't square a circle[1] is very hard, and when I was one of the
postmasters at MIT, we'd get kooks every so often saying that they had
a proof that they could square the circle, but everyone was being
unfair and ignoring them, and could we please forward this to the head
of MIT's math department with their amazing discovery.  We learned a
long time ago that it's not worth trying to argue with kooks like
that.  It's like trying teaching a pig to sing.  It frustrates you,
and it annoys the pig.

[1] http://en.wikipedia.org/wiki/Squaring_the_circle

If you give me a specific approach, I can tell you why it won't work,
or why it won't be accepted by the kernel maintainers (for example,
because it involves pouring far too much complexity into the kernel).
But for me to list all possible approaches and tell you why each one
is not going to work?  You'll have to pay me before I'm willing to
invest that kind of time.

Best regards,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110105002537.gi2...@thunk.org

Re: Safe File Update (atomic)

[Ted Ts'o]

On Wed, Jan 05, 2011 at 12:55:22PM +0100, Olaf van der Spek wrote:
> > If you give me a specific approach, I can tell you why it won't work,
> > or why it won't be accepted by the kernel maintainers (for example,
> > because it involves pouring far too much complexity into the kernel).
> 
> Let's consider the temp file workaround, since a lot of existing apps
> use it. A request is to commit the source data before committing the
> rename. Seems quite simple.

Currently ext4 is initiating writeback on the source file at the time
of the rename.  Given performance measurements others (maybe it was
you, I can't remember, and I don't feel like going through the
literally hundreds of messages on this and related threads) have
cited, it seems that btrfs is doing something similar.  The problem
with doing a full commit, which means surviving a power failure, is
that you have to request a barrier operation to make sure the data
goes all the way down to the disk platter --- and this is expensive
(on the order of at least 20-30ms, more if you've written a lot to the
disk).

We have had experience with forcing data writeback (what you call
"commit the source data") before the rename --- ext3 did that.  And it
had some very nasty performance problems which showed up very busy
systems where people were doing a lot of different things at the same
time: large background writes from bittorrents and/or DVD ripping,
compiles, web browsing, etc.  If you force a large amount of data out
when you do a commit, everything else that tries to write to the file
system at that point stops, and if you have stupid programs (i.e.,
firefox trying to do database updates on its UI loop), it can cause
programs to apparently lock up, and users get really upset.

So one of the questions is how much should be penalizing programs that
are doing things right (i.e., using fsync), versus programs which are
doing things wrong (i.e., using rename and trusting to luck).  This is
a policy question, for which you might have a different opinion than I
might have on the subject.

We could also simply force a synchronous data writeback at rename
time, instead of merely starting writeback at the point of the rename.
In the case of a program which has already done an fsync(), the
synchronous data writeback would be a no-op, so that's good in terms
of not penalizing programs which do things right.  But the problem
there is that there could be some renames where forcing data writeback
is not needed, and so we would be forcing the performance hit of the
"commit the source data" even when it might not be needed (or wanted)
by the user.

How often does it happen that someone does a rename on top of an
already-existing file, where the fsync() isn't wanted.  Well, I can
think up scenarios, such as where an existing .iso image is corrupted
or needs to be updated, and so the user creates a new one and then
renames it on top of the old .iso image, but then gets surprised when
the rename ends up taking minutes to complete.  Is that a common
occurrence?  Probably not, but the case of the system crashing right
after the rename() is someone unusual as well.  

Humans in general suck at reasoning about low-probability events;
that's why we are allowing low-paid TSA workers to grope
air-travellers to avoid terrorist blowing up planes midflight, while
not being up in arms over the number of deaths every year due to
automobile accidents.

For this reason, I'm cautious about going overboard at forcing commits
on renames; doing this has real performance implications, and it is a
computer science truism that optimizing for the uncommon/failure case
is a bad thing to do.

OK, what about simply deferring the commit of the rename until the
file writeback has naturally completed?  The problem with that is
"entangled updates".  Suppose there is another file which is written
to the same directory block as the one affected by the rename, and
*that* file is fsync()'ed?  Keeping track of all of the data
dependencies is **hard**.   See: http://lwn.net/Articles/339337/

> > But for me to list all possible approaches and tell you why each one
> > is not going to work?  You'll have to pay me before I'm willing to
> > invest that kind of time.
> 
> That's not what I asked.

Actually, it is, although maybe you didn't realize it.  Look above,
and how I had to present multiple alternatives, and then shoot them
all down, one at a time.  There are hundreds of solutions, all of them
wrong.

Hence why *my* counter is --- submit patches.  The mere act of
actually trying to code an alternative will allow you to determine why
your approach won't work, or failing that, others can take your patch,
apply them, and then demonstrate use cases where your idea completely
falls apart.  But it means that you do most of the work, which is fair
since you're the one demanding the feature.

It doesn't scale for me to spend a huge amount of time composing
e-mails like this, which is why it's rare that I do that.  Y

Re: Safe File Update (atomic)

[Ted Ts'o]

On Wed, Jan 05, 2011 at 09:38:30PM +0100, Olaf van der Spek wrote:
> 
> Performance is important, I agree.
> But you're trading performance for safety here.

... but if the safety is not needed, then you're paying for no good
reason.  And if performance is needed, then use fsync().

> > OK, what about simply deferring the commit of the rename until the
> > file writeback has naturally completed?  The problem with that is
> > "entangled updates".  Suppose there is another file which is written
> > to the same directory block as the one affected by the rename, and
> > *that* file is fsync()'ed?  Keeping track of all of the data
> > dependencies is **hard**.   See: http://lwn.net/Articles/339337/
> 
> Ah. So performance isn't the problem, it's just hard too implement.
> Would've been a lot faster if you said that earlier.

"Too hard to implement" doesn't go far enough.  It's also a matter of
near impossibility to add new features later.  BSD FFS didn't get
ACL's, extended attributes, and many other features ***years*** after
Linux had them.  Complexity is evil; it leads to bugs, makes things
hard to maintain, and it makes it harder to add new features later.

But hey, if you're so smart, you go ahead and implement them yourself.
You can demonstrate how you can do it better than everyone else.
Otherwise you're just wasting everybody's time.  Complex ideas are not
valid ones; or at least they certainly aren't good ones.

   - Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110105213737.gp2...@thunk.org

Re: Safe File Update (atomic)

[Ted Ts'o]

On Wed, Jan 05, 2011 at 10:47:03PM +0100, Olaf van der Spek wrote:
> 
> That was about soft updates. I'm not sure this is just as complex.

Then I invite you to implement it, and start discovering all of the
corner cases for yourself.  :-)  As I predicted, you're not going to
believe me when I tell you it's too hard.

> I was thinking, doesn't ext have this kind of dependency tracking already?
> It has to write the inode after writing the data, otherwise the inode
> might point to garbage.

No, it doesn't.  We use journaling, and forced data writeouts, to
ensure consistency.

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110106005456.gq2...@thunk.org

Re: Safe File Update (atomic)

[Ted Ts'o]

On Thu, Jan 06, 2011 at 12:57:07AM +, Ian Jackson wrote:
> Ted Ts'o writes ("Re: Safe File Update (atomic)"):
> > Then I invite you to implement it, and start discovering all of the
> > corner cases for yourself.  :-)  As I predicted, you're not going to
> > believe me when I tell you it's too hard.
> 
> How about you reimplement all of Unix userland, first, so that it
> doesn't have what you apparently think is a bug!

I think you are forgetting the open source way, which is you scratch
your own itch.

The the main programs I use where I'd care about this (e.g., emacs)
got this right two decades ago; I even remember being around during
the MIT Project Athena days, almost 25 years ago, when we needed to
add error checking to the fsync() call because Transarc's AFS didn't
actually try to send the file you were saving to the file server until
the fsync() or the close() call, and so if you got an over-quota
error, it was reflected back at fsync() time, and not at the write()
system call which was what emacs had been expecting and checking.
(All of which is POSIX compliant, so the bug was clearly with emacs;
it was fixed, and we moved on.)

If there was a program that I used and where I'd care about it, I'd
scratch my own itch and fix it.  Olaf seems to really concerned about
this theoretical use case, and if he cares so much, he can either
stick with ext3, which has the property he wants purely by accident,
but which has terrible performance problem under some circumstances as
a result, or he can fix it in the programs that he cares about --- or
he can try to create his own file system (and he can either impress us
if he actually can solve it without disastrous performance problems,
or he can be depressed when no one uses it because it is dog slow).

Note that all of the modern file systems (and all of the historical
ones too, with the exception of ext3) have always had the same
property.  If you care about the data, you use fsync().  If you don't,
then you can take advantage of the fact that compiles are really,
really fast.  (After all, in the very unlikely case that you crash,
you can always rebuild, and why should you optimize for an unlikely
case?  And if you have crappy proprietary drivers that cause you to
crash all the time, then maybe you should rethink using said
proprietary drivers.)

That's the open source way --- you scratch your own itch.  I'm
perfectly satisifed with the open source tools that I use.  Unless you
think the programmers two decades ago were smarter, and people have
gotten dumber since then (Are we not men?  We are Devo!), it really
isn't that hard to follow the rules.

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110106040123.ga27...@thunk.org

Re: Safe file update library ready (sort of)

[Ted Ts'o]

On Wed, Jan 26, 2011 at 06:14:42PM +0100, Olaf van der Spek wrote:
> On Wed, Jan 26, 2011 at 5:36 PM, Hendrik Sattler
>  wrote:
> > BTW: KDE4 is a very good example for failure with modern filesystems. I
> > regularly loose configuration files when suspend-to-ram fails even if the
> > configuration of the running programs were not changed. Yay :-( And this is
> > with XFS, not Ext4! Filed a bug a looong time ago in KDE BTS. Reaction:
> > none!
> 
> Maybe complain to the Linux kernel people instead.

It won't be just XFS or ext4, but any file system except ext3 (which
has performance problems specifically *because* of an implementation
detail accidentally provided this feature you like), and I think what
you'll find is that most Linux kernel developers will tell you is that
it's a bug in the application.  

If you don't like that answer, you'll find that it's true for any
other OS (i.e., BSD, OpenSolaris, etc.)  --- so either KDE needs to
get with the program, or find its users gradually switching to other
windowing systems that have sanely written libraries.

- Ted

P.S.  There is a kernel options that provide improved ext3
performance, to wit, CONFIG_EXT3_DEFAULTS_TO_ORDERED=no, which will
also mean that you had better use fsync() if you want files pushed out
to disk.  So strictly speaking, it's not even true that KDE4 is
guaranteed to be safe if you use ext3.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110128022642.ga15...@thunk.org

Re: Safe file update library ready (sort of)

[Ted Ts'o]

On Fri, Jan 28, 2011 at 07:37:02AM +0100, Olaf van der Spek wrote:
> 
> Is there a way to log cases where (potentially) unsafe writes happen?
> Cases like truncation of an existing file, rename on a target that's
> not yet synced, etc.

Not really, because there are plenty of cases where it's perfectly OK
not to sync a file on close or on rename.  Any files created during a
build, for example, can easily be reproduced in the unlikely case of a
systme crash.  If you are untaring a source tree, it's fine not to
worry about syncing out the files, since presumably you can always
repeat the untar operation.  Or take git; when git is checking out
files into the working directory, there's no reason that has to be
done in a super-safe way.  On the other hand, when it is writing the
git object files and pack files, those had better be done safely.  At
the end of the day the application programmer needs to understand what
is going on, and write his code appropriately based on the needs of
his application with respect to reliability after a power crash.

So how can you just log warnings that the program has just done
something unsafe?  It's unsafe only if there's no other way to
reconstitute the data that was just written.  But that's not something
which is easily knowable.

(I know, I'm being *so* unfair; I'm expecting application programmers
to be competent...)

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110129125308.ga19...@thunk.org

Re: udeb and data.tar.xz files?

[Ted Ts'o]

On Mon, May 14, 2012 at 09:51:43PM +0200, Niels Thykier wrote:
> 
> Lintian is outdated (#664600) and the fix has been commited to the git
> repository[1]

I saw a bug report requesting that packages that failed the lintian
udeb-uses-non-gzip-data-tarball check should be summarily rejected.
Did this actually get implemeted on the ftp server?  i.e., do I need
to wait until the new version of lintian gets propgated out to the
debian servers, or if I'm in a hurry, hack dh_builddeb to generate a
udeb that doesn't use data.tar.xz?

I just want to know what my options are.

Thanks,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120514200131.ga17...@thunk.org

Re: udeb and data.tar.xz files?

[Ted Ts'o]

On Mon, May 14, 2012 at 10:20:08PM +0200, Philipp Kern wrote:
> 
> as soon as we get a hold of an ftp-master the autoreject will be dropped.
> We'll certainly don't wait until Lintian is backported. ;-)
> 

Great, thanks for the clarification.  I wasn't aware of

http://ftp-master.debian.org/static/lintian.tags

as how the auto-reject was configured, so I didn't know how to check
on my own or that it would be really easy to fix.

I'll wait until that gets fixed up and upload my updated packages
then.

Thanks again,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120514202909.gd1...@thunk.org

Re: Moving /tmp to tmpfs makes it useless

[Ted Ts'o]

On Fri, May 25, 2012 at 11:11:06AM +0200, Salvo Tomaselli wrote:
> > Files which are written on a regular filesystem stay on memory. This is
> > called the buffer cache. Whenever they are not used and/or the system
> > needs to reclaim memory, they are trashed.
> > Files which are written on a tmpfs stay on memory. Whenever they are not
> > used and/or the system needs to reclaim memory, they are swapped.
> > 
> > See? No difference.
> 
> You seem to forget that memory is not an unlimited resource, the
> system might need it for other things, and in that case a large
> tmpfs causes severe slowdown (and even complete freeze).

So what?  If you write to a normal file system, it goes into the page
cache, which is pretty much the same as writing into tmpfs.  In both
cases if you have swap configured, the data will get pushed to disk;
either to the file system or to swap, as memory requirements dictate.

The main advantage of tmpfs is that it gets wiped on reboot, and so it
prevents people and applications from thinking that they can keep
stuff in /tmp forever.  It's also faster because a file system has to
do extra work to make sure the files are preserved after a reboot.

 - Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120525125238.ga5...@thunk.org

Re: Moving /tmp to tmpfs makes it useful

[Ted Ts'o]

On Fri, May 25, 2012 at 02:49:14PM +0100, Will Daniels wrote:
> On 25/05/12 13:52, Ted Ts'o wrote:
> >So what?  If you write to a normal file system, it goes into the page
> >cache, which is pretty much the same as writing into tmpfs.  In both
> >cases if you have swap configured, the data will get pushed to disk;
> 
> That's not at all the same, the page cache is more temporary, it's
> getting flushed to disk pretty quick if memory is tight (presumably)
> but in the same situation using tmpfs going to swap is surely going
> to be more disruptive?

There will be some, but really, not that much difference between going
from tmpfs to swap compared to files written to a filesystem (in both
cases the data is stored in the page cache, whether it's a tmpfs file
or an ext2/3/4 or xfs or btrfs file) in many cases.

The major difference is that tmpfs pages only get written out to swap
when the system is under memory pressure.  In contrast, pages which
are backed by a filesystem will start being written to disk after 30
seconds _or_ if the system is under memory pressure.

So if you have a super-fast networking connection, it won't matter;
the download fill the memory very quickly, at which point it will get
written to swap and/or the file location on disk at the same rate,
since you'll be able to download faster than you can save to disk, so
the network connection will get throttled due to TCP/IP backpressure
to roughly the same rate as you are writing to the HDD.

If you have a slow networking connection, it's possible that the 30
second writeback timer will kick off before you start filling the
memory --- but in that case, it's not going to be that disrupting in
the tmpfs case, either.  You'll hit memory pressure, and at that point
you'll start writing to disk perhaps a bit later than the 30 second
writeback timer.  But at the same time, the download is coming in
slowly enough that you're probably not overwhelming the speed at which
you can write to the HDD or SSD.

The place where it will make a difference is if you have a very large
amount of memory, *and* you are downloading a really big file to /tmp
(substantially bigger than your physical memory), *and* your effective
end-to-end download speed is faster than your HDD write speed, but
slow enough that it takes substantially longer than 30 seconds to
download enough to fill your free physical memory.  But that's
actually a pretty narrow window.

> And anyway, not everybody uses swap, in which case this "default" is
> not entirely viable. I, for one, had no idea this had become default
> for Debian and I think it's likely to be one of those things that
> jumps out to bite people who weren't expecting it at some
> inconvenient moment.

Well, it's all about defaults, right?  It's easy enough to set up a
swap partition, or even just a swap file by default in the installer.
You can set up a swap file on the fly, so it's not that hard to deal
with it after the fact.

> I'm sure the project veterans and more attentive readers of this
> list are tired of recurring arguments like this, but usually if
> something is recurring it is for a reason. Given my general "no
> swap" preference, I'm glad this has come up again so that I'm aware
> of it this time.
> 
> The tmpfs setup seems far more appropriate as a performance tweak
> for admins than as a default. Where there is plenty of RAM, buffer
> cache makes the difference largely negligible. But where there isn't
> an abundance of RAM, it could needlessly cause problems (especially
> without swap).

If you're worried about installations which don't have much memory
(i.e., the 512mb netbook), then swap is absolutely mandatory, I would
think!

And if you consider how much memory most desktop/laptops have, and how
often people **really** are downloading multi-gigabyte files to /tmp
(note that browsers tend to default downloads to ~/Downloads), I
really think the people who are agitating against tmpfs are really
making a much more theoretical argument than one that seems likely to
be hit by an unsophisticated user --- and a sophistcated user can
easily decide whether to use /tmp on disk or not.

 - Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120526015655.ga5...@thunk.org

Re: /tmp on multi-FS set-ups, or: block users from using /tmp?

[Ted Ts'o]

On Sat, May 26, 2012 at 09:29:30PM +0700, Ivan Shmakov wrote:
>   … But that makes me recall a solution to both the /tmp and quota
>   issues I've seen somewhere: use ~/tmp/ instead of /tmp.  This
>   way, user's temporary files will be subject to exactly the same
>   limits as all the other his or her files.
> 
>   (Still, we may need to identify the software that ignores TMPDIR
>   and tries to write to /tmp unconditionally.)
> 
>  > (Snark aside, does tmpfs support quotas yet/will it ever?)

These days I'd argue that multi-user is such a corner case that it's
not worth optimizing for it as far as defaults are concerned.  If
you're trying to run a secure multi-user system, you need to be an
expert system administrator, keep up with all security patches, and
even then, good luck to you.  (The reality is that these days, no
matter what OS you're talking about, shell == root.  And that's
probably even true on the most unusably locked down SELinux system.)

What I'd do in that situation is to use per-user /tmp directories,
where each user would get its own mount namespace, and so each user
would have its own /tmp --- either a bind-mounted $(HOME)/tmp to /tmp
if you want to enforce quotas that way, or a separate tmpfs for each
user --- and then you can specify the size of the per-user tmpfs
mounted on each user's version of /tmp.

Cheers,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120526183215.ga11...@thunk.org

Re: multiarch, required packages, and multiarch-support

[Ted Ts'o]

On Thu, Jun 14, 2012 at 09:22:43PM -0700, Russ Allbery wrote:
> "Theodore Ts'o"  writes:
> 
> > If a required package (such as e2fslibs, which is required by e2fsprogs)
> > provides multiarch support, then Lintian requires that the package have
> > a dependency on the package "multiarch-support"[1].
> 
> > However, this causes debcheck to complain because you now have a
> > required package depending on a package, multiarch-support, which is
> > only at "standard" priority[2] 
> 
> multiarch-support should be priority: required.  It's already a dependency
> of several other priority: required packages, such as libselinux1 and
> zlib1g.
> 
> That implies that in the interim you should ignore debcheck.

Thanks, I've filed a bug against multiarch-support to this effect.


  - Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120615131712.gb30...@thunk.org

Re: EFI in Debian

[Ted Ts'o]

On Sun, Jul 08, 2012 at 10:00:05AM -0600, Paul Wise wrote:
> On Sun, Jul 8, 2012 at 7:15 AM, Wookey wrote:
> > Will Android machines make secure boot turn-offable or another key
> > installable, or will thay follow the Microsoft lead and lock
> > everything down too?
> 
> Are there any Android devices that aren't *already* bootloader locked
> or require jailbreaking to get root? I don't think Microsoft is
> creating a trend here, locked down ARM devices are already the norm
> AFAICT.

The Galaxy Nexus (and Nexus devices in general) can be unlocked by
simply running the "fastboot oem unlock" command which is distributed
as part of the Android SDK.  The unlock process will erase all of the
user data for security reasons (so that if someone steals your phone,
they can't use the unlock process to break security and grab all of
your data, including silly things like the authentication cookies
which would allow an attacker access to your google eaccount).

HTC and ASUS have also been selling their newer android with an
unlocked bootloader.  Most Samsung devices are shipped with unlocking
tools, so it came as a bit a surprise when the Verizon Samsung Galaxy
S3 came with a locked bootloader.  Some have blamed Verizon, but
there's no proof of that as far as I know.

So in answer to your question, there are plenty of Android devices
which are trivially unlockable.  (And once a Nexus phone is unlocked,
it's you can get a root shell trivially; no jail-breaking necessary.
Of course this is true for an attacker/thief who has managed to steal
your phone, but if you want to unlock the phone, it's easily doable on
many Android devices.)

   - Ted

P.S.  Personally, I recommend that people buy SIM unlocked, and easily
boot-unlocked Android phones; and if you get Google Experience Nexus
that isn't subsidized by Carriers, its firmware updates don't have to
get approved by carriers.  It also means you don't get any
carrier-mandated or handset-manfacturer-mandated bloatware.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120708233048.ga24...@thunk.org

Re: EFI in Debian

[Ted Ts'o]

On Fri, Jul 06, 2012 at 05:32:44AM +0100, Ben Hutchings wrote:
> 
> 2. Upstream kernel support: when booted in Secure Boot mode, Linux would
> only load signed kernel modules and disable the various debug interfaces
> that allow code injection.  I'm aware that David Howells, Matthew
> Garrett and others are working on this.

Matthew Garret believes that this is a requirement; however, there is
no documented paper trail indicating that this is actually necessary.
There are those who believe that Microsoft wouldn't dare revoke a
Linux key because of the antitrust issues that would arise.

This would especially true if the bootloader displayed a spash screen
with a huge penguin on it, and the user was obliged to hit a key
acknowledging the spash screen before the boot was allowed to
continue.  James is working on a signed bootloader which would do
this.

It's not even obvious that the spash screen is needed, BTW.  Canonical
is not using a splash screen and is not signing the kernel or kernel
modules.  It will be *very* interesting if Microsoft dares to revoke
Canonical's certificate, or refuse to issue a certificate.  I'm sure
there are developers in Europe who would be delighted to call this to
the attention of the European Anti-Trust regulators --- you know, the
ones who have already fined Microsoft to the tune of 860 million Euros
($1.1 billion USD).

So personally, I would hope that at least some distributions will
patch out the splash screen, and apply for a certificate.  If we have
multiple distributions using different signing policies and slightly
different approaches (which is the beauty of free/open source boot
loaders; everyone can tweak things slightly), we can see how Microsoft
will react.

It should be entertaining

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120708235244.gb24...@thunk.org

Re: EFI in Debian

[Ted Ts'o]

On Mon, Jul 09, 2012 at 04:48:38PM +0100, Matthew Garrett wrote:
> In article <20120708235244.gb24...@thunk.org>  Ted Ts'o wrote:
> > Matthew Garret believes that this is a requirement; however, there is
> > no documented paper trail indicating that this is actually necessary.
> > There are those who believe that Microsoft wouldn't dare revoke a
> > Linux key because of the antitrust issues that would arise.
> 
> Hey, it's hardly my fault that nobody else bothered turning up to the
> well-advertised events where this got discussed...

If it's documented on paper, it didn't happen.  :-)

Discussions in smoke-filled rooms, even if they are well-advertised,
don't really impress me  (This isn't your fault, but Microsoft's.)

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120709162649.ga7...@thunk.org

Re: Bug#652275: Guided partitioning should not offer separate /usr, /var, and /tmp partitions; leave that to manual partitioning

[Ted Ts'o]

On Fri, Dec 16, 2011 at 02:38:11PM +, Lars Wirzenius wrote:
> On Fri, Dec 16, 2011 at 02:13:29PM +0100, Stig Sandbeck Mathisen wrote:
> > Simon McVittie  writes:
> > 
> > > life's too short to spend time booting in single-user mode and
> > > resizing LVs.
> > 
> > That's probably why we now have online resizing of LVs and filesystems
> 
> resize2fs, at least, only supports online resizing to make the filesystem
> larger, not smaller. It's not particularly useful for, say, the root
> filesystem.

FYI, the resize2fs proram does support shrinking off-line shrinking of
ext3 file systems.  It doesn't currently support off-line resizing of
ext4 file systems (just on-line growth), but that's something I
consider a bug that I just haven't had the time to get around to fix.

Regards,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111227212937.gb10...@thunk.org

Re: Bug#652275: Guided partitioning should not offer separate /usr, /var, and /tmp partitions; leave that to manual partitioning

[Ted Ts'o]

On Wed, Dec 21, 2011 at 02:32:58PM +0100, Goswin von Brederlow wrote:
> > If we want to improve fsck time then the best thing to do would be
> > to consider a different default value for the -i option of mke2fs.

This advice is not applicable for ext4, since it will not read unused
portions of the inode table.  There have been a number of improvements
in the ext4 file system format which means that in general fsck times
for ext4 are around 7-12 times faster than the equivalent ext3 file
system.

> > As an aside "mke2fs -t ext4" includes huge_file, dir_nlink, and
> > extra_isize while mke4fs doesn't.  This difference seems wrong to
> > me.
> 
> Urgs. +1.

I've never heard of "mke4fs" --- who thought up that abortion?

"mke2fs -t ext4" and "mkfs.ext4" will both do the right thing, as far
as creating file systems that have the correct ext4 file system
features for a file system designed to be mounted using the ext4 file
system driver in modern Linux kernels.

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111227211736.ga10...@thunk.org

Re: e2fsprogs as Essential: yes?

[Ted Ts'o]

On Sat, Mar 26, 2011 at 10:42:09PM +, Mark Hymers wrote:
> 
> The only other thing I can see is that e2fsprogs contains lsattr and
> chattr - a quick grep through my local /var/lib/dpkg/info shows that
> chattr is used in the postfix postinst without an explicit dependency.
> I wonder if there are more instances of that?

What to do with lsattr and chattr is actually a somewhat interesting
question.  They are most definitely ext2/3/4 specific, and the
ext2/3/4 development team still adds new flags to it from time to
time, so we had no plans moving those two commands out of e2fsprogs
any time soon.  On the other hand, other file systems, including
reiserfs and btrfs, have used the same ioctl and command line
interface, and we do coordinate flags to prevent conflicts.  So other
users of other file systems will still need lsattr and chattr.

There are similar, although less serious, issues with filefrag -v
(which will work on other file systems), but which also has some
ext2/3/4 specific code it in.

Another binary which is used by other packages includes the logsave
utility, which is also in e2fsprogs, and which is used by
/etc/init.d/checkfs.sh and /etc/init.d/checkroot.sh in the initscripts
package.

Regards,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110326230729.gg2...@thunk.org

Re: Bug#616317: base: commit= ext3 mount option in fstab has no effect.

[Ted Ts'o]

reassign 616317 base
thanks

This isn't a bug in e2fsprogs; e2fsprogs has absolutely nothing to do
with mounting the file system.

Debian simply doesn't support the mount options for the root file
system in /etc/fstab having any effect on how the root file system is
mounted.  The root file system is mounted by the kernel, and the mount
options used by the kernel are specified by the rootflags= option on
the kernel's boot command line.

This is effectively a feature request, and I debated what was the best
way to deal with this bug.  I could close it, and say, "not a bug",
since Debian has never worked this way, and I suspect it was
deliberate.

Or, I could assign it to initramfs-tools, since what some other
distributions do is look in /etc/fstab, parse out the mount options in
for the root file system in /etc/fstab, and then insert into initrd
image the appropriate root mount options.  The problem with this is,
(a) it's a bit of a hack, (b) it only takes effect the next time you
install a new kernel, or if you deliberately and explicitly run
mkinitramfs, which has fairly baroque options that most users would
never figure out, and (c) not all Debian installations use an initrd,
so whether or not it works would depend on how the boot sequence was
set up.  If you don't use an initrd, you'd have to edit it into the
grub's configuration file.  But then, not all Debian systems use grub
as their boot loader.

Neither these seemed obviously the right choice.

So I'm going to do the cowardly thing, and choose the third option,
which is to reassign this back to base, cc'ing debian-devel.  I'm not
sure what the right thing is to do here, since honoring this feature
request would require making changes to multiple different packages:
initramfs-tools, all of the bootloaders, etc.

Should we try to make this work (at best badly) since a change in
mount options in /etc/fstab would only take effect at the next
mkinitramfs and/or update-grub invocation?  Or should we just close
out this bug and say, "tough luck, kid; if you want to change the root
file system's mount options, you need to edit your kernel's boot
options using whatever bootloader you might happen to be using"?

I have a slight preference for the latter, since it's a lot less
complexity that won't really work right anyway, but let's see what
other people think.

Regards,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110508024329.ga15...@thunk.org

Re: Could the multiarch wiki page be explicit about pkgconfig files?

[Ted Ts'o]

On Mon, Sep 19, 2011 at 08:51:00AM +0200, Tollef Fog Heen wrote:
> ]] "Theodore Ts'o" 
> 
> | and that's the correct location of pkgconfig files, which currently are
> | stored at /usr/lib/pkgconfig/.pc.   The Wiki page seems to imply
> | the correct location is /usr/lib//pkgconfig/.pc.   And
> | I've received patches that drop them there.
> 
> /usr/lib//pkgconfig/.pc is correct.
> 
> pkg-config 0.26-1 and newer looks there by default, so you probably want
> to either depend on that version or newer or add a Breaks against older
> versions.

OK, how about /usr/lib//debug/sbin/e2fsck?

I just checked and gdb doesn't find the debugging symbols if I drop the
debug files under /usr/lib/.  What is the planned "correct"
place for the debug files?

Thanks,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110919165201.ga29...@thunk.org

Re: Could the multiarch wiki page be explicit about pkgconfig files?

[Ted Ts'o]

On Mon, Sep 19, 2011 at 10:00:35PM +0200, Josselin Mouette wrote:
> Le lundi 19 septembre 2011 à 18:56 +0100, Simon McVittie a écrit : 
> > > The correct place for debug files is a hash-based path, instead of the
> > > crapfuck we have today.
> > 
> > ... but until then, for gdb to pick them up, debug symbols for $THING must 
> > be
> > in /usr/lib/debug/$THING (a general rule, independent of multiarch),
> 
> No, gdb is perfectly able to pick them from /usr/lib/debug/.build-id/.

Is this the right long-term path?  It seems wierd that we would be
using a "hidden" dot-file.  Is there a reason why it isn't
/usr/lib/debug/build-id?

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110925122251.ga17...@thunk.org