Hi!
On Wed, 2014-09-17 at 11:11:30 +0200, Petter Reinholdtsen wrote:
> I did some testing installing using eatmydata to see how much it could
> reduce the installation time. I used the enclosed test script to
> compare the installation time for three test setup. One is the normal
> one, the other is using dpkg-divert to divert apt-get, aptitude and
> dpkg, while the third uses the Dir::Bin::dpkg setting to use a dpkg
> wrapper with eatmydata enabled.
You asked in the past why the current implementation is the way it
is. A quick summary would be that, dpkg has always done fsync() on
the database (not on its directories but still), and got support to
perform fsync() on the unpacked filesystem, due to new filesystems
being very unsafe. And then other people using those filesystems that
force to choose between safety or speed saw a substantial performance
loss, so they could decide what they preferred, and could go back to
the previous (broken) behavior.
Of course, because these same filesystems show a very poor performance
with applications that are doing proper safe file handling, some
people also started to use the --force-unsafe-io option when doing
throwaway installations or upgrades, like on first install, or on
buildds and similar.
In any case, the point of the option is that, even if you get your
unpacked files corruped with the 0-length issue, or similar on your
day-to-day system, you should always be able to restore it from a
recovery media, as you might only need to reinstall damaged packages,
and you know which ones those might be, because the database would be
in a sane state.
Part of this is explained in
<https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slow_when_using_new_filesystems_such_as_btrfs_or_ext4.3F>
> This was the result. The number is in seconds.
>
> Installing kde-standard
>
> Wed Sep 17 09:54:50 CEST 2014 used: 357 divert
> Wed Sep 17 10:00:51 CEST 2014 used: 359 dpkg_conf
> Wed Sep 17 10:09:38 CEST 2014 used: 525 default
>
> Installing kde-full with policy-rc.d in place.
> Wed Sep 17 10:29:33 CEST 2014 used: 424 divert
> Wed Sep 17 10:36:29 CEST 2014 used: 413 dpkg_conf
> Wed Sep 17 10:45:35 CEST 2014 used: 543 default
>
> As you can see, the reduction in installation time is in the range
> 21-32 percent of the current default. It is not obvious to me why the
> Dir::Bin::dpkg approach can be quicker than the divert approach. This
> might be caused by other issues, as the last run was done just after
> boot. Perhaps the order these tests are executed matter?
You should either clear the kernel cache or reboot on each iteration
to try to get a similar initial state. The former can be done with
something like:
sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
You might also want to try with the attached dpkg patch which should
disable all fsync() calls in the main dpkg program, to see how the
rest of the system affects your performance, besides dpkg itself.
Thanks,
Guillem
diff --git a/lib/dpkg/atomic-file.c b/lib/dpkg/atomic-file.c
index 07ab49c..cf83c4a 100644
--- a/lib/dpkg/atomic-file.c
+++ b/lib/dpkg/atomic-file.c
@@ -68,8 +68,10 @@ atomic_file_sync(struct atomic_file *file)
ohshite(_("unable to write new file '%.250s'"), file->name_new);
if (fflush(file->fp))
ohshite(_("unable to flush new file '%.250s'"), file->name_new);
+#if 0
if (fsync(fileno(file->fp)))
ohshite(_("unable to sync new file '%.250s'"), file->name_new);
+#endif
}
void
diff --git a/lib/dpkg/dbmodify.c b/lib/dpkg/dbmodify.c
index 9acba1f..3e44a5b 100644
--- a/lib/dpkg/dbmodify.c
+++ b/lib/dpkg/dbmodify.c
@@ -359,9 +359,11 @@ modstatdb_note_core(struct pkginfo *pkg)
if (ftruncate(fileno(importanttmp), uvb.used))
ohshite(_("unable to truncate for updated status of `%.250s'"),
pkg_name(pkg, pnaw_nonambig));
+#if 0
if (fsync(fileno(importanttmp)))
ohshite(_("unable to fsync updated status of `%.250s'"),
pkg_name(pkg, pnaw_nonambig));
+#endif
if (fclose(importanttmp))
ohshite(_("unable to close updated status of `%.250s'"),
pkg_name(pkg, pnaw_nonambig));
diff --git a/lib/dpkg/dir.c b/lib/dpkg/dir.c
index 34bdac6..7313b25 100644
--- a/lib/dpkg/dir.c
+++ b/lib/dpkg/dir.c
@@ -43,6 +43,7 @@
static void
dir_sync(DIR *dir, const char *path)
{
+#if 0
int fd;
fd = dirfd(dir);
@@ -52,6 +53,7 @@ dir_sync(DIR *dir, const char *path)
if (fsync(fd))
ohshite(_("unable to sync directory '%s'"), path);
+#endif
}
/**
diff --git a/src/main.c b/src/main.c
index 4ec4ecd..0795414 100644
--- a/src/main.c
+++ b/src/main.c
@@ -191,7 +191,7 @@ int fc_breaks=0, fc_badpath=0, fc_overwritediverted=0, fc_architecture=0;
int fc_nonroot=0, fc_overwritedir=0, fc_conff_new=0, fc_conff_miss=0;
int fc_conff_old=0, fc_conff_def=0;
int fc_conff_ask = 0;
-int fc_unsafe_io = 0;
+int fc_unsafe_io = 1;
int fc_badverify = 0;
int fc_badversion = 0;
