Hello, Jesus,
I have transitioned a number of systems roughly by the
same procedure as you've outlined. Sadly, my notes are
not in English so they wouldn't be of much help directly;
but I can report that I had success with similar "in-place"
manual transitions from mirrored SVM (pre-solaris 10u4)
to new ZFS root pools, as well as various transitions
of ZFS root pools from one layout to another, on systems
with limited numbers of disk drives (2-4 overall).
As I've recently reported on the list, I've also done
such "migration" for my faulty single-disk rpool at home
via the data pool and backwards, changing the "copies"
setting enroute.
Overall, your plan seems okay and has more failsafes
than we've had - because longer downtimes were affordable ;)
However, when doing such low-level stuff, you should make
sure that you have remote access to your systems (ILOM,
KVM, etc.; remotely-controlled PDUs for externally enforced
poweroff-poweron are welcome), and that you can boot the
systems over ILOM/rKVM with an image of a LiveUSB/LiveCD/etc
in case of bigger trouble.
In the steps 6-7, where you reboot the system to test
that new rpool works, you might want to keep the zones
down, i.e. by disabling the zones service in the old BE
just before reboot, and zfs-sending this update to the
new small rpool. Also it is likely that in the new BE
(small rpool) your old "data" from the big rpool won't
get imported by itself and zones (or their services)
wouldn't start correctly anyway before steps 7-8.
---
Below I'll outline our experience from my notes, as it
successfully applied to an even more complicated situation
than yours:
On many Sol10/SXCE systems with ZFS roots we've also
created a hierarchical layout (separate /var, /usr, /opt
with compression enabled), but this procedure HAS FAILED
for newer OpenIndiana systems. So for OI we have to use
the default single-root layout and only seperate some of
/var/* subdirs (adm, log, mail, crash, cores, ...) in
order to set quotas and higher compression on them.
Such datasets are also kept separate from OS upgrades
and are used in all boot environments without cloning.
To simplify things, most of the transitions were done
in off-hours time so it was okay to shut down all the
zones and other services. In some cases for Sol10/SXCE
the procedure involved booting in the "Failsafe Boot"
mode; for all systems this can be done with the BootCD.
For usual Solaris 10 and OpenSolaris SXCE maintenance
we did use LiveUpgrade, but at that time its ZFS support
was immature, so we circumvented LU and transitioned
manually. In those cases we used LU to update systems
to the base level supporting ZFS roots (Sol10u4+) while
running from SVM mirrors (one mirror for main root,
another mirror for LU root for new/old OS image).
After the transition to ZFS rpool, we cleared out the
LU settings (/etc/lu/, /etc/lutab) by using defaults
from the most recent SUNWlu* packages, and when booted
from ZFS - we created the "current" LU BE based on the
current ZFS rpool.
When the OS was capable of booting from ZFS (sol10u4+,
snv_100 approx), we broke the SVM mirrors, repartitioned
the second disk to our liking (about 4-10Gb for rpool,
rest for data), created the new rpool and dataset
hierarchy we needed and had in mounted under "/zfsroot".
Note that in our case we used a "minimized" install
of Solaris which fit under 1-2Gb per BE, we did not use
a separate /dump device and the swap volume was located
in the ZFS data pool (mirror or raidz for 4-disk systems).
Zoneroots were also separate from the system rpool and
were stored in the data pool. This DID yield problems
for LiveUpgrade, so zones were detached before LU and
reattached-with-upgrade after the OS upgrade and disk
migrations.
Then we copied the root FS data like this:
# cd /zfsroot && ( ufsdump 0f - / | ufsrestore -rf - )
If the source (SVM) paths like /var, /usr or /boot are
separate UFS filesystems - repeat likewise, changing the
current paths in the command above.
For non-UFS systems, such as migration from VxFS or
even ZFS (if you need a different layout, compression,
etc. - so ZFS send/recv is not applicable), you can use
Sun cpio (it should carry over extended attributes and
ACLs). For example, if you're booted from the LiveCD
and the old UFS root is mounted in "/usfroot" and new
ZFS rpool hierarchy is in "/zfsroot", you'd do this:
# cd /ufsroot && ( find . -xdev -depth -print | cpio -pvdm /zfsroot )
The example above also copies only the data from
current FS, so you need to repeat it for each UFS
sub-fs like /var, etc.
Another problem we've encountered while cpio'ing live
systems (when not running from failsafe/livecd) is that
"find" skips mountpoints of sub-fses. While your new ZFS
hierarchy would provide usr, var, opt under /zfspool,
you might need to manually create some others - see the
list in your current "df" output. Example:
# cd /zfsroot
# mkdir -p tmp proc devices var/run system/contract system/object
etc/svc/volatile
# touch etc/mnttab etc/dfs/sharetab
Also some system libraries might be overlay-mounted on
top of "filenames", for example a hardware-optimised
version of libc, or the base "/dev" (used during boot)
or its components might be overlayed by dynamic devfs:
# df -k | grep '.so'
/usr/lib/libc/libc_hwcap2.so.1 2728172 522776 2205396 20%
/lib/libc.so.1
# mount | egrep ^/dev
/devices on /devices read/write/setuid/devices/dev=5340000 on Thu Dec 1
06:14:26 2011
/dev on /dev read/write/setuid/devices/dev=5380000 on Thu Dec 1
06:14:26 2011
/dev/fd on fd read/write/setuid/devices/dev=5640001 on Thu Dec 1
06:14:38 2011
In order to copy the original libs and dev-paths from
a live system, you would need FS-aware tools like
ufsdump or zfs-send, or you might try loopback-mounts:
# mount -F lofs -o nosub / /mnt
# (cd /mnt; tar cvf - devices dev lib/libc.so.1 ) | (cd /zfsroot; tar xvf -)
# umount /mnt
There might be some more pitfalls regarding use of the
/etc/vfstab along with canmount=noauto, mountpoint=legacy
and such - but these are mostly relevant for our split
ZFS-root hierarchy on some OS releases.
Also, if you use LiveUpgrade, you might want to apply
Jens Elkner's patches and/or consider his troubleshooting
tips: http://iws.cs.uni-magdeburg.de/~elkner/luc/lutrouble.html
One of these pathces in particular allowed LU to ignore
some paths which you know are irrelevant to OS upgrade,
i.e. "/export/home/*" - this can speed up LU by many
minutes per run.
Note that these patches were applicable back when I updated
our systems (Sol10u4-Soll10u8, SXCE~100), so these tricks
may or may not be included in current LU software.
Hope this helps, don't hesitate to ask for more details
(my notes became a "textbook" for our admins which
approached 100 pages, so not everything is in this email)
//Jim Klimov
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
