Log in

No account? Create an account

ZFS-only installation of FreeBSD 8.1 from DVD

Update: If you also want to encrypt everything except /boot, refer to this version.

Update: This works for FreeBSD 8.2 as well.

This is a step-by-step guide, mostly for my own reference, but you might find this useful too.

Disclaimer: This article is about clean installation of operating system, where “clean” means that, if you follow these instructions, all existing contents of the hard drive will be destroyed.  Do not follow these instructions except on an empty hard drive.  I shall not be held responsible for any loss of data.

  1. Boot using the FreeBSD 8.1 DVD.

  2. Drop into the fixit shell, using “CDROM/DVD” as the fixit media.

  3. Load necessary kernel modules:

    Fixit# chroot /mnt2 kldload nullfs zfs
  4. Wipe out any existing partition table on the disk (da0 in this example):

    Fixit# dd if=/dev/zero of=/dev/da0 bs=1m count=128 128+0 records in 128+0 records out 134217728 bytes transferred in 3.627965 secs (36995321 bytes/sec)
  5. Initialize the disk with a GUID partition table (GPT):

    Fixit# gpart create -s GPT da0 da0 created
  6. Install GPT boot code “pmbr” into the MBR:

    Fixit# gpart bootcode -b /mnt2/boot/pmbr da0 da0 has bootcode
  7. Create the boot partition that pmbr expects1, then install stage 2 boot code “gptzfsboot” into it:

    Fixit# gpart add -t freebsd-boot -l crimson-boot -s 128 da0 da0p1 added Fixit# gpart bootcode -p /mnt2/boot/gptzfsboot -i 1 da0
  8. Create a swap partition (1GB in this example):

    Fixit# gpart add -t freebsd-swap -l crimson-swap -s 1G da0 da0p2 added
  9. Create a ZFS partition, then create a ZFS pool with it:

    Fixit# gpart add -t freebsd-zfs -l crimson-001 da0 da0p3 added Fixit# zpool create -o altroot=/mnt -o cachefile=/tmp/zpool.cache crimson gpt/crimson-001 Fixit# df -h /mnt Filesystem Size Used Avail Capacity Mounted on crimson 6.8G 18K 6.8G 0% /mnt
  10. Make the ZFS pool available for booting2:

    Fixit# zpool set bootfs=crimson crimson
  11. Set the mountpoint of the root filesystem to /3:

    Fixit# zfs set mountpoint=/ crimson
  12. Now we will copy the FreeBSD installer (sysinstall) into the new filesystem and run it in a chroot-ed environment4sysinstall needs a couple of things, and we have to prepare them.  First, make /dev and /dist (FreeBSD installation DVD) available inside the filesystem:

    Fixit# mkdir /mnt/dev /mnt/mnt Fixit# mount -t devfs devfs /mnt/dev Fixit# mount_nullfs /dist /mnt/mnt
  13. Copy sysinstall and friends, residing in /stand:

    Fixit# find -xd /stand -print0 | cpio -pdum0 /mnt 5172 blocks
  14. sysinstall expects a number of tools in /usr/bin; copy them:

    Fixit# mkdir -p /mnt/usr/bin Fixit# (cd /stand && find gunzip cpio -print0 | cpio -pdum0 /mnt/usr/bin)
  15. We will instruct sysinstall to emit progress messages to a file named sysinstall.debug.  Start dumping its contents onto ttyv4 (accessible by pressing Alt-F5):

    Fixit# tail -Fc+0 /mnt/sysinstall.debug > /dev/ttyv4 & Fixit# stty -f /dev/ttyv4 $(stty -g)
  16. Now chroot into the filesystem and run sysinstall there:

    Fixit# env debug=YES chroot /mnt /stand/sysinstall
  17. In the main menu, go to the “Configure” menu.

  18. In the configuration menu, choose “Media.”

  19. In the media selection menu, choose “File System.”

  20. sysinstall asks you where the installation file is.  Enter “/mnt” (without quotes).

  21. Back in the configuration menu, choose “Distributions.”

  22. In the distribution selection menu, select all that you want to install, then select “Exit.”  You will at least need the base and kernels at the very minimum.

  23. sysinstall starts installation of selected distributions.  Wait until installation finishes, and you are brought back to the configuration menu.

  24. Configure the installed system further as necessary.  Common tasks include:

    • Setting root password;
    • Setting the system timezone;
    • Adding initial user/group;
    • Setting at least one network interface (where you will set the hostname too);
    • Enabling sshd.
  25. Exit to the main menu, then select “Exit Install” to end sysinstall.</p>
  26. Now we need to do additional, ZFS-specific configuration.  First, copy zpool.cache, which was created earlier when we ran zfs create) into /boot/zfs5:

    Fixit# cd /mnt/boot Fixit# mkdir -p zfs Fixit# cp -p /tmp/zpool.cache zfs/zpool.cache
  27. The previous sysinstall installed the kernel not in /boot/kernel but in /boot/GENERIC.  Instruct BTX loader to look for the kernel there instead:

    Fixit# echo 'kernel="GENERIC"' >> loader.conf
  28. Load the ZFS kernel module when booting:

    Fixit# echo 'zfs_load="YES"' >> loader.conf
  29. Load geom_eli too, so that we can use encrypted swap:

    Fixit# echo 'geom_eli_load="YES"' >> loader.conf
  30. Instruct that the ZFS root pool (“crimson”) is also the root filesystem:

    Fixit# echo 'vfs.root.mountfrom="zfs:crimson"' >> loader.conf
  31. Tell ZFS not to disable prefetch:

    Fixit# echo 'vfs.zfs.prefetch_disable="0"' >> loader.conf
  32. Enable ZFS in /etc/rc.conf:

    Fixit# cd /mnt/etc Fixit# echo 'zfs_enable="YES"' >> rc.conf
  33. Add the swap partition:

    Fixit# echo '/dev/gpt/crimson-swap.eli none swap sw 0 0' >> fstab
  34. Exit the fixit shell and go back to the FreeBSD installer:

    Fixit# exit
  35. Go to the main menu, then select “Exit Install” to reboot.

1 ^ pmbr locates a GUID partition of freebsd-boot type then loads and executes the next-stage boot code from it.

2 ^ gptzfsboot locates a ZFS pool with the bootfs (boot filesystem) property set, then loads and executes BTX loader (a.k.a. /boot/loader) from that filesystem.

3 ^ In fact we don't have to do this, at least as of FreeBSD 8.1.  As a little-known and undocumented side effect of specifying an altroot at the time of pool creation, the mountpoint property of root filesystem has already been set to /.  I included this step just in case the default behavior of “zpool create altroot=…” changes in a future version of ZFS.

4 ^ Note that an instance of sysinstall is already running on ttyv0, accessible by pressing Alt-F1; that's where we started the fixit shell in the first place. We are going to start another instance of sysinstall from the fixit shell on ttyv3 (Alt-F4).

5 ^ zpool.cache contains information about system pools (i.e. pools imported without the -R option).  Various stages of booting process need it in order to locate the root/boot pool.

Tags: ,


Eugene is still alive! When can we see a real post?

What are you talking about, this is a real post!  XD


Good job

Thanks for the post. I have a situation here, how can i running ZFS mirroring on 3 different hard drives. My end product here i virtualization.
Thank you

Multi-OS drives

dd if=/dev/zero of=/dev/da0 bs=1m count=128

I presume that if I've got at least one other OS (e.g., Windows) on the drive, I should replace /dev/da0 with the FreeBSD partition?

Re: Multi-OS drives

I'm not sure if you can sub-partition a slice, say da0s2, using GPT. The real question is whether gptzfsloader can still correctly operate in that case.

Think I'm gonna experiment it some time and get back to you. That's an interesting (and probably pretty common) situation...

Thank you for reporting!