In this guide I will demonstrate how you can install a fully functional full root on ZFS FreeBSD9 using a GPT scheme with a non legacy root ZFS mountpoint optimized for 4K drives. We will also use ZFS for SWAP.
You can use this as a reference guide for a single or mirror installation.
(1) Boot from a FreeBSD9 installation DVD or memstick and choose “Live CD”.
(2) Create the necessary partitions on the disk(s) and add ZFS aware boot code.
a) For a single disk installation.
gpart create -s gpt ada0 gpart add -b 34 -s 94 -t freebsd-boot ada0 gpart add -t freebsd-zfs -l disk0 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
b) Repeat the procedure for the second drive if you want a mirror installation.
gpart create -s gpt ada1 gpart add -b 34 -s 94 -t freebsd-boot ada1 gpart add -t freebsd-zfs -l disk1 ada1 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
(3) Align the Disks for 4K and create the pool.
a) For a single disk installation.
gnop create -S 4096 /dev/gpt/disk0 zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.nop zpool export zroot gnop destroy /dev/gpt/disk0.nop zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
b) For a mirror installation.
gnop create -S 4096 /dev/gpt/disk0 gnop create -S 4096 /dev/gpt/disk1 zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot mirror /dev/gpt/disk0.nop /dev/gpt/disk1.nop zpool export zroot gnop destroy /dev/gpt/disk0.nop gnop destroy /dev/gpt/disk1.nop zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
(4) Set the bootfs property and checksums.
zpool set bootfs=zroot zroot zfs set checksum=fletcher4 zroot
(5) Create appropriate filesystems (feel free to improvise!).
zfs create zroot/usr zfs create zroot/usr/home zfs create zroot/var zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp zfs create -o compression=lzjb -o setuid=off zroot/usr/ports zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash zfs create -o exec=off -o setuid=off zroot/var/db zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg zfs create -o exec=off -o setuid=off zroot/var/empty zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail zfs create -o exec=off -o setuid=off zroot/var/run zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
(6) Add swap space and disable checksums. In this case I add 4GB of swap.
zfs create -V 4G zroot/swap zfs set org.freebsd:swap=on zroot/swap zfs set checksum=off zroot/swap
(7) Create a symlink to /home and fix some permissions.
chmod 1777 /mnt/tmp cd /mnt ; ln -s usr/home home chmod 1777 /mnt/var/tmp
(8) Instal FreeBSD.
sh cd /usr/freebsd-dist export DESTDIR=/mnt for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done
(9) Copy zpool.cache (very important!!!)
cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache
(10) Create the rc.conf, loader.conf and an empty fstab (otherwise the system will complain).
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf echo 'zfs_load="YES"' >> /mnt/boot/loader.conf echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf touch /mnt/etc/fstab
Reboot, adjust time zone info, add a password for root, add a user and enjoy!!!
Hello, very helpful guide, thank you.
However there is a problem with the presentation that caused some confusion, the problem is the lowercase l (L) and 1(one) looks exactly the same on the font you used (monospace), take a look at the following:
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
You can’t tell which one is 1 and which one is l in “-l disk1″ and “-i 1 ada1″ when it is inside which is using the monospace fontjust, I am sure for a pro like yourself it doesn’t matter, but it could cause some head scratching for newbies.
Thanks and keep up the good work
gkontos,
Thanks for this site. It’s a great help to newbies like me.
I have a question:
I followed this guide exactly – I didn’t even improvise in step 5.
After reboot all I see is a blank screen with blinking cursor.
FreeBSD does not boot up.
Any idea what could be the problem?
Thanks again!
Never mind. For some reason after this install, even though it was set up correctly beforehand, my bios did not recognize the zfs hard drives as boot devices. It didn’t even let me select then from the list. Strangely enough, after I rebooted again, they showed up and everything is fine.
Thanks again!
Sorry to bother you but I have another question…
I set up my system for mirroring according to this guide.
How do I check if it is actually working?
I tried typing “gmirror status” but it says ‘status’ command unavailable.
Am I typing the wrong command or do I still have to set up gmirror for my RAID1 setup to work?
Thanks!
Hi Reuven,
gmirror is for mirroring ufs filesystems. You have set up a ZFS mirror. To check the status of your pool just type:
zpool status
Awesome!
Everything looks good!
Thanks for the help!
Would setting the zpool with noatime screw anything up?
I haven’t tried it but I don’t think you will get any benefit either.
Good guide for 9; I had installed on FreeBSD 8.2 previously. One thing to note: The gnop needs to be only done on a single disk. It does not need to be done on all of them. I read through the zpool source code, and it chooses the block size that will fit on all drives, so if you run gnop on only a single drive, it’ll still choose 4K blocks. I also confirmed this when I setup my 6 disk raidz2 a few months ago.
there is one missing step before first Reboot – at least for me, i had to do: zfs set mountpoint=/ zroot, otherwise system gets error – mounting from zfs:zroot failed with error 2: unknown file system
But the rest is fine, thanks for a nice guide !
This is strange bad I am glad you made it worked.
How does it go if it is 4 drives with RAIDZ1 parity with a separate boot image on a CF/USB stick?
I am not sure I understand the question. Do you want to boot from a UFS CF/USB stick and then mount your ZFS datasets?
Yep, that would be ideal, isn’t it?
In that case you might want to store all the OS data in the CF and separate the rest in a RAIDZ1 pool. That works pretty well on some storage servers.
thanks for this guide. Just one question, if you want.
I have 6GB of RAM (and i usually use 3 or 4 GB max), can i bypass STEP 6 of your guide ? (install without swap).
Thanks in advance.
If your system ever needs to swap you will get an “out of memory” warning. This might eventually lead to a kernel panic.
Thanks.
Then, i imagine i have to tune correctly those options (to be sure that i do not go to an “out of memory”)
vm.kmem_size=xxx
vm.kmem_size_max=xxx
vfs.zfs.arc_max=xxx
vfs.zfs.vdev.cache.size=xxx
(remember with my actual config and ufs, i allways have near 2GB of free memory)
No, don’t touch anything. Start by letting the OS deal with all the values. If you really need tuning just change this value: vfs.zfs.arc_max=(1/2 RAM)
If you have 2GB of free memory then either remove it from your system or find something useful for it. Free memory is wasted memory!