The new bsdInstaller is here but unfortunately as we saw it still doesn’t support a full ZFS installation. Also, due to the new layout of the installer the traditional way of installing from <FIXIT> doesn’t work anymore. The installation files are now located under /usr/freebsd-dist and the new “Live CD” option doesn’t create any kind of live fs. Therefore we have to improvise.
For the purpose of this demonstration am using a 9-Current that I created on April 29th. Boot with your 9-Current and choose “Lice CD” when prompted.
UPDATE: You can see a newer version here !
The first thing we need to do is to make /tmp writable so that we can store the zpool.cache file.
umount /dev/md1 mdmfs -s 512M md1 /tmp
That should be enough. Now, lets partition our drive. I will be using a GPT scheme and create a native swap partition of 4 GB Ram, the rest will be allocated to ZFS. This guide installs FreeBSD on a single disk but this could be easily reproduced for a mirror or a raidz1 pool.
gpart create -s gpt ada0 gpart add -b 34 -s 64k -t freebsd-boot ada0 gpart add -s 4G -t freebsd-swap -l swap0 ada0 gpart add -t freebsd-zfs -l disk0 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
My disk is now ready so I will create a pool and call it zroot and then mount it under /mnt. During this procedure some error messages will appear “can not mount, failed to create mount point”. The reason for those is that /mnt is read only but you can safely ignore them since we will export and import back our pool.
zpool create zroot /dev/gpt/disk0 zpool set bootfs=zroot zroot zfs set checksum=fletcher4 zroot
Now lets create the file system layout. You can safely ignore the error messages about mounting again since the pool hasn’t been exported/imported yet.
zfs set mountpoint=/mnt zroot zfs create zroot/usr zfs create zroot/usr/home zfs create zroot/var zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp zfs create -o compression=lzjb -o setuid=off zroot/usr/ports zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash zfs create -o exec=off -o setuid=off zroot/var/db zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg zfs create -o exec=off -o setuid=off zroot/var/empty zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail zfs create -o exec=off -o setuid=off zroot/var/run zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
Now at this point I will export the pool, import back while preserving the zpool.cache in /tmp
zpool export zroot zpool import -o cachefile=/tmp/zpool.cache zroot
The pool is now mounted under /mnt and we can now proceed with the final steps.
chmod 1777 /mnt/tmp cd /mnt ; ln -s usr/home home chmod 1777 /mnt/var/tmp
Next we need to install FreeBSD. Like I said things have changed therefore the procedure is different. This guide assumes that you also want to install source and ports. For a minimal installation only base.txz lib32.txz kernel.txz are necessary.
sh
cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done
Now let’s save the zpool.cache
cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache
Done! Now, all that is left is to create the rc.conf, loader.conf and fstab.
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf echo 'zfs_load="YES"' >> /mnt/boot/loader.conf echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf cat << EOF > /mnt/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/gpt/swap0 none swap sw 0 0 EOF
Final steps
zfs set readonly=on zroot/var/empty zfs umount -a
If you get a device busy message try umount -f /mnt
zfs set mountpoint=legacy zroot zfs set mountpoint=/tmp zroot/tmp zfs set mountpoint=/usr zroot/usr zfs set mountpoint=/var zroot/var
Reboot, adjust time zone info, add a password for root, add a user and enjoy!!!
For an automated installation see Olaf Klein comment !
Hello,
thanks for the great tutorial. i made a script out of it, with following benefits:
- name your pool
- create raidz, mirror or even raid10 szenario
HOWTO:
Startup the FreeBSD livefs (i used memstick). Go into the Fixit console. and prepare:
# tcsh
# set autolist
# umount /dev/md1
# mdmfs -s 512M md1 /tmp
# ifconfig
# dhclient nfe0 (or whatever your NIC is)
# mkdir -p /tmp/bsdinstall_etc
# echo nameserver 10.0.0.1 >/etc/resolv.conf
# fetch http://www.monkeytower.net/go9.sh
# chmod +x go9.sh
Execute the script with the following parameter:
-p sets the geom provider to use, you can use multiple. Add a name for the GPT labels: -p ad4=black -p ad6=white
-s sets the swap_partition_size to create, you can use m/M for megabyte or g/G for gigabyte
-S sets the zfs_partition_size to create, you can use m/M for megabyte or g/G for gigabyte, default is all available size
-n sets the name of the zpool to create
-m sets the zpool raid-mode, stripe (only single disk), mirror (at least two disks) and raidz (at least three disks) or raid10 with at least 4 disks
-d sets local directory to get distribution packages from
You can use more than one device, creating a mirror. To specify more than one device, use multiple -p options.
EXAMPLE:
# ./go9.sh -p ada1=red -p ada2=blue -pada3=yellow -s 4G -n tank
in case something goes wrong and you want to start over:
# zpool destroy tank
might be a good idea.
DISCLAIMER: Use at your own risk! Always make backups, don’t blame me if this renders your system unusable or you lose any data!
This only works/only tested with FreeBSD 9.0 rc2, you have been warned!
Thanks to Philipp Wuensche and gkontos. See credits inside file.
enjoy. Feedback welcome to ok@monkeytower.net
regards.
olaf.
Last night I forgot to put my Iphone in silent mode and around 3:00 am I woke up from an email indicating that “A new comment on the post “Root On ZFS @ FreeBSD 9″ is waiting for your approval”.
Usually I don’t read my emails at 3:00am but this seemed to be very interesting. So, I read it.
I soon started to realize that I must be a complete moron!
I was trying to create a script that would be inserted into a custom FreeBSD distribution, when a simple fetch could do the job!!!
Thanks Olaf for this fantastic work!!!
there is still an issue with this implementation:
if you set up a mirror, start the system and disconnect one drive, the system will continue working, but will panic when you try to reboot.
I found out it has something to do with swap! When disabling the gpt/swap partitions by removing the entries from /etc/fstab the system will then (after a fresh start) survive a reboot command without panic.
Probably system would also panic when it would need to use swap first time and finds one drive removed.. i only testet without load…
can someone confirm this behavior? I tested with onbord s-ata and 3-ware (twed) controller.
There is also a way to put swap into the pool. i think that is the solution. what is best implementation for that?
thanks for your replies.
Mirroring swap can be tricky.
One way is to specify both of your swap partitions in your fstab.
The other is to use gmirror and load the relevant swap partition.
“gmirror label -b prefer swap gpt/swap0 gpt/swap1″
“/dev/mirror/swap none swap”
In the first case if you disconnect a disk the system might panic when it tries to mount the swap from the disconnected drive. (it shouldn’t actually)
on FreeBSD-wiki (http://wiki.freebsd.org/RootOnZFS) i found:
2.2. ZFS Swap Volume
Creating a swap partition on the ZFS Filesystem using a ZFS Volume:
Fixit# zfs create -V 2G zroot/swap
Fixit# zfs set org.freebsd:swap=on zroot/swap
Fixit# zfs set checksum=off zroot/swap
is it really that easy?
Yes! The only disadvantage is that you can not store core dumps on ZFS swap. But who cares …
Hi Olaf
Thanks for this successful script. It saved my time. May be swap on ZFS can be added to this script as an option. Also if user select ZFS mirror option, script must be create gmirror automatically on gpt swap partitions.
Regards.
Hasan Alp İNAN
Hi, gkontos:
I think that you have to run “gpart show” after “gpart create -s gpt ada0″. You must know LBA number for freebsd-boot. This number has value 34 or 40.
See “http://en.wikipedia.org/wiki/GUID_Partition_Table” key word “LBA 40″.
Well, there’s also the disadvantage that if you run out of RAM you might not be able to write to swap in some circumstances, as ZFS needs to allocate RAM to write to disk… (And that if you are doing snapshots on zroot, you’ll be taking snapshots of swap.)
I’m currently running swap on ZFS. When I first installed, I was short on RAM. This caused lockups. I added RAM, but I remain aware that doing a RAM-intensive process could crash my box.
Hi, DStaal:
I think your problem is small RAM memory. The ZFS requires memory bigger than 1GB! Please, check your RAM size.
Hi, gkontos:
Please, detailed shell-commands after following words:
>>Final steps
What is goal these steps?
and this detailed please:
>>The pool is now mounted under /mnt and we can now proceed with the final steps.
P.S.:
Your article is very cool work!
This tutorial worked great with 9.0 RC2. One change that is needed is /usr/ports needs to have exec enabled so:
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/usr/ports
should get the job done. Without this option the configure scripts cannot be executed during a make.
Hello Matt,
thanks, i updated the script at
http://www.monkeytower.net/go9.sh
regards.
.olaf
P.S: is it only me who has to try at least 3 captchas on this site until messages gets posted? (and i really took care to type correctly..)