In this article we will create two identical HAST nodes, hast1 and hast2. Both devices will use one NIC connected to a vlan for data synchronization and another NIC will be configured via CARP in order to share the same IP address across the network. The first node will be called “storage1.hast.test”, the second “storage2.hast.test” and they will both listen to a common IP address which we will bind to “storage.hast.test”

HAST binds its resource names according to the machine’s hostname. Therefore, we will use “hast1.freebsd.loc” and “hast2.freebsd.loc”  as the machines’s hostnames so that HAST can operate without complaining.

For starters, lets set up two identical nodes. For this example I have installed FreeBSD 9.0-RELEASE on two deferent instances using a Linux KVM. Both nodes have 512MB of RAM, one SATA drive containing the OS and three SATA drives which will be used to create our shared Raidz1 pool. The final result looks like this:

In order for carp to work we don’t have to compile a new kernel. We can just load it as a module by adding to /boot/loader.conf

 if_carp_load="YES"

Our both nodes are set up, it is time to make some adjustments. First a descent /etc/rc.conf for the first node:

zfs_enable="YES"
 
###Primary Interface##
ifconfig_re0="inet 10.10.10.181  netmask 255.255.255.0"
 
###Secondary Interface for HAST###
ifconfig_re1="inet 192.168.100.100  netmask 255.255.255.0"
 
defaultrouter="10.10.10.1"
sshd_enable="YES"
hostname="hast1.freebsd.loc"
 
##CARP INTERFACE SETUP##
cloned_interfaces="carp0"
ifconfig_carp0="inet 10.10.10.180 netmask 255.255.255.0 vhid 1 pass mypassword advskew 0"
 
hastd_enable=YES

The second node we will also much the first except for the IP addressing:

zfs_enable="YES"
 
###Primary Interface##
ifconfig_re0="inet 10.10.10.182  netmask 255.255.255.0"
 
###Secondary Interface for HAST###
ifconfig_re1="inet 192.168.100.101  netmask 255.255.255.0"
 
defaultrouter="10.10.10.1"
sshd_enable="YES"
hostname="hast2.freebsd.loc"
 
##CARP INTERFACE SETUP##
cloned_interfaces="carp0"
ifconfig_carp0="inet 10.10.10.180 netmask 255.255.255.0 vhid 1 pass mypassword advskew 100"
 
hastd_enable=YES

At this point we have assigned re1 with two IPs for HAST synchronization. We have also assigned two IPs to re0 which in turn we share with a third common IP assigned to carp0.
As a result, re1 is being used for HAST synchronization in a vlan while carp0 which is cloned by re0 used under the same vlan with the rest of our clients.

In order for HAST to function correctly we have to resolve the correct IPs on every node. We don’t want to rely on DNS for this because DNS can fail. Instead we will use /etc/hosts same on every node.

::1			localhost localhost.freebsd.loc
127.0.0.1		localhost localhost.freebsd.loc
192.168.100.100		hast1.freebsd.loc hast1
192.168.100.101		hast2.freebsd.loc hast2
 
10.10.10.181          	storage1.hast.test storage1
10.10.10.182          	storage2.hast.test storage2
10.10.10.180	      	storage.hast.test  storage

Next, we have to create the /etc/hast.conf file. Here we will declare the resources that we want to create. All resources will eventually create devices located under /dev/hast on the primary node. Every resource indicates a physical device specifying a local and remote IP device. The /etc/hast.conf must be exactly the same on every node.

resource disk1 {
        on hast1 {
                local /dev/ad1
                remote hast2
        }
        on  hast2 {
                local /dev/ad1
                remote hast1
        }
}
 
resource disk2 {
        on  hast1 {
                local /dev/ad2
                remote hast2
        }
        on  hast2 {
                local /dev/ad2
                remote hast1
        }
}
 
resource disk3 {
        on  hast1 {
                local /dev/ad3
                remote hast2
        }
        on  hast2 {
                local /dev/ad3
                remote hast1
        }
}

In this example we are sharing three resources, disk1, disk2 and disk3. Each resource indicates a device the local and the remote IP address. With this configuration in place, we are ready to begin setting up out HAST devices.

Lets start hastd on both nodes first:

hast1#/etc/rc.d/hastd start

hast2#/etc/rc.d/hastd start

Now on the primary node we will initialize our resources, create them and finally assign a primary role:

hast1#hastctl role init disk1
hast1#hastctl role init disk2
hast1#hastctl role init disk3
hast1#hastctl create disk1
hast1#hastctl create disk2
hast1#hastctl create disk3
hast1#hastctl role primary disk1
hast1#hastctl role primary disk2
hast1#hastctl role primary disk3

Next, on the secondary node we will initialize our resources, create them and finally assign a secondary role:

hast2#hastctl role init disk1
hast2#hastctl role init disk2
hast2#hastctl role init disk3
hast2#hastctl create disk1
hast2#hastctl create disk2
hast2#hastctl create disk3
hast2#hastctl role secondary disk1
hast2#hastctl role secondary disk2
hast2#hastctl role secondary disk3

There are other ways for creating and assigning roles to each resource. Having repeat this procedure a few times, I saw that this usually always works.

Now check the status on both nodes:

hast1# hastctl status
disk1:
  role: primary
  provname: disk1
  localpath: /dev/ada1
  ...
  remoteaddr: hast2
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...
disk2:
  role: primary
  provname: disk2
  localpath: /dev/ada2
  ...
  remoteaddr: hast2
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...
disk3:
  role: primary
  provname: disk3
  localpath: /dev/ada3
  ...
  remoteaddr: hast2
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...

The first node looks good. Status is complete.

hast2# hastctl status
disk1:
  role: secondary
  provname: disk1
  localpath: /dev/ada1
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...
disk2:
  role: secondary
  provname: disk2
  localpath: /dev/ada2
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...
disk3:
  role: secondary
  provname: disk3
  localpath: /dev/ada3
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...

So does the second. Like I mentioned earlier there are different ways for doing this the first time. You have to look for the word status: complete. If you get a degraded status you can always repeat the procedure.

Now it is time to create our ZFS pool. The primary node should have a /dev/hast directory containing our resources. This directory appears only at the active node.

hast1# zpool create zhast raidz1 /dev/hast/disk1 /dev/hast/disk2 /dev/hast/disk3
hast1# zpool status zhast
 pool: zhast
 state: ONLINE
 scan: none requested
 config:
 
	NAME            STATE     READ WRITE CKSUM
	zhast           ONLINE       0     0     0
	  raidz1-0      ONLINE       0     0     0
	    hast/disk1  ONLINE       0     0     0
	    hast/disk2  ONLINE       0     0     0
	    hast/disk3  ONLINE       0     0     0

We can now use hastctl status on each node to see if everything looks ok. The magic word we are looking for here is: replication: fullsync

At this point both of our nodes should be available for failover. We have storage1 running as primary and sharing a pool called zhast. Our storage2 is currently in a standby mode. If we have set DNS properly we can ssh to storage.hast.test or by using its carp IP to 10.10.10.180.

In order to perform a failover we have to first export our pool from the first node, change the role of each resource to secondary. Then change the role of each resource to primary on the standby node and import the pool. This procedure will be done manually to test if failover really works. But for a real HA solution we will eventually create a script that will take care of this.

First lets export our pool and change our resources role:

hast1# zpool export zhast
hast1# hastctl role secondary disk1
hast1# hastctl role secondary disk2
hast1# hastctl role secondary disk3

Now, lets reverse the procedure on the standby node:

hast2# hastctl role primary disk1
hast2# hastctl role primary disk2
hast2# hastctl role primary disk3
hast2# zpool import zhast

The roles have successfully changed, lets see our pool status:

hast2# zpool status zhast
 pool: zhast
 state: ONLINE
 scan: none requested
 config:
 
	NAME            STATE     READ WRITE CKSUM
	zhast           ONLINE       0     0     0
	  raidz1-0      ONLINE       0     0     0
	    hast/disk1  ONLINE       0     0     0
	    hast/disk2  ONLINE       0     0     0
	    hast/disk3  ONLINE       0     0     0
 
errors: No known data errors

Again, by using hastctl status on each node we can verify that the roles have indeed changed and that the status is complete. This is a sample output from the second node now in charge:

hast2# hastctl status
disk1:
  role: primary
  provname: disk1
  localpath: /dev/ad1
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  ...
disk2:
  role: primary
  provname: disk2
  localpath: /dev/ad2
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  ...
disk3:
  role: primary
  provname: disk3
  localpath: /dev/ad3
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  ...

It is now time to automate this procedure. When do we want our servers to automatically failover?
One reason would be if the primary node is not responding to the external network thus not being able to serve its clients. Using a devd event we can catch a carp interface going up or down and a state change.

Add the following lines to /etc/devd.conf on both nodes:

notify 30 {
	match "system" "IFNET";
	match "subsystem" "carp0";
	match "type" "LINK_UP";
	action "/usr/local/bin/failover master";
};
 
notify 30 {
	match "system" "IFNET";
	match "subsystem" "carp0";
	match "type" "LINK_DOWN";
	action "/usr/local/bin/failover slave";
};

And now lets create the failover script which will be responsible for doing automatically what we did before manually:

#!/bin/sh
 
# Original script by Freddie Cash <fjwcash@gmail.com>
# Modified by Michael W. Lucas <mwlucas@BlackHelicopters.org>
# and Viktor Petersson <vpetersson@wireload.net>
# Modified by George Kontostanos <gkontos.mail@gmail.com>
 
# The names of the HAST resources, as listed in /etc/hast.conf
resources="disk1 disk2 disk3"
 
# delay in mounting HAST resource after becoming master
# make your best guess
delay=3
 
# logging
log="local0.debug"
name="failover"
pool="zhast"
 
# end of user configurable stuff
 
case "$1" in
	master)
		logger -p $log -t $name "Switching to primary provider for ${resources}."
		sleep ${delay}
 
		# Wait for any "hastd secondary" processes to stop
		for disk in ${resources}; do
			while $( pgrep -lf "hastd: ${disk} \(secondary\)" > /dev/null 2>&1 ); do
				sleep 1
			done
 
			# Switch role for each disk
			hastctl role primary ${disk}
			if [ $? -ne 0 ]; then
				logger -p $log -t $name "Unable to change role to primary for resource ${disk}."
				exit 1
			fi
		done
 
		# Wait for the /dev/hast/* devices to appear
		for disk in ${resources}; do
			for I in $( jot 60 ); do
				[ -c "/dev/hast/${disk}" ] && break
				sleep 0.5
			done
 
			if [ ! -c "/dev/hast/${disk}" ]; then
				logger -p $log -t $name "GEOM provider /dev/hast/${disk} did not appear."
				exit 1
			fi
		done
 
		logger -p $log -t $name "Role for HAST resources ${resources} switched to primary."
 
 
		logger -p $log -t $name "Importing Pool"
		# Import ZFS pool. Do it forcibly as it remembers hostid of
                # the other cluster node.
                out=`zpool import -f "${pool}" 2>&1`
                if [ $? -ne 0 ]; then
                    logger -p local0.error -t hast "ZFS pool import for resource ${resource} failed: ${out}."
                    exit 1
                fi
                logger -p local0.debug -t hast "ZFS pool for resource ${resource} imported."
 
	;;
 
	slave)
		logger -p $log -t $name "Switching to secondary provider for ${resources}."
 
		# Switch roles for the HAST resources
		zpool list | egrep -q "^${pool} "
        	if [ $? -eq 0 ]; then
                	# Forcibly export file pool.
                	out=`zpool export -f "${pool}" 2>&1`
               		 if [ $? -ne 0 ]; then
                        	logger -p local0.error -t hast "Unable to export pool for resource ${resource}: ${out}."
                        	exit 1
                	 fi
                	logger -p local0.debug -t hast "ZFS pool for resource ${resource} exported."
        	fi
		for disk in ${resources}; do
			sleep $delay
			hastctl role secondary ${disk} 2>&1
			if [ $? -ne 0 ]; then
				logger -p $log -t $name "Unable to switch role to secondary for resource ${disk}."
				exit 1
			fi
			logger -p $log -t $name "Role switched to secondary for resource ${disk}."
		done
	;;
esac

Let’s try it and see if it works. Log into both the currently active and standby node. Make sure that you are on the active by issuing a hastctl status command. Then force a failover by bringing the interface which is associated with carp0 downL

hast1# ifconfig er0 down

Watch at the generated messages:

hast1# tail -f /var/log/debug.log
 
Feb  6 15:01:41 hast1 failover: Switching to secondary provider for disk1 disk2 disk3.
Feb  6 15:01:49 hast1 hast: ZFS pool for resource  exported.
Feb  6 15:01:52 hast1 failover: Role switched to secondary for resource disk1.
Feb  6 15:01:55 hast1 failover: Role switched to secondary for resource disk2.
Feb  6 15:01:58 hast1 failover: Role switched to secondary for resource disk3.

hast2# tail -f /var/log/debug.log
 
Feb  6 15:02:15 hast2 failover: Switching to primary provider for disk1 disk2 disk3.
Feb  6 15:02:19 hast2 failover: Role for HAST resources disk1 disk2 disk3 switched to primary.
Feb  6 15:02:19 hast2 failover: Importing Pool
Feb  6 15:02:52 hast2 hast: ZFS pool for resource  imported.

Voila! The failover worked like a charm and now hast2 has assumed the primary role.

Further considerations:

What we did today is a basic setup of two nodes sharing a raidz1 pool with automatic role failover in case of a failure that would result in a loss of a carp interface.
Obviously, a similar devd event would be generated in case we loose a HAST replication interface. This is something that needs to be addressed similarly since losing that interface will leave us with no synchronization at all.
Going further, we would have to add scripts that will bring up and down services during a failover.
 

 

You can use this as a reference guide for a single or mirror installation.

(1) Boot from a FreeBSD9 installation DVD or memstick and choose “Live CD”.

(2) Create the necessary partitions on the disk(s) and add ZFS aware boot code.

a) For a single disk installation.

gpart create -s gpt ada0
gpart add -b 34 -s 94 -t freebsd-boot ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

b) Repeat the procedure for the second drive if you want a mirror installation.

gpart create -s gpt ada1
gpart add -b 34 -s 94 -t freebsd-boot ada1
gpart add -t freebsd-zfs -l disk1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

(3) Align the Disks for 4K and create the pool.

a) For a single disk installation.

gnop create -S 4096 /dev/gpt/disk0
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.nop
zpool export zroot
gnop destroy /dev/gpt/disk0.nop
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot

b) For a mirror installation.

gnop create -S 4096 /dev/gpt/disk0
gnop create -S 4096 /dev/gpt/disk1
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot mirror /dev/gpt/disk0.nop /dev/gpt/disk1.nop
zpool export zroot
gnop destroy /dev/gpt/disk0.nop
gnop destroy /dev/gpt/disk1.nop
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot

(4) Set the bootfs property and checksums.

zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot

(5) Create appropriate filesystems (feel free to improvise!).

zfs create zroot/usr
zfs create zroot/usr/home
zfs create zroot/var
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp

(6) Add swap space and disable checksums. In this case I add 4GB of swap.

zfs create -V 4G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set checksum=off zroot/swap

(7) Create a symlink to /home and fix some permissions.

chmod 1777 /mnt/tmp
cd /mnt ; ln -s usr/home home
chmod 1777 /mnt/var/tmp

(8) Instal FreeBSD.

sh
cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

(9) Copy zpool.cache (very important!!!)

cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache

(10) Create the rc.conf, loader.conf and an empty fstab (otherwise the system will complain).

echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
touch /mnt/etc/fstab

Reboot, adjust time zone info, add a password for root, add a user and enjoy!!!
 

 

Today, we will see how we can use PF to effectively secure a FreeBSD host on the Internet.
Before we even start talking about PF, it is essential to point out a few things about network firewalls.

• Network firewalls as opposed to packet filtering devices use statefull inspection in order to decide if a packet is allowed to pass or not.
• Statefull inspection was inspired by the concept of state and the 3-way handshake which we find only on TCP type base connections.
• Statefull inspection has progressed a lot over the time, making it somehow possible to maintain it on stateless protocols such us UDP or ICMP.

Network firewalls work mainly on layers 3 and 4 of the OSI model. Therefore, application inspection is not possible since it requires the ability to inspect at the layer 7.

General DO & DON’T

• Log everything! (except Internet noise)
• Be similarly restrictive to inbound and outbound traffic!
• Process your most used rules first, respect your CPU and memory!
• Don’t be tempted into using exotic rules that you read somewhere to prevent port scans!

Let’s start by building our first policy for a webserver:

### macro definitions
ext_if = "em0"
dns_server = "123.123.123.123"
webservices = "{80, 443, 22}"
icmp_types = "echoreq"
netbios_tcp = "{445, 137, 138, 139}"
netbios_udp = "{445, 137, 138, 139}"
tcp_out = "{5999, 80, 21}"

First we have define our macros. Macros are variables in a sense, holding information regarding our interfaces and tcp / udp ports. It is always a good idea to group port numbers since it can greatly reduce our rules

### all incoming traffic on external interface is normalized and fragmented
### packets are reassembled.
scrub in on $ext_if all fragment reassemble
 
### exercise antispoofing on the external interface, but add the local
### loopback interface as an exception, to prevent services utilizing the
### local loop from being blocked accidentally.
set skip on lo0
antispoof for $ext_if inet

Next we tell the firewall to reassemble all fragmented packets, skip any packet filtering rules for the loopback interface and finally to check for spoofed IP addresses on the external interface.

### get rid quick of Internet noise like microsoft netbios service.
### This accounts to 80% of dropped traffic. We don't need to log this also
block in quick on $ext_if proto tcp from any to any port $netbios_tcp
block in quick on $ext_if proto udp from any to any port $netbios_udp

This part you don’t usually find in PF rules but I think that it is worth having it.
If you look at the logs on an Internet faced firewall, you will see that most of the dropped traffic is netbios. These are usually Zombie hosts infected by some nasty old windows virus.
By blocking them first without logging this traffic, we relax both our firewall from having to read all the rules before it drops this. We also maintain cleaner logs.

###clean up rule
block log all

Finally, we tell our firewall to block any traffic, inbound and outbound, that doesn’t much a rule. We also like to log everything.

### set a rule that allows inbound traffic with synproxy handshaking.
pass in quick log on $ext_if proto tcp from any to any port $webservices flags S/SA synproxy state
pass in quick log inet proto icmp all icmp-type $icmp_types keep-state

Like I mentioned earlier, the most used rules should come first. Since this is a web server, we allow http, https and ssh incoming traffic. We also like to be able to ping our server. Notice that I am using the “synproxy state” keyword. That way I am instructing the firewall to proxy the 3-way tcp handshake, keeping syn flood attacks away.

### keep state on any outbound tcp, udp or icmp traffic. modulate the isn of
### outgoing packets. (initial sequence number) broken operating systems
### sometimes don't randomize this number, making it guessable.
pass out quick log on $ext_if proto tcp from any to any port $tcp_out modulate state
pass out quick log on $ext_if proto udp from any to $dns_server port domain modulate state
pass out quick log inet proto icmp all icmp-type $icmp_types keep-state

Finally, I allow my webserver outbound access for http, ftp and cvs. This could and should become stricter and allow this type of communication only to certain destinations.

This is just an example of a host based firewall. A real firewall would have at least two interfaces and would probably perform NAT. The point is that an effective policy is a policy that will not create extra burden, will be clean and easy to read and maintain.

Always start by processing your most used rules first. Group ports and servers and log all interesting traffic.

There are two prerequisites for this procedure to work. First, the machine that is receiving the snapshots must be running the same or higher ZFS version from the machine that is being backed up. Second, we must either have root access to the receiving machine or an account that has been delegated with create, receive ZFS properties.

First lets prepare both host and target machines. We need to create and send our first full snapshot from our host to a ZFS folder on the target.

The host:

core2duo# zfs snapshot -r zroot/usr/src@-2012-01-10
core2duo# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot/usr/src 349M 64.4G 349M /usr/src
zroot/usr/src@-2012-01-10 0 - 349M -
zroot/usr/src/mytest 31K 64.4G 31K /usr/src/mytest
zroot/usr/src/mytest@-2012-01-10 0 - 31K -

The target:

hp# zfs create tank/test
hp# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank/test 40.0K 747G 40.0K /tank/test

Now, lets send our first full snapshot:

core2duo# zfs send -R zroot/usr/src@-2012-01-10 | ssh root@hp zfs receive -Fduv tank/test

Password:
receiving full stream of zroot/usr/src@-2012-01-10 into tank/test/usr/src@-2012-01-10
received 732MB stream in 72 seconds (10.2MB/sec)
receiving full stream of zroot/usr/src/mytest@-2012-01-10 into tank/test/usr/src/mytest@-2012-01-10
received 47.4KB stream in 1 seconds (47.4KB/sec)

Now, lets create a new snapshot and send them both incremental.

core2duo# zfs snapshot -r zroot/usr/src@-2012-01-11
core2duo# zfs send -R -i zroot/usr/src@-2012-01-10 zroot/usr/src@-2012-01-11 | ssh root@hp zfs receive -Fduv tank/test

receiving full stream of zroot/usr/src@-2012-01-10 into tank/test/usr/src@-2012-01-10
received 732MB stream in 72 seconds (10.2MB/sec)
receiving full stream of zroot/usr/src/mytest@-2012-01-10 into tank/test/usr/src/mytest@-2012-01-10
received 47.4KB stream in 1 seconds (47.4KB/sec)

Notice that I am using some special switches in my send/receive commands.
When sending, -R allows me to send the snapshots, their children and their properties.
When receiving, -F forces a rollback to the most recent snapshot, -d maintains my naming scheme and -u makes sure that the associated file systems do not get mounted.

Lets now see what we have done so far, first the host:

core2duo# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot/usr/src 349M 64.4G 349M /usr/src
zroot/usr/src@-2012-01-11 0 - 349M -
zroot/usr/src/mytest 31K 64.4G 31K /usr/src/mytest
zroot/usr/src/mytest@-2012-01-11 0 - 31K -

And the target:

hp# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank/test 364M 747G 41.3K /tank/test
tank/test/usr 364M 747G 40.0K /tank/test/usr
tank/test/usr/src 364M 747G 364M /tank/test/usr/src
tank/test/usr/src@-2012-01-10 1.33K - 364M -
tank/test/usr/src@-2012-01-11 0 - 364M -
tank/test/usr/src/mytest 65.3K 747G 40.0K /tank/test/usr/src/mytest
tank/test/usr/src/mytest@-2012-01-10 25.3K - 40.0K -
tank/test/usr/src/mytest@-2012-01-11 0 - 40.0K -

Looks like it is working. The only thing left is to somehow automate the procedure. Let’s create a script that will create a snapshot which we will call it today, check if there was a snapshot yesterday and send them incrementally to a remote host.

#!/bin/sh
 
pool="zroot/usr/src"
destination="tank/test"
host="10.10.10.4"
 
today=`date +"$type-%Y-%m-%d"`
yesterday=`date -v -1d +"$type-%Y-%m-%d"`
 
# create today snapshot
snapshot_today="$pool@$today"
# look for a snapshot with this name
if zfs list -H -o name -t snapshot | sort | grep "$snapshot_today$" > /dev/null; then
echo " snapshot, $snapshot_today, already exists"
exit 1
else
echo " taking todays snapshot, $snapshot_today"
zfs snapshot -r $snapshot_today
fi
 
# look for yesterday snapshot
snapshot_yesterday="$pool@$yesterday"
if zfs list -H -o name -t snapshot | sort | grep "$snapshot_yesterday$" > /dev/null; then
echo " yesterday snapshot, $snapshot_yesterday, exists lets proceed with backup"
 
zfs send -R -i $snapshot_yesterday $snapshot_today | ssh root@$host zfs receive -Fduv $destination
 
echo " backup complete destroying yesterday snapshot"
zfs destroy -r $snapshot_yesterday
exit 0
else
echo " missing yesterday snapshot aborting, $snapshot_yesterday"
exit 1
fi

pool is the ZFS pool we want to backup
destination is the destination pool that will receive the backup
host is our backup host!

With a cronjob in place this would be a nice candidate for a daily ZFS incremental backup policy. Just make sure you use public/private key for ssh authentication.
 

 

In this article I will add a few steps necessary when upgrading FreeBSD trough a major release. A good example is going from FreeBSD 8.2-RELEASE to FreeBSD 9.0-RELEASE. In reality the steps are more or less the same.

First and for most, make sure that your ports are up to date. If you are using portupgrade like myself do a:

 portsnap fetch update && portversion -v | grep "<" 

This will so you which ports if any need to be updated. Read /usr/ports/UPDATING for special instructions on ports that might affect you and upgrade all your ports. Follow the instructions you read and after you finish if you are using portupgrade this will do the job:

 portupgrade -arR

At this point you have your sources synchronized and your ports updated. Follow the instructions you read in the previous article with the exception of  running make installworld and mergemaster while in single user mode.

After your system boots it is time to rebuild all of your ports. If you are using portupgrade you will have to start by rubby:

cd /usr/ports/lang/ruby18/
make deinstall && make install clean

cd /usr/ports/ports-mgmt/portupgrade/
make deinstall && make install clean

Now that portupgrade is done continue with the rest:

 portupgrade -af

This should be enough. Reboot your system and you will have a perfect functional upgraded FreeBSD.
 

 

Today, I see that many people have started using the freebsd-update utility. I think it is good that a binary update option is available. But to be honest I have only used it once, out of curiosity mostly. Don’t get me wrong I am not the type of person who believes in hard core solutions. I just feel that going source gives you more control. Besides, a relatively descent box will not need more than an hour for the whole procedure of building world && kernel. Also, freebsd-update can not be used yet for tracking stable.

To all of you who want to go source but are afraid to,  keep reading.

Synchronize Source 

The first thing you have to do is to synchronize your sources with the release you want to follow. The cvsup utility was used for many years for this job. While it is still being described in the handbook as a method of updating your sources it is considered obsolete and has been replaced by csup which is integrated in to world. This means that you don’t need to install any extra packages in order to synchronize your sources.

You need however to use a sup file for the version you are following. In the directory below you will find many examples:

# cd /usr/share/examples/cvsup/ && ls
README doc-supfile ports-supfile refuse.README standard-supfilecvs-supfile gnats-supfile refuse stable-supfile www-supfile

From all these files only 2 are the ones that we are interested in.

stable-supfile
standard-supfile

The first one is used to track stable while the second one is used for release. So, grab a copy of standard and copy it to a directory, I usually keep my sup files under /root

Lets spend a few minutes to examine the important lines here.

# Defaults that apply to all the collections
#
# IMPORTANT: Change the next line to use one of the CVSup mirror sites
# listed at http://www.freebsd.org/doc/handbook/cvsup.html#CVSUP-MIRRORS.
*default host=CHANGE_THIS.FreeBSD.org

This specifies from where to get the source tree.

*default host=cvsup.freebsd.org

Is usually a good choice although you might consider using a mirror closer to you.

# The following line is for 8-stable. If you want 7-stable, 6-stable,
# 5-stable, 4-stable, 3-stable, or 2.2-stable, change to "RELENG_7",
# "RELENG_6", "RELENG_5", "RELENG_4", "RELENG_3", or "RELENG_2_2"
# respectively.
*default release=cvs tag=RELENG_9_0
*default delete use-rel-suffix

This is the most important part of the file. It specifies which source tree you wish to follow.
If you are tracking 8-STABLE or 9-STABLE it should read like this:

*default release=cvs tag=RELENG_8

*default release=cvs tag=RELENG_9

If you want to track a RELEASE it should read like this:

*default release=cvs tag=RELENG_8_0

*default release=cvs tag=RELENG_9_0

The first specifies 8.0-RELEASE while the second specifies 9.0-RELEASE respectably.

So, go ahead and adjust your sup file with the mirror you want to grab the source from and the version you want to follow. A typical supfile for the upcoming 8.3-RELEASE will look like this:

# Defaults that apply to all the collections
#
# IMPORTANT: Change the next line to use one of the CVSup mirror sites
# listed at http://www.freebsd.org/doc/handbook/cvsup.html#CVSUP-MIRRORS.
*default host=cvsup.FreeBSD.org
*default base=/var/db
*default prefix=/usr
# The following line is for 8-stable. If you want 7-stable, 6-stable,
# 5-stable, 4-stable, 3-stable, or 2.2-stable, change to "RELENG_7",
# "RELENG_6", "RELENG_5", "RELENG_4", "RELENG_3", or "RELENG_2_2"
# respectively.
*default release=cvs tag=RELENG_8_3
*default delete use-rel-suffix

Now that your supfile is complete, rename it into something memorable e.g release8.3 and fire csup with it.

 csup /root/release8.3

After the process is finished you will have a complete functional source tree that you can use it to build FreeBSD and update your distribution.

UPDATE FREEBSD

The whole procedure is divided into 4 different stages. Build world and kernel, Install kernel, Install world, mergemaster. After a while you will discover that those 4 different stages can be merged in 2. Build world and kernel, install kernel and world and mergemaster.

1) Build everything

cd /usr/src
make -j8 buildworld && make -j4 buildkernel

2) Install the new kernel

make install kernel
shutdown -r now

3) Install the new world 

cd /usr/src
make install world

4) Run mergemaster

mergemaster

That’s it, reboot once more and your system will be updated.

As I mentioned earlier there is a faster way of doing this without so many reboots.

cd /usr/src && make -j6 buildworld && make -j4 buildkernel
make installkernel
make istallworld
mergemaster
shutdown -r now

The above procedure is highly unorthodox but if you track STABLE regularly you will see that there are not so many changes in world and kernel.

Troubleshooting

What to do if the world or kernel refuses to compile ?

If you track stable then there is a slight chance that your source tree is partially synchronized  or just happened to hit a bug in the road.

a) Use csup again, there is a chance that you have missed an update that perhaps is stalling the procedure.
b) Check the stable mailing list for any know problems regarding the version you are following.
c) Clean your source tree like this:

chflags -R noschg /usr/obj/usr
rm -rf /usr/obj/usr
cd /usr/src
make cleandir
make cleandir

If you are tracking RELEASE then things should run more smoothly.

a) Make sure that you are following the correct RELEASE by checking your supfile.
b) Repeat the above procedure for cleaning up your source tree.
c) Remove any tuning you might have in make.conf

My kernel doesn’t boot !

No problem, you can always boot your previous kernel. FreeBSD automatically saves your kernel in /boot/kernel.old The procedure is fairly easily, just choose command line from the boot menu and:

>unload
ok
>load /boot/kernel.old/kernel.ko
>ok
>boot

Notes:
In the handbook it is recommended to update the world files while in single user mode. During my involvement with FreeBSD, I saw that there is really no special need for this.

Next, see how to perform a major UPGRADE

In this guide I will demonstrate how you can install a fully functional full ZFS FreeBSD9 using a GPT scheme. We will also use ZFS for SWAP

You can use this as a reference guide for a single or mirror installation.

(1) Boot from a FreeBSD9 installation DVD or memstick and choose “Live CD”.

(2) Create the necessary partitions on the disk(s) and add ZFS aware boot code.

a) For a single disk installation.

gpart create -s gpt ada0
gpart add -b 34 -s 94 -t freebsd-boot ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

b) Repeat the procedure for the second drive if you want a mirror installation.

gpart create -s gpt ada1
gpart add -b 34 -s 94 -t freebsd-boot ada1
gpart add -t freebsd-zfs -l disk1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

(3) Create the pool.(ignore any warnings regarding mounting)

a) For a single disk installation.

zpool create zroot /dev/gpt/disk0

b) For a mirror installation.

zpool create zroot mirror /dev/gpt/disk0 /dev/gpt/disk1

(4) Set bootfs property, checksums and mountpoints.

zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot
zfs set mountpoint=/mnt zroot

(5)) At this point export and import the pool while preserving zroot.cache in /var/tmp.

zpool export zroot
zpool import -o cachefile=/var/tmp/zpool.cache zroot

(6) Create appropriate filesystems (feel free to improvise!).

zfs create zroot/usr
zfs create zroot/usr/home
zfs create zroot/var
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp

(7) Add swap space and disable checksums. In this case I add 4GB of swap.

zfs create -V 4G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set checksum=off zroot/swap

(8) Create a symlink to /home and fix some permissions.

chmod 1777 /mnt/tmp
cd /mnt ; ln -s usr/home home
chmod 1777 /mnt/var/tmp

(9) Instal FreeBSD.

sh
cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

(10) Copy zpool.cache (very important!!!)

cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache

(11) Create the rc.conf, loader.conf and an empty fstab (otherwise the system will complain).

echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
touch /mnt/etc/fstab

(12) Unmount everything and fix mountpoints for system boot.

zfs set readonly=on zroot/var/empty
zfs umount -af
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var

Reboot, adjust time zone info, add a password for root, add a user and enjoy!!!

If you are looking for a 4K optimized guide see my next guide!

A network firewall like the one you have in your company, works in the 3rd and 4th level of the OSI model. That is the network and transport layer. It’s primary role is to permit or deny packets to flow based upon certain access lists using statefull inspection. It can also detect and prevent certain network attacks .

An application firewall on the other hand works only in the 7th layer of the OSI model. It’s primary role is to inspect if a request that originates from a client towards the application server is legitimate. The application firewall will determine the legitimacy of that request based upon protocol enforcement and signatures that contain common illegitimate requests.

Today, most attacks occur on web applications and web servers that are exposed to the internet. Network firewalls allow incoming traffic to http & https ports which means that our applications are exposed. The more complex an application is the more difficult is to secure it in the code level. Sql injections, code injections and cross site scripting attacks are them most common type of attacks. Here is where web application firewalls come in to give us a helping hand in avoiding this type of attacks. The bad thing is that they don’t come cheap. Fortunately there is open source software available and a very good and effective.

Implementing mod_security on FreeBSD

Mod_security started as an IDS for apache and has evolved into a powerful web application firewall that can be deployed in a web server. In FreeBSD it is offered from the ports collection but it is still in version 2.5.13.

 cd /usr/ports/www/mod_security/ && make install clean

This will install mod_security under /usr/local/etc/apache22/Includes. Contrary to the messages you will receive after the installation assuring you that everything has been configured, mode_security will need some further installation steps before it can work even in log only mode.

First you have to download the latest rules. You can grab the latest tarball from here. Extract them and copy everything to /usr/local/etc/apache22/Includes/mod_security2. Your directory should look similar to this:

drwxr-xr-x  9 root  wheel    15B Sep 12 22:18 .
drwxr-xr-x  3 root  wheel    20B Sep 15 14:02 ..
-rw-r--r--  1 root  wheel    27k Nov 29  2010 CHANGELOG
-rw-r--r--  1 root  wheel    17k Nov 29  2010 LICENSE
-rw-r--r--  1 root  wheel    16k Nov 29  2010 README
drwxr-xr-x  2 root  wheel    24B Sep 21 13:49 activated_rules
drwxr-xr-x  2 root  wheel    24B Jul 20 19:42 base_rules
drwxr-xr-x  2 root  wheel    18B Jul 20 19:42 experimental_rules
drwxr-xr-x  2 root  wheel     6B Jul 20 19:42 lua
-rw-r--r--  1 root  wheel     8k Sep 13 12:01 modsecurity_crs_10_config.conf
-rw-r--r--  1 root  wheel   2.7k Nov 29  2010 modsecurity_crs_48_local_exceptions.conf.example
drwxr-xr-x  2 root  wheel    18B Jul 20 19:42 optional_rules
drwxr-xr-x  2 root  wheel    30B Jul 20 19:42 slr_rules
drwxr-xr-x  2 root  wheel     9B Sep 12 19:27 util

Now, edit /usr/local/etc/apache22/Includes/mod_security2.conf  and change the values to reflect the location of your configuration files.

Include /usr/local/etc/apache22/Includes/mod_security2/*.conf
Include /usr/local/etc/apache22/Includes/mod_security2/activated_rules/*.conf

As you see above there is a directory called /activated_rules. A good practice is to use this directory for all your rules and copy there all rules you wish to enable from other directories. Start by copying the base_rules. That way when you change a rule you can still have the original copy kept.

It is time now to navigate and explore the modsecurity_crs_10_config.conf located in the base directory of mod_security installation which is located under: /usr/local/etc/apache22/Includes/mod_security2/

This file contains the core configuration of mod_security and it is pretty well commented so it is worth browsing it for a while. Keep backups and start adjusting the values to suit your needs. However, like I mentioned earlier, the default configuration doesn’t block anything so once you are comfortable with it you will have to make a few adjustments.

WARNING WARNUNG ADVERTENCIA

The following changes will put mod_security in blocking mode. Try them on a testing environment before putting them into production.

Activate rules

SecRuleEngine On
SecDefaultAction log,auditlog,pass,status:403,phase:2,t:lowercase

Enable the audit engine for relevant traffic only. Don’t forget to rotate the logs later.

SecAuditEngine RelevantOnly
SecAuditLog /var/log/httpd-audit.log

Restart apache, check that your sites are working and keep an eye on httpd-error.log . That is where mode_security logs all denied traffic. One thing is for sure, you will get a few false positives. The good thing is that you can catch them very fast if you have someone trying to access all the applications that you serve.

The files that according to my experience will create the most false positives are:

• modsecurity_crs_20_protocol_violations.conf

• modsecurity_crs_41_sql_injection_attacks.conf

The error logs will tell you the type of violation, the file and the line. So, you can act accordingly. If you feel that this is legit traffic then just comment the rule.

With mod_security the sky is the limit. You can create your own rules, create exceptions for certain applications, use rules that are specific to popular open source CMS.

Effective defense

Mod_security is only a tool to protect you against malicious attacks. Installing a web application firewall does not make you immune to hackers.

• Keep your access and errors logs for at least for a week. Make a habit of examining the error logs daily and combine them with access logs. That way you will know better how an intruder is targeting your site(s).

• Research the offensive IPs. Just a simple whois can reveal valuable information. You will often find that a certain type of attack originates from many different IPs. Look for a pattern, most likely they will come from the same ISP or from ISPs that belong to a certain country or area.

• Port scanning is considered illegitimate and should not be used without prior approval. However, you have every right to use it against offensive IPs. In most cases it will reveal that your attackers are zombies.

• Finally, use your network firewall to block those IPs before they reach your web server.

At that time the 13 inch 2.2GH Intel Core2Duo MacBook with 1Gb Ram looked very promising.

Bundled with Leopard, the successor of Tiger, it run really well and covered almost all of my needs. I used to get 4 hours of real battery time and I was very happy. Although I always though that the hardware was overpriced. I got so used to that thing that I decided to upgrade the memory because I wanted to run a couple of virtual machines as well. I must admit that with 2Gb Ram the MacBook became really fast. I started installing new software, got mac ports working too!

And then came Snow Leopard.  I was a bit skeptical to upgrade because there weren’t many new features. But it turned out that this was a good thing. In fact, you could say that Snow Leopard was the perfect continuation of Leopard. The OS became much faster, more reliable and less memory hungry. It was for me the ultimate OS one could pay for.

In the mean time, Apple started making tons of money and saw their stock price go high, by selling cell phones. Well, they called them smartphones but if you think about it for a while they are just overpriced cell phones. Those cell phones were based on a closed source hybrid darwin kernel with many bugs. Apple thinks that this new type of software should be passed on to personal computers as well. So, OS X Lion was born…

Why Lion Sucks

Some features that supposedly will change the way you work:

Multi-Touch Gestures

It is so useless that I wouldn’t even mention anything about this. Except Apple who things that is so cool that it has to be the number one new feature in their web side about Lion.

Full-Screen Apps

That was something missing (?) Most OS can run an application in full screen and they don’t need to brag about it. With a Mac it is different because many applications refuse to resize properly in full screen mode. Unfortunately for Lion, and for you, if you try this you will soon end up searching how to escape from this mode. It is so annoying and complicated that I only used it once.

Mission Control

A better version of  “expose”. Nothing fancy, but nice to have.

Mac App Store

This is hardly a Lion feature. The App Store is available for Snow Leopard as well.

Launchpad

I don’t really need a Iphone style application menu. But again it is good to have.

Resume

Here things begin to get ugly. You open your wordproccessor and suddenly the last document you were working on appears. Hey, I am not that lazy, if I need that document I will open it again. It really sucks though when I have to close it just because Apple thinks I will work on it again.

 Some facts that will definitely change the way you work:

•  If you have 2Gb of Ram then you can just forget about it. Lion does very bad memory management and you soon end up drained, swaping for some memory.

• Your battery will drain very quick. I have read a few discussions regarding how Lion needs better battery calibration. Sorry but this is BS. The only reason why the battery time is reduced is because your CPU will work more intensively and your load averages will increase. When this happens the CPU temperature also rises and as a result the fans start spinning like hell. And it all come down to higher energy consumption.

• Be prepared to use the disk utility quite often since you will have to repair disk permissions very often.

• If you happen to own a NAS (Network Area Storage) to store and share your documents you might experience some issues with Samba protocol. AFP seems to be working without problems.

• Generally, you will have to be more patient as it will take more time for your applications to load. But don’t worry you can always get to watch the apple mouse spinning ball while you wait. Sometimes they might even crash also.

to be continued … (till then I switched back to Snow Leopard)
 

 

In OS X Lion 10.7.0 there is finally the option to use TimeMachine over your network as long as your storage supports AFP the Apple Filing Protocol.  Apple still recommends that you run OS X Lion server but as we will see it works pretty well with FreeBSD so now you can use your NAS to keep those backups, or even create a central repository for all the clients.

• First you have to install the AFP protocol provided by NetATalk.

Important!!! Make sure that you only select APPLETALK and remove ZEROCONF from the options!

cd /usr/ports/net/netatalk/ && make install clean

• Next edit /usr/local/etc/AppleVolumes.default and adjust to your needs:

/tank/apple/ Time Machine allow:gkontos options:tm,usedots

• Add the following to your rc.conf:

netatalk_enable="YES"
cnid_metad_enable="YES"
afpd_enable="YES"

• Start the service

/usr/local/etc/rc.d/netatalk start

And you are ready to connect, afp://xxx.xxx.xxx.xxx

Now open your Time Machine preferences and select your new volume as the backup volume. You should be able to run Time machine and backup your files.  Some extra configuration is needed in order to be able to perform a full restore with the boot DVD.

• Install Apple’s mDNSResponder:

cd /usr/ports/net/mDNSResponder && make install clean

• Create the file /usr/local/etc/rc.d/mdnsadv with the following content:

#!/bin/sh

# PROVIDE: mdnsadv
# REQUIRE: LOGIN mdnsd
#
# Add the following lines to /etc/rc.conf to enable the mDNSResponder service advertising:
#
# mdnsadv_enable="YES"

. /etc/rc.subr

name="mdnsadv"
rcvar=${name}_enable

command="/usr/local/bin/mDNSResponderPosix"

load_rc_config $name

: ${mdnsadv_enable="NO"}

command_args="-f /usr/local/etc/mdnsadv.conf >/dev/null 2>&1 &"

run_rc_command "$1"

• Don’t forget to make the file executable by issuing:

chmod ugo+x /usr/local/etc/rc.d/mdnsadv

• Create the advertising configuration file /usr/local/etc/mdnsadv.conf with the following contents:

AFP Server
_afpovertcp._tcp
548
title='File Server'

TM Volume
_adisk._tcp
9
sys=waMA=XX:XX:XX:XX:XX:XX,adVF=0x100
dk0=adVF=0x83,adVN=Backup

XX:XX:XX:XX:XX:XX is your storage mac address.

• Add to /etc/rc.conf the following line:

mdnsadv_enable="YES"

Now start mdnsadv service and you will be able to do a full restore from Apple’s boot DVD.

UPDATE October 2011

It looks like this will also work with Snow Leopard at least in version 10.6.8