I remember once that when I asked for remote console they gave me another customer’s server. I could reboot it, mount my own ISO and change the root password.

Now comes this:

Dear Client

At the end of last week, Hetzner technicians discovered a “backdoor” in one
of our internal monitoring systems (Nagios).

An investigation was launched immediately and showed that the administration
interface for dedicated root servers (Robot) had also been affected. Current
findings would suggest that fragments of our client database had been copied
externally.

As a result, we currently have to consider the client data stored in our Robot
as compromised.

To our knowledge, the malicious program that we have discovered is as yet
unknown and has never appeared before.

The malicious code used in the “backdoor” exclusively infects the RAM. First
analysis suggests that the malicious code directly infiltrates running Apache
and sshd processes. Here, the infection neither modifies the binaries of the
service which has been compromised, nor does it restart the service which has
been affected.

The standard techniques used for analysis such as the examination of checksum
or tools such as “rkhunter” are therefore not able to track down the malicious
code.

We have commissioned an external security company with a detailed analysis of
the incident to support our in-house administrators. At this stage, analysis
of the incident has not yet been completed.

The access passwords for your Robot client account are stored in our database
as Hash (SHA256) with salt. As a precaution, we recommend that you change your
client passwords in the Robot.

With credit cards, only the last three digits of the card number, the card type
and the expiry date are saved in our systems. All other card data is saved
solely by our payment service provider and referenced via a pseudo card number.
Therefore, as far as we are aware, credit card data has not been compromised.

Hetzner technicians are permanently working on localising and preventing possible
security vulnerabilities as well as ensuring that our systems and infrastructure
are kept as safe as possible. Data security is a very high priority for us. To
expedite clarification further, we have reported this incident to the data
security authority concerned.

Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in
regard to this incident.

Naturally, we shall inform you of new developments immediately.

We very much regret this incident and thank you for your understanding and
trust in us.

A special FAQs page has been set up at
http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further
enquiries.

Kind regards

Martin Hetzner

Hetzner Online AG
Stuttgarter Str. 1
91710 Gunzenhausen / Germany
Tel: +49 (9831) 61006-1
Fax: +49 (9831) 61006-2
security-mailing@hetzner.de
http://www.hetzner.com

Let us start by creating the tunnel.

Endpoint1
IPv6 address: fd58:0c86:b78c:d09a::86
IPv4 network: 10.10.10.0/24

#ifconfig gif0 create
#ifconfig gif0 ipv6 tunnel fd58:0c86:b78c:d09a::86 fd85:af00:29c1:f685::2093
#ifconfig gif0 alias 10.10.10.4 10.1.1.3

Endpoint2
IPv6 address: fd85:af00:29c1:f685::2093
IPv4 network: 10.1.1.0/24

#ifconfig gif0 create
#ifconfig gif0 ipv6 tunnel fd85:af00:29c1:f685::2093 fd58:0c86:b78c:d09a::86
#ifconfig gif0 alias 10.1.1.3 10.10.10.4

At this point the tunnel should be up and endpoint1 can ping endpoint2

ping 10.1.1.3
PING 10.1.1.3 (10.1.1.3): 56 data bytes
64 bytes from 10.1.1.3: icmp_seq=0 ttl=64 time=187.772 ms
64 bytes from 10.1.1.3: icmp_seq=1 ttl=64 time=184.516 ms
64 bytes from 10.1.1.3: icmp_seq=2 ttl=64 time=185.563 ms

Time to add IPsec into the game. For this you will need to compile your KERNEL adding the following options:

OPTIONS      IPSEC
DEVICE       crypto

Now, install ipsec-tools on both endpoints:

cd /usr/ports/security/ipsec-tools && make install clean

I usually keep all my configuration files in /usr/local/etc/racoon This directory does not exist so you will need to create it.

Working configuration for endpoint1:

psk.conf

fd58:0c86:b78c:d09a::86 averystrongsecret

setkey.conf

flush;
spdflush;
spdadd fd85:af00:29c1:f685::2093 fd58:0c86:b78c:d09a::86 any -P out ipsec esp/transport//require;
spdadd fd58:0c86:b78c:d09a::86 fd85:af00:29c1:f685::2093 any -P in ipsec esp/transport//require;

racoon.conf

path include "/usr/local/etc/racoon" ;
path    pre_shared_key  "/usr/local/etc/racoon/psk.txt"; #location of pre-shared key file
log     notify;	#log verbosity setting: set to 'notify' when testing and debugging is complete
 
padding	# options are not to be changed
{
        maximum_length  20;
        randomize       off;
        strict_check    off;
        exclusive_tail  off;
}
 
timer	# timing options. change as needed
{
        counter         5;
        interval        20 sec;
        persend         1;
#       natt_keepalive  15 sec;
        phase1          30 sec;
        phase2          15 sec;
}
 
listen	# address [port] that racoon will listening on
{
        isakmp          fd85:af00:29c1:f685::2093 [500];
}
 
remote  fd58:0c86:b78c:d09a::86 [500]
{
        exchange_mode   main,aggressive;
        doi             ipsec_doi;
        situation       identity_only;
        lifetime        time 8 hour;
        passive         off;
        proposal_check  obey;
#       nat_traversal   off;
        generate_policy off;
 
                        proposal {
                                encryption_algorithm    blowfish;
                                hash_algorithm          md5;
                                authentication_method   pre_shared_key;
                                lifetime time           30 sec;
                                dh_group                1;
                        }
}
sainfo anonymous 	# address $network/$netmask $type address $network/$netmask $type ( $type being any or esp)
{								# $network must be the two internal networks you are joining.
        pfs_group       1;
        lifetime        time    36000 sec;
        encryption_algorithm    blowfish,3des,des;
        authentication_algorithm        hmac_md5,hmac_sha1;
        compression_algorithm   deflate;
}

The same but opposite configuration should be applied to endpoint2!

Modify /etc/rc.conf

ipsec_enable="YES"
ipsec_file="/usr/local/etc/racoon/setkey.conf"
racoon_enable="YES"
racoon_flags="-l /var/log/racoon.log"

Assuming there is no firewall running at both endpoints you can start ipsec and racoon and all communication from endpoint1 to endpoint2 will be encrypted.
 
The following is a minimal PF configuration that will allow this to happen:

pass in log on $ext_if proto esp from $vpn_gateway 
pass in log on $ext_if proto ah from $vpn_gateway
pass in log on $ext_if proto ipencap from $vpn_gateway
pass in log on $ext_if inet6 proto udp from $vpn_gateway to any port 500

At this point all traffic should be traveling encrypted!
 
Finally modify again your /etc/rc.conf so that the tunnel comes up after a reboot:

cloned_interfaces="gif0"
gif_interfaces="gif0"
ifconfig_gif0_ipv6="inet6 tunnel fd85:af00:29c1:f685::2093 fd58:0c86:b78c:d09a::86"
ifconfig_gif0="alias 10.10.10.4 10.1.1.3"

The IPv6 world launch day has come and gone. But how ready are we for IPv6?

According to google we are not that ready yet. The fact is that goolge is not fully ready either! Their ads do not display over an IPv6 only network.

This post is the beginning of a series regarding a full dual stack implementation using FreeBSD as a dual stack Web Server, Mail server and DNS server. We will also cover the basic topics on how to implement a full dual stack network using FreeBSD as a IPv4 & IPv6 router/firewall.

For the time being, aisecure.net is a full dual stack web site. Meaning that it uses dual stack DNS servers, dual stack Web Server and dual stack mail server.

Most sites that get an IPv6 validation would not work if IPv4 was completely gone! Why? Because they rely on IPv4 only DNS servers that simply have AAAA records.

First of all lets see some interesting facts about IPv6:

IPsec

IPSec, is a framework of open standards that define policies for secure communication in a network. In addition, these standards also describe how to enforce these policies. Some of the interesting features of IPsec are:

• AH (Authentication Header) that provides authenticity guarantee for transported packets.
• ESP (Encapsulating Security Payload) that provides encryption of packets.

The most interesting feature is ESP which provides confidentiality. In other words, by using ESP all traffic between 2 hosts is encrypted.

When IPv6 was designed IPsec was supposed to be mandatory. Every IP communication used by IPv6 must use IPsec. Unfortunately, recent RFCs have changed the word “must” to “should”.  See here for more details.

In FreeBSD if we want to take advantage of the IPsec we still need to compile a new KERNEL as described in the handbook. I personally find this to be very disturbing and I believe that this option should be default.

IPv6 address space

The 128 bits of IPv6 addresses mean the size of the IPv6 address space is, quite literally, astronomical, like the numbers that describe the number of stars in a galaxy or the distance to the furthest pulsars, the number of addresses that can be supported in IPv6 is mind-boggling.

Since IPv6 addresses are 128 bits long, the theoretical address space if all addresses were used is 2¹²⁸ addresses. This number, when expanded out, is 340,282,366,920,938,463,463,374,607,431,768,211,456, which is normally expressed in scientific notation as about 3.4*10³⁸ addresses.

Types and categories

The type of a IPv6 address is identified by the high-order bits of the address.

Three categories of IP addresses are supported in IPv6:

• Unicast

An identifier for a single interface. A packet sent to a unicast address is delivered to the interface identified by that address. It can be link-local scope, site-local scope, or global scope.

• Multicast

An identifier for a group of interfaces (typically belonging to different nodes). A packet sent to a multicast address is delivered to all interfaces identified by that address.

• Anycast

An identifier for a group of interfaces (typically belonging to different nodes). A packet sent to an anycast address is delivered to the closest member of a group, according to the routing protocols’ measure of distance.

In this article we will create two identical HAST nodes, hast1 and hast2. Both devices will use one NIC connected to a vlan for data synchronization and another NIC will be configured via CARP in order to share the same IP address across the network. The first node will be called “storage1.hast.test”, the second “storage2.hast.test” and they will both listen to a common IP address which we will bind to “storage.hast.test”

HAST binds its resource names according to the machine’s hostname. Therefore, we will use “hast1.freebsd.loc” and “hast2.freebsd.loc”  as the machines’s hostnames so that HAST can operate without complaining.

For starters, lets set up two identical nodes. For this example I have installed FreeBSD 9.0-RELEASE on two deferent instances using a Linux KVM. Both nodes have 512MB of RAM, one SATA drive containing the OS and three SATA drives which will be used to create our shared Raidz1 pool. The final result looks like this:

In order for carp to work we don’t have to compile a new kernel. We can just load it as a module by adding to /boot/loader.conf

 if_carp_load="YES"

Our both nodes are set up, it is time to make some adjustments. First a descent /etc/rc.conf for the first node:

zfs_enable="YES"
 
###Primary Interface##
ifconfig_re0="inet 10.10.10.181  netmask 255.255.255.0"
 
###Secondary Interface for HAST###
ifconfig_re1="inet 192.168.100.100  netmask 255.255.255.0"
 
defaultrouter="10.10.10.1"
sshd_enable="YES"
hostname="hast1.freebsd.loc"
 
##CARP INTERFACE SETUP##
cloned_interfaces="carp0"
ifconfig_carp0="inet 10.10.10.180 netmask 255.255.255.0 vhid 1 pass mypassword advskew 0"
 
hastd_enable=YES

The second node we will also much the first except for the IP addressing:

zfs_enable="YES"
 
###Primary Interface##
ifconfig_re0="inet 10.10.10.182  netmask 255.255.255.0"
 
###Secondary Interface for HAST###
ifconfig_re1="inet 192.168.100.101  netmask 255.255.255.0"
 
defaultrouter="10.10.10.1"
sshd_enable="YES"
hostname="hast2.freebsd.loc"
 
##CARP INTERFACE SETUP##
cloned_interfaces="carp0"
ifconfig_carp0="inet 10.10.10.180 netmask 255.255.255.0 vhid 1 pass mypassword advskew 100"
 
hastd_enable=YES

At this point we have assigned re1 with two IPs for HAST synchronization. We have also assigned two IPs to re0 which in turn we share with a third common IP assigned to carp0.
As a result, re1 is being used for HAST synchronization in a vlan while carp0 which is cloned by re0 used under the same vlan with the rest of our clients.

In order for HAST to function correctly we have to resolve the correct IPs on every node. We don’t want to rely on DNS for this because DNS can fail. Instead we will use /etc/hosts same on every node.

::1			localhost localhost.freebsd.loc
127.0.0.1		localhost localhost.freebsd.loc
192.168.100.100		hast1.freebsd.loc hast1
192.168.100.101		hast2.freebsd.loc hast2
 
10.10.10.181          	storage1.hast.test storage1
10.10.10.182          	storage2.hast.test storage2
10.10.10.180	      	storage.hast.test  storage

Next, we have to create the /etc/hast.conf file. Here we will declare the resources that we want to create. All resources will eventually create devices located under /dev/hast on the primary node. Every resource indicates a physical device specifying a local and remote IP device. The /etc/hast.conf must be exactly the same on every node.

resource disk1 {
        on hast1 {
                local /dev/ad1
                remote hast2
        }
        on  hast2 {
                local /dev/ad1
                remote hast1
        }
}
 
resource disk2 {
        on  hast1 {
                local /dev/ad2
                remote hast2
        }
        on  hast2 {
                local /dev/ad2
                remote hast1
        }
}
 
resource disk3 {
        on  hast1 {
                local /dev/ad3
                remote hast2
        }
        on  hast2 {
                local /dev/ad3
                remote hast1
        }
}

In this example we are sharing three resources, disk1, disk2 and disk3. Each resource indicates a device the local and the remote IP address. With this configuration in place, we are ready to begin setting up out HAST devices.

Lets start hastd on both nodes first:

hast1#/etc/rc.d/hastd start

hast2#/etc/rc.d/hastd start

Now on the primary node we will initialize our resources, create them and finally assign a primary role:

hast1#hastctl role init disk1
hast1#hastctl role init disk2
hast1#hastctl role init disk3
hast1#hastctl create disk1
hast1#hastctl create disk2
hast1#hastctl create disk3
hast1#hastctl role primary disk1
hast1#hastctl role primary disk2
hast1#hastctl role primary disk3

Next, on the secondary node we will initialize our resources, create them and finally assign a secondary role:

hast2#hastctl role init disk1
hast2#hastctl role init disk2
hast2#hastctl role init disk3
hast2#hastctl create disk1
hast2#hastctl create disk2
hast2#hastctl create disk3
hast2#hastctl role secondary disk1
hast2#hastctl role secondary disk2
hast2#hastctl role secondary disk3

There are other ways for creating and assigning roles to each resource. Having repeat this procedure a few times, I saw that this usually always works.

Now check the status on both nodes:

hast1# hastctl status
disk1:
  role: primary
  provname: disk1
  localpath: /dev/ada1
  ...
  remoteaddr: hast2
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...
disk2:
  role: primary
  provname: disk2
  localpath: /dev/ada2
  ...
  remoteaddr: hast2
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...
disk3:
  role: primary
  provname: disk3
  localpath: /dev/ada3
  ...
  remoteaddr: hast2
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...

The first node looks good. Status is complete.

hast2# hastctl status
disk1:
  role: secondary
  provname: disk1
  localpath: /dev/ada1
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...
disk2:
  role: secondary
  provname: disk2
  localpath: /dev/ada2
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...
disk3:
  role: secondary
  provname: disk3
  localpath: /dev/ada3
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  dirty: 0 (0B)
  ...

So does the second. Like I mentioned earlier there are different ways for doing this the first time. You have to look for the word status: complete. If you get a degraded status you can always repeat the procedure.

Now it is time to create our ZFS pool. The primary node should have a /dev/hast directory containing our resources. This directory appears only at the active node.

hast1# zpool create zhast raidz1 /dev/hast/disk1 /dev/hast/disk2 /dev/hast/disk3
hast1# zpool status zhast
 pool: zhast
 state: ONLINE
 scan: none requested
 config:
 
	NAME            STATE     READ WRITE CKSUM
	zhast           ONLINE       0     0     0
	  raidz1-0      ONLINE       0     0     0
	    hast/disk1  ONLINE       0     0     0
	    hast/disk2  ONLINE       0     0     0
	    hast/disk3  ONLINE       0     0     0

We can now use hastctl status on each node to see if everything looks ok. The magic word we are looking for here is: replication: fullsync

At this point both of our nodes should be available for failover. We have storage1 running as primary and sharing a pool called zhast. Our storage2 is currently in a standby mode. If we have set DNS properly we can ssh to storage.hast.test or by using its carp IP to 10.10.10.180.

In order to perform a failover we have to first export our pool from the first node, change the role of each resource to secondary. Then change the role of each resource to primary on the standby node and import the pool. This procedure will be done manually to test if failover really works. But for a real HA solution we will eventually create a script that will take care of this.

First lets export our pool and change our resources role:

hast1# zpool export zhast
hast1# hastctl role secondary disk1
hast1# hastctl role secondary disk2
hast1# hastctl role secondary disk3

Now, lets reverse the procedure on the standby node:

hast2# hastctl role primary disk1
hast2# hastctl role primary disk2
hast2# hastctl role primary disk3
hast2# zpool import zhast

The roles have successfully changed, lets see our pool status:

hast2# zpool status zhast
 pool: zhast
 state: ONLINE
 scan: none requested
 config:
 
	NAME            STATE     READ WRITE CKSUM
	zhast           ONLINE       0     0     0
	  raidz1-0      ONLINE       0     0     0
	    hast/disk1  ONLINE       0     0     0
	    hast/disk2  ONLINE       0     0     0
	    hast/disk3  ONLINE       0     0     0
 
errors: No known data errors

Again, by using hastctl status on each node we can verify that the roles have indeed changed and that the status is complete. This is a sample output from the second node now in charge:

hast2# hastctl status
disk1:
  role: primary
  provname: disk1
  localpath: /dev/ad1
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  ...
disk2:
  role: primary
  provname: disk2
  localpath: /dev/ad2
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  ...
disk3:
  role: primary
  provname: disk3
  localpath: /dev/ad3
  ...
  remoteaddr: hast1
  replication: fullsync
  status: complete
  ...

It is now time to automate this procedure. When do we want our servers to automatically failover?
One reason would be if the primary node is not responding to the external network thus not being able to serve its clients. Using a devd event we can catch a carp interface going up or down and a state change.

Add the following lines to /etc/devd.conf on both nodes:

notify 30 {
	match "system" "IFNET";
	match "subsystem" "carp0";
	match "type" "LINK_UP";
	action "/usr/local/bin/failover master";
};
 
notify 30 {
	match "system" "IFNET";
	match "subsystem" "carp0";
	match "type" "LINK_DOWN";
	action "/usr/local/bin/failover slave";
};

And now lets create the failover script which will be responsible for doing automatically what we did before manually:

#!/bin/sh
 
# Original script by Freddie Cash 
# Modified by Michael W. Lucas 
# and Viktor Petersson 
# Modified by George Kontostanos 
 
# The names of the HAST resources, as listed in /etc/hast.conf
resources="disk1 disk2 disk3"
 
# delay in mounting HAST resource after becoming master
# make your best guess
delay=3
 
# logging
log="local0.debug"
name="failover"
pool="zhast"
 
# end of user configurable stuff
 
case "$1" in
	master)
		logger -p $log -t $name "Switching to primary provider for ${resources}."
		sleep ${delay}
 
		# Wait for any "hastd secondary" processes to stop
		for disk in ${resources}; do
			while $( pgrep -lf "hastd: ${disk} \(secondary\)" > /dev/null 2>&1 ); do
				sleep 1
			done
 
			# Switch role for each disk
			hastctl role primary ${disk}
			if [ $? -ne 0 ]; then
				logger -p $log -t $name "Unable to change role to primary for resource ${disk}."
				exit 1
			fi
		done
 
		# Wait for the /dev/hast/* devices to appear
		for disk in ${resources}; do
			for I in $( jot 60 ); do
				[ -c "/dev/hast/${disk}" ] && break
				sleep 0.5
			done
 
			if [ ! -c "/dev/hast/${disk}" ]; then
				logger -p $log -t $name "GEOM provider /dev/hast/${disk} did not appear."
				exit 1
			fi
		done
 
		logger -p $log -t $name "Role for HAST resources ${resources} switched to primary."
 
		logger -p $log -t $name "Importing Pool"
		# Import ZFS pool. Do it forcibly as it remembers hostid of
                # the other cluster node.
                out=`zpool import -f "${pool}" 2>&1`
                if [ $? -ne 0 ]; then
                    logger -p local0.error -t hast "ZFS pool import for resource ${resource} failed: ${out}."
                    exit 1
                fi
                logger -p local0.debug -t hast "ZFS pool for resource ${resource} imported."
 
	;;
 
	slave)
		logger -p $log -t $name "Switching to secondary provider for ${resources}."
 
		# Switch roles for the HAST resources
		zpool list | egrep -q "^${pool} "
        	if [ $? -eq 0 ]; then
                	# Forcibly export file pool.
                	out=`zpool export -f "${pool}" 2>&1`
               		 if [ $? -ne 0 ]; then
                        	logger -p local0.error -t hast "Unable to export pool for resource ${resource}: ${out}."
                        	exit 1
                	 fi
                	logger -p local0.debug -t hast "ZFS pool for resource ${resource} exported."
        	fi
		for disk in ${resources}; do
			sleep $delay
			hastctl role secondary ${disk} 2>&1
			if [ $? -ne 0 ]; then
				logger -p $log -t $name "Unable to switch role to secondary for resource ${disk}."
				exit 1
			fi
			logger -p $log -t $name "Role switched to secondary for resource ${disk}."
		done
	;;
esac

Let’s try it and see if it works. Log into both the currently active and standby node. Make sure that you are on the active by issuing a hastctl status command. Then force a failover by bringing the interface which is associated with carp0 downL

hast1# ifconfig er0 down

Watch at the generated messages:

hast1# tail -f /var/log/debug.log
 
Feb  6 15:01:41 hast1 failover: Switching to secondary provider for disk1 disk2 disk3.
Feb  6 15:01:49 hast1 hast: ZFS pool for resource  exported.
Feb  6 15:01:52 hast1 failover: Role switched to secondary for resource disk1.
Feb  6 15:01:55 hast1 failover: Role switched to secondary for resource disk2.
Feb  6 15:01:58 hast1 failover: Role switched to secondary for resource disk3.

hast2# tail -f /var/log/debug.log
 
Feb  6 15:02:15 hast2 failover: Switching to primary provider for disk1 disk2 disk3.
Feb  6 15:02:19 hast2 failover: Role for HAST resources disk1 disk2 disk3 switched to primary.
Feb  6 15:02:19 hast2 failover: Importing Pool
Feb  6 15:02:52 hast2 hast: ZFS pool for resource  imported.

Voila! The failover worked like a charm and now hast2 has assumed the primary role.

Further considerations:

What we did today is a basic setup of two nodes sharing a raidz1 pool with automatic role failover in case of a failure that would result in a loss of a carp interface.
Obviously, a similar devd event would be generated in case we loose a HAST replication interface. This is something that needs to be addressed similarly since losing that interface will leave us with no synchronization at all.
Going further, we would have to add scripts that will bring up and down services during a failover.

You can use this as a reference guide for a single or mirror installation.

(1) Boot from a FreeBSD9 installation DVD or memstick and choose “Live CD”.

(2) Create the necessary partitions on the disk(s) and add ZFS aware boot code.

a) For a single disk installation.

gpart create -s gpt ada0
gpart add -b 34 -s 94 -t freebsd-boot ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

b) Repeat the procedure for the second drive if you want a mirror installation.

gpart create -s gpt ada1
gpart add -b 34 -s 94 -t freebsd-boot ada1
gpart add -t freebsd-zfs -l disk1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

(3) Align the Disks for 4K and create the pool.

a) For a single disk installation.

gnop create -S 4096 /dev/gpt/disk0
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.nop
zpool export zroot
gnop destroy /dev/gpt/disk0.nop
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot

b) For a mirror installation.

gnop create -S 4096 /dev/gpt/disk0
gnop create -S 4096 /dev/gpt/disk1
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot mirror /dev/gpt/disk0.nop /dev/gpt/disk1.nop
zpool export zroot
gnop destroy /dev/gpt/disk0.nop
gnop destroy /dev/gpt/disk1.nop
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot

(4) Set the bootfs property and checksums.

zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot

(5) Create appropriate filesystems (feel free to improvise!).

zfs create zroot/usr
zfs create zroot/usr/home
zfs create zroot/var
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp

(6) Add swap space and disable checksums. In this case I add 4GB of swap.

zfs create -V 4G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set checksum=off zroot/swap

(7) Create a symlink to /home and fix some permissions.

chmod 1777 /mnt/tmp
cd /mnt ; ln -s usr/home home
chmod 1777 /mnt/var/tmp

(8) Instal FreeBSD.

sh
cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

(9) Copy zpool.cache (very important!!!)

cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache

(10) Create the rc.conf, loader.conf and an empty fstab (otherwise the system will complain).

echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
touch /mnt/etc/fstab

Reboot, adjust time zone info, add a password for root, add a user and enjoy!!!
 

 

Today, we will see how we can use PF to effectively secure a FreeBSD host on the Internet.
Before we even start talking about PF, it is essential to point out a few things about network firewalls.

• Network firewalls as opposed to packet filtering devices use statefull inspection in order to decide if a packet is allowed to pass or not.
• Statefull inspection was inspired by the concept of state and the 3-way handshake which we find only on TCP type base connections.
• Statefull inspection has progressed a lot over the time, making it somehow possible to maintain it on stateless protocols such us UDP or ICMP.

Network firewalls work mainly on layers 3 and 4 of the OSI model. Therefore, application inspection is not possible since it requires the ability to inspect at the layer 7.

General DO & DON’T

• Log everything! (except Internet noise)
• Be similarly restrictive to inbound and outbound traffic!
• Process your most used rules first, respect your CPU and memory!
• Don’t be tempted into using exotic rules that you read somewhere to prevent port scans!

Let’s start by building our first policy for a webserver:

### macro definitions
ext_if = "em0"
dns_server = "123.123.123.123"
webservices = "{80, 443, 22}"
icmp_types = "echoreq"
netbios_tcp = "{445, 137, 138, 139}"
netbios_udp = "{445, 137, 138, 139}"
tcp_out = "{5999, 80, 21}"

First we have define our macros. Macros are variables in a sense, holding information regarding our interfaces and tcp / udp ports. It is always a good idea to group port numbers since it can greatly reduce our rules

### all incoming traffic on external interface is normalized and fragmented
### packets are reassembled.
scrub in on $ext_if all fragment reassemble
 
### exercise antispoofing on the external interface, but add the local
### loopback interface as an exception, to prevent services utilizing the
### local loop from being blocked accidentally.
set skip on lo0
antispoof for $ext_if inet

Next we tell the firewall to reassemble all fragmented packets, skip any packet filtering rules for the loopback interface and finally to check for spoofed IP addresses on the external interface.

### get rid quick of Internet noise like microsoft netbios service.
### This accounts to 80% of dropped traffic. We don't need to log this also
block in quick on $ext_if proto tcp from any to any port $netbios_tcp
block in quick on $ext_if proto udp from any to any port $netbios_udp

This part you don’t usually find in PF rules but I think that it is worth having it.
If you look at the logs on an Internet faced firewall, you will see that most of the dropped traffic is netbios. These are usually Zombie hosts infected by some nasty old windows virus.
By blocking them first without logging this traffic, we relax both our firewall from having to read all the rules before it drops this. We also maintain cleaner logs.

###clean up rule
block log all

Finally, we tell our firewall to block any traffic, inbound and outbound, that doesn’t much a rule. We also like to log everything.

### set a rule that allows inbound traffic with synproxy handshaking.
pass in quick log on $ext_if proto tcp from any to any port $webservices flags S/SA synproxy state
pass in quick log inet proto icmp all icmp-type $icmp_types keep-state

Like I mentioned earlier, the most used rules should come first. Since this is a web server, we allow http, https and ssh incoming traffic. We also like to be able to ping our server. Notice that I am using the “synproxy state” keyword. That way I am instructing the firewall to proxy the 3-way tcp handshake, keeping syn flood attacks away.

### keep state on any outbound tcp, udp or icmp traffic. modulate the isn of
### outgoing packets. (initial sequence number) broken operating systems
### sometimes don't randomize this number, making it guessable.
pass out quick log on $ext_if proto tcp from any to any port $tcp_out modulate state
pass out quick log on $ext_if proto udp from any to $dns_server port domain modulate state
pass out quick log inet proto icmp all icmp-type $icmp_types keep-state

Finally, I allow my webserver outbound access for http, ftp and cvs. This could and should become stricter and allow this type of communication only to certain destinations.

This is just an example of a host based firewall. A real firewall would have at least two interfaces and would probably perform NAT. The point is that an effective policy is a policy that will not create extra burden, will be clean and easy to read and maintain.

Always start by processing your most used rules first. Group ports and servers and log all interesting traffic.

There are two prerequisites for this procedure to work. First, the machine that is receiving the snapshots must be running the same or higher ZFS version from the machine that is being backed up. Second, we must either have root access to the receiving machine or an account that has been delegated with create, receive ZFS properties.

First lets prepare both host and target machines. We need to create and send our first full snapshot from our host to a ZFS folder on the target.

The host:

core2duo# zfs snapshot -r zroot/usr/src@-2012-01-10
core2duo# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot/usr/src 349M 64.4G 349M /usr/src
zroot/usr/src@-2012-01-10 0 - 349M -
zroot/usr/src/mytest 31K 64.4G 31K /usr/src/mytest
zroot/usr/src/mytest@-2012-01-10 0 - 31K -

The target:

hp# zfs create tank/test
hp# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank/test 40.0K 747G 40.0K /tank/test

Now, lets send our first full snapshot:

core2duo# zfs send -R zroot/usr/src@-2012-01-10 | ssh root@hp zfs receive -Fduv tank/test

Password:
receiving full stream of zroot/usr/src@-2012-01-10 into tank/test/usr/src@-2012-01-10
received 732MB stream in 72 seconds (10.2MB/sec)
receiving full stream of zroot/usr/src/mytest@-2012-01-10 into tank/test/usr/src/mytest@-2012-01-10
received 47.4KB stream in 1 seconds (47.4KB/sec)

Now, lets create a new snapshot and send them both incremental.

core2duo# zfs snapshot -r zroot/usr/src@-2012-01-11
core2duo# zfs send -R -i zroot/usr/src@-2012-01-10 zroot/usr/src@-2012-01-11 | ssh root@hp zfs receive -Fduv tank/test

receiving full stream of zroot/usr/src@-2012-01-10 into tank/test/usr/src@-2012-01-10
received 732MB stream in 72 seconds (10.2MB/sec)
receiving full stream of zroot/usr/src/mytest@-2012-01-10 into tank/test/usr/src/mytest@-2012-01-10
received 47.4KB stream in 1 seconds (47.4KB/sec)

Notice that I am using some special switches in my send/receive commands.
When sending, -R allows me to send the snapshots, their children and their properties.
When receiving, -F forces a rollback to the most recent snapshot, -d maintains my naming scheme and -u makes sure that the associated file systems do not get mounted.

Lets now see what we have done so far, first the host:

core2duo# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot/usr/src 349M 64.4G 349M /usr/src
zroot/usr/src@-2012-01-11 0 - 349M -
zroot/usr/src/mytest 31K 64.4G 31K /usr/src/mytest
zroot/usr/src/mytest@-2012-01-11 0 - 31K -

And the target:

hp# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank/test 364M 747G 41.3K /tank/test
tank/test/usr 364M 747G 40.0K /tank/test/usr
tank/test/usr/src 364M 747G 364M /tank/test/usr/src
tank/test/usr/src@-2012-01-10 1.33K - 364M -
tank/test/usr/src@-2012-01-11 0 - 364M -
tank/test/usr/src/mytest 65.3K 747G 40.0K /tank/test/usr/src/mytest
tank/test/usr/src/mytest@-2012-01-10 25.3K - 40.0K -
tank/test/usr/src/mytest@-2012-01-11 0 - 40.0K -

Looks like it is working. The only thing left is to somehow automate the procedure. Let’s create a script that will create a snapshot which we will call it today, check if there was a snapshot yesterday and send them incrementally to a remote host.

#!/bin/sh
 
pool="zroot/usr/src"
destination="tank/test"
host="10.10.10.4"
 
today=`date +"$type-%Y-%m-%d"`
yesterday=`date -v -1d +"$type-%Y-%m-%d"`
 
# create today snapshot
snapshot_today="$pool@$today"
# look for a snapshot with this name
if zfs list -H -o name -t snapshot | sort | grep "$snapshot_today$" > /dev/null; then
echo " snapshot, $snapshot_today, already exists"
exit 1
else
echo " taking todays snapshot, $snapshot_today"
zfs snapshot -r $snapshot_today
fi
 
# look for yesterday snapshot
snapshot_yesterday="$pool@$yesterday"
if zfs list -H -o name -t snapshot | sort | grep "$snapshot_yesterday$" > /dev/null; then
echo " yesterday snapshot, $snapshot_yesterday, exists lets proceed with backup"
 
zfs send -R -i $snapshot_yesterday $snapshot_today | ssh root@$host zfs receive -Fduv $destination
 
echo " backup complete destroying yesterday snapshot"
zfs destroy -r $snapshot_yesterday
exit 0
else
echo " missing yesterday snapshot aborting, $snapshot_yesterday"
exit 1
fi

pool is the ZFS pool we want to backup
destination is the destination pool that will receive the backup
host is our backup host!

With a cronjob in place this would be a nice candidate for a daily ZFS incremental backup policy. Just make sure you use public/private key for ssh authentication.
 

 

In this article I will add a few steps necessary when upgrading FreeBSD trough a major release. A good example is going from FreeBSD 8.2-RELEASE to FreeBSD 9.0-RELEASE. In reality the steps are more or less the same.

First and for most, make sure that your ports are up to date. If you are using portupgrade like myself do a:

 portsnap fetch update && portversion -v | grep "<" 

This will so you which ports if any need to be updated. Read /usr/ports/UPDATING for special instructions on ports that might affect you and upgrade all your ports. Follow the instructions you read and after you finish if you are using portupgrade this will do the job:

 portupgrade -arR

At this point you have your sources synchronized and your ports updated. Follow the instructions you read in the previous article with the exception of  running make installworld and mergemaster while in single user mode.

After your system boots it is time to rebuild all of your ports. If you are using portupgrade you will have to start by rubby:

cd /usr/ports/lang/ruby18/
make deinstall && make install clean

cd /usr/ports/ports-mgmt/portupgrade/
make deinstall && make install clean

Now that portupgrade is done continue with the rest:

 portupgrade -af

This should be enough. Reboot your system and you will have a perfect functional upgraded FreeBSD.
 

 

Today, I see that many people have started using the freebsd-update utility. I think it is good that a binary update option is available. But to be honest I have only used it once, out of curiosity mostly. Don’t get me wrong I am not the type of person who believes in hard core solutions. I just feel that going source gives you more control. Besides, a relatively descent box will not need more than an hour for the whole procedure of building world && kernel. Also, freebsd-update can not be used yet for tracking stable.

To all of you who want to go source but are afraid to,  keep reading.

Synchronize Source 

The first thing you have to do is to synchronize your sources with the release you want to follow. The cvsup utility was used for many years for this job. While it is still being described in the handbook as a method of updating your sources it is considered obsolete and has been replaced by csup which is integrated in to world. This means that you don’t need to install any extra packages in order to synchronize your sources.

You need however to use a sup file for the version you are following. In the directory below you will find many examples:

# cd /usr/share/examples/cvsup/ && ls
README doc-supfile ports-supfile refuse.README standard-supfilecvs-supfile gnats-supfile refuse stable-supfile www-supfile

From all these files only 2 are the ones that we are interested in.

stable-supfile
standard-supfile

The first one is used to track stable while the second one is used for release. So, grab a copy of standard and copy it to a directory, I usually keep my sup files under /root

Lets spend a few minutes to examine the important lines here.

# Defaults that apply to all the collections
#
# IMPORTANT: Change the next line to use one of the CVSup mirror sites
# listed at http://www.freebsd.org/doc/handbook/cvsup.html#CVSUP-MIRRORS.
*default host=CHANGE_THIS.FreeBSD.org

This specifies from where to get the source tree.

*default host=cvsup.freebsd.org

Is usually a good choice although you might consider using a mirror closer to you.

# The following line is for 8-stable. If you want 7-stable, 6-stable,
# 5-stable, 4-stable, 3-stable, or 2.2-stable, change to "RELENG_7",
# "RELENG_6", "RELENG_5", "RELENG_4", "RELENG_3", or "RELENG_2_2"
# respectively.
*default release=cvs tag=RELENG_9_0
*default delete use-rel-suffix

This is the most important part of the file. It specifies which source tree you wish to follow.
If you are tracking 8-STABLE or 9-STABLE it should read like this:

*default release=cvs tag=RELENG_8

*default release=cvs tag=RELENG_9

If you want to track a RELEASE it should read like this:

*default release=cvs tag=RELENG_8_0

*default release=cvs tag=RELENG_9_0

The first specifies 8.0-RELEASE while the second specifies 9.0-RELEASE respectably.

So, go ahead and adjust your sup file with the mirror you want to grab the source from and the version you want to follow. A typical supfile for the upcoming 8.3-RELEASE will look like this:

# Defaults that apply to all the collections
#
# IMPORTANT: Change the next line to use one of the CVSup mirror sites
# listed at http://www.freebsd.org/doc/handbook/cvsup.html#CVSUP-MIRRORS.
*default host=cvsup.FreeBSD.org
*default base=/var/db
*default prefix=/usr
# The following line is for 8-stable. If you want 7-stable, 6-stable,
# 5-stable, 4-stable, 3-stable, or 2.2-stable, change to "RELENG_7",
# "RELENG_6", "RELENG_5", "RELENG_4", "RELENG_3", or "RELENG_2_2"
# respectively.
*default release=cvs tag=RELENG_8_3
*default delete use-rel-suffix

Now that your supfile is complete, rename it into something memorable e.g release8.3 and fire csup with it.

 csup /root/release8.3

After the process is finished you will have a complete functional source tree that you can use it to build FreeBSD and update your distribution.

UPDATE FREEBSD

The whole procedure is divided into 4 different stages. Build world and kernel, Install kernel, Install world, mergemaster. After a while you will discover that those 4 different stages can be merged in 2. Build world and kernel, install kernel and world and mergemaster.

1) Build everything

cd /usr/src
make -j8 buildworld && make -j4 buildkernel

2) Install the new kernel

make install kernel
shutdown -r now

3) Install the new world 

cd /usr/src
make install world

4) Run mergemaster

mergemaster

That’s it, reboot once more and your system will be updated.

As I mentioned earlier there is a faster way of doing this without so many reboots.

cd /usr/src && make -j6 buildworld && make -j4 buildkernel
make installkernel
make istallworld
mergemaster
shutdown -r now

The above procedure is highly unorthodox but if you track STABLE regularly you will see that there are not so many changes in world and kernel.

Troubleshooting

What to do if the world or kernel refuses to compile ?

If you track stable then there is a slight chance that your source tree is partially synchronized  or just happened to hit a bug in the road.

a) Use csup again, there is a chance that you have missed an update that perhaps is stalling the procedure.
b) Check the stable mailing list for any know problems regarding the version you are following.
c) Clean your source tree like this:

chflags -R noschg /usr/obj/usr
rm -rf /usr/obj/usr
cd /usr/src
make cleandir
make cleandir

If you are tracking RELEASE then things should run more smoothly.

a) Make sure that you are following the correct RELEASE by checking your supfile.
b) Repeat the above procedure for cleaning up your source tree.
c) Remove any tuning you might have in make.conf

My kernel doesn’t boot !

No problem, you can always boot your previous kernel. FreeBSD automatically saves your kernel in /boot/kernel.old The procedure is fairly easily, just choose command line from the boot menu and:

>unload
ok
>load /boot/kernel.old/kernel.ko
>ok
>boot

Notes:
In the handbook it is recommended to update the world files while in single user mode. During my involvement with FreeBSD, I saw that there is really no special need for this.

Next, see how to perform a major UPGRADE

In this guide I will demonstrate how you can install a fully functional full ZFS FreeBSD9 using a GPT scheme. We will also use ZFS for SWAP

You can use this as a reference guide for a single or mirror installation.

(1) Boot from a FreeBSD9 installation DVD or memstick and choose “Live CD”.

(2) Create the necessary partitions on the disk(s) and add ZFS aware boot code.

a) For a single disk installation.

gpart create -s gpt ada0
gpart add -b 34 -s 94 -t freebsd-boot ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

b) Repeat the procedure for the second drive if you want a mirror installation.

gpart create -s gpt ada1
gpart add -b 34 -s 94 -t freebsd-boot ada1
gpart add -t freebsd-zfs -l disk1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

(3) Create the pool.(ignore any warnings regarding mounting)

a) For a single disk installation.

zpool create zroot /dev/gpt/disk0

b) For a mirror installation.

zpool create zroot mirror /dev/gpt/disk0 /dev/gpt/disk1

(4) Set bootfs property, checksums and mountpoints.

zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot
zfs set mountpoint=/mnt zroot

(5)) At this point export and import the pool while preserving zroot.cache in /var/tmp.

zpool export zroot
zpool import -o cachefile=/var/tmp/zpool.cache zroot

(6) Create appropriate filesystems (feel free to improvise!).

zfs create zroot/usr
zfs create zroot/usr/home
zfs create zroot/var
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp

(7) Add swap space and disable checksums. In this case I add 4GB of swap.

zfs create -V 4G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set checksum=off zroot/swap

(8) Create a symlink to /home and fix some permissions.

chmod 1777 /mnt/tmp
cd /mnt ; ln -s usr/home home
chmod 1777 /mnt/var/tmp

(9) Instal FreeBSD.

sh
cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

(10) Copy zpool.cache (very important!!!)

cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache

(11) Create the rc.conf, loader.conf and an empty fstab (otherwise the system will complain).

echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
touch /mnt/etc/fstab

(12) Unmount everything and fix mountpoints for system boot.

zfs set readonly=on zroot/var/empty
zfs umount -af
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var

Reboot, adjust time zone info, add a password for root, add a user and enjoy!!!

If you are looking for a 4K optimized guide see my next guide!