This article was written by Bob Feldbauer.
Continuing from yesterday's storage discussion on DRBD, let's introduce another option in the storage space.
When your environment has grown beyond direct attached storage (internal drives, or an external drive array) and basic network attached storage (NAS), the next step is generally to consider implementing a storage area network (SAN); however, the cost and complexity of existing Fibre Channel and iSCSI SAN solutions can be daunting. Fortunately, ATA over Ethernet (AoE) can often be used as a simpler, lower-cost alternative.
Before diving into AoE, you should be aware of its limitations. The major idiosyncrasy of AoE is that it does not use TCP/IP; therefore, it is not routable. Also, compared to iSCSI, it lacks encryption and user-level access control. AoE truly shines when you simply need storage over a local network, and can limit access by control over physical ports on a switch, VLANs, etc.
AoE is supported on a wide variety of operating systems, including Linux, Windows, FreebSD, VMWare ESX, Solaris, Mac OS X, and OpenBSD. Like Fibre Channel, iSCSI, and other storage protocols, the AoE protocol implements an initiator-target architecture - the initiator sends commands, and the target receives them. On Linux, various pieces of software are used to provide initiator and target functionality. Four independently developed AoE targets exist for Linux: qaoed, ggaoed, kvblade, and vblade. Since vblade is part of aoetools, it generally seems to have the most active development community; therefore, we'll use that for our example setup. Depending on the version of AoE you use, it can be used with Linux kernels 2.4.x or 2.6.x; however, it is best used with kernel 2.6.14 or higher and newer versions of aoetools.
For our example configuration, we'll be setting up AoE on Debian Linux (Lenny), and using LVM2. The AoE module should be included if you're using a standard Debian kernel, but let's check using:
grep ATA_OVER /boot/config-`uname -r`Which should return:
CONFIG_ATA_OVER_ETH=m
That means AoE is supported through a kernel module. If you aren't already using AoE, the module is probably not loaded yet. Let's load it, and add it to /etc/modules so it is automatically loaded at when the system is started in the future:
modprobe aoe echo "aoe" >> /etc/modulesOn Debian, we'll use apt to install the remaining necessary components:
apt-get update apt-get install lvm2 vblade vblade-persist aoetools
Note that aoetools includes the following tools:
| aoecfg | manipulate AoE configuration strings |
| aoe-discover | trigger discovery of AoE devices |
| aoe-flush | flush down devices out of the AoE driver |
| aoe-interfaces | restrict network interfaces used for AoE |
| aoe-mkdevs | create character and block device files |
| aoe-mkshelf | create block device files for one shelf address |
| aoeping | simple userland communication with AoE devices |
| aoe-revalidate | revalidate the disk size of an AoE device |
| aoe-stat | print status information for AoE devices |
| aoe-version | print AoE-related software version information |
| coraid-update | upload an update file to a Coraid appliance |
For our example, we'll use LVM2 and AoE to allocate space and make it available over the network, using two disks. For a recent project, I used 13 drives on a hardware RAID controller and divided them into two RAID6 arrays of 6 (4TB) and 7 (5TB) drives.
Assuming the disks are the second and third disks on a Linux system, configure LVM2 to recognize the physical volumes:
pvcreate /dev/sdb pvcreate /dev/sdc
Create two LVM2 volume groups on the physical volumes - for our example, let's call them content and backups:
vgcreate content /dev/sdb vgcreate backups /dev/sdc
Then create two 1TB LVM2 logical partitions (/dev/content/server1, /dev/backups/server1):
lvcreate -L 1TB -n server1 content lvcreate -L 1TB -n server1 backups
Although there are many different types of filesystems available under Linux, for familiarity and to avoid unnecessary complexity in our example, we'll use ext3:
mkfs.ext3 /dev/content/server1 mkfs.ext3 /dev/backups/server1
Now that we have our drives configured with LVM2 and formatted with a usable filesystem, we can setup the AoE target using vblade-persist:
vblade-persist setup 0 1 eth0 /dev/content/server1 vblade-persist setup 0 2 eth0 /dev/backups/server1 vblade-persist start 0 1 vblade-persist start 0 2
To mount our newly created AoE devices on a remote server, run the following on a second server:
modprobe aoe apt-get install aoetools aoe-discover aoe-stat # should show the available AoE exports mkdir /mountpoint # replace e0.1 with the appropriate device from aoe-stat mount /dev/etherd/e0.1 /mountpoint
Your options in the storage are many. This introduction should give you the necessary tools to decide if AoE is the storage solution that meets your requirements.
- ATA over Ethernet: Putting Hard Drives on the LAN at Linux Journal
- Using AoE on Debian Lenny on HowtoForge
- Debian NAS using AoE by the author
- Using AoE on Fedora 10 on HowtoForge
- ATA over Ethernet on Wikipedia
Posts
