HA-iSCSI

HA-iSCSI is a high-availability storage system which serves most of the Xen Virtual Machines. Royal and Magellanic are the lab's storage servers.

Hardware
Royal and Magellanic each have a QLogic ISP2312-based 2Gb Fibre Channel HBA in them. These HBAs are then connected via fiber-optic links to a pair of Sun StorEdge 3510 FC JBOD Arrays. These connections are made by daisy-chaining so that in the event of a storage array failure, one of the servers will still be able to reach the working array.

Each array has 12x72GB FC SCSI Drives. A single RAID-Z2 zpool was created on each array; these are named Fryingpan (array ID 0) and Dutchoven (array ID 1). Individual ZVOLs are then created in these zpools for exporting as VM partitions. In addition, space is reserved on Fryingpan for the vminfo NFS share that is used to store the VM configurations and kernels. As of November 2011, Fryingpan is used for storing production VMs while Dutchoven is used for testing purposes and as a warm backup should Fryingpan catastrophically fail.

Software
iSCSI management is done using the COMSTAR iSCSI target built into NexentaOS.

High availability management is done using the SimpleHA package of scripts from the Rochester Center for Brain Imaging. The scripts are located in /opt/SimpleHA/ on both Royal and Magellanic. This system uses a primary heartbeat over TCP/IP on the SAN VLAN with a secondary heartbeat over a null serial cable connecting the two servers. A tie-breaking quorum check is made by pinging the 4506. In the event the slave node decides that it needs to take over operations, it will first attempt to kill the active primary node via an iLO power reset and will only bring up services if it can confirm the reset was successful.

Monitoring Cluster Status
On either node, cat /opt/SimpleHA/status.txt, possible states are MASTER, SLAVE, and INIT (the node is in the process of becoming MASTER).