Gentoo Workstation Image in Xen

Rationale
A persistent annoyance the lab has dealt with is that of maintaining two (or more) separate user-visible systems -- the workstation image plus a remote access, LTSP, or high performance server -- and keeping them reasonably consistent and updated. In the 2005-2006 year, the remote server (which ran on robustus most of the time) was plagued by a deficiency of desired software, especially libraries for building graphical programs. In the 2006-2007 year, LTSP was used to give a second life to the lab's aging desktops (under half of the machines). Unfortunately this resulted in two competing environments which tended to break if users switched between them.

The lab's currently ongoing move to Xen-based virtualization of services has opened the possibility of a clean, low-maintenance solution to this problem. By encapsulating a copy of the workstation image in a virtual machine (DomU in Xen speak), the workstation system can run on a server with minimal modification, while still allowing the server to be used for a multitude of other services.

Installation

 * Read the "Tweaks" section to find out what special things you may need to do to the system being put in the DomU
 * Create a new partition (or logical volume, etc.) on the Xen host (or shared storage)
 * Mount it somewhere in the Dom0 (xen host) filesystem
 * Use rsync to copy the image from the image server
 * Configure the hostname -- this can't happen automatically as it's not booted from dhcp
 * Manually run the post-imaging scripts -- be mindful of ssh host keys and Kerberos keytabs
 * Unmount, find a kernel, set up the Xen config file (/etc/xen/ .cfg)
 * Start the new VM and watch it boot via "xm create -c"
 * A few further tweaks will be necessary, especially with kernel-related stuff. Those necessary for a Gentoo image are described below.
 * The local X server will of course be unable to start, but gdm can still accept XDMCP connections. Ideally gdm would be configured not to start a local X server, but if not, its XKeepsCrashing script will kick in and everything will be fine.

Glibc
Xen uses segmentation on 32-bit x86 systems for the purpose of protecting the memory in use by the hypervisor. Xen itself runs in ring 0, guest OSes run in ring 1, and guest user applications run in ring 3. Paging only says that rings 0-2 are the supervisor rings, which does not provide protection for Xen against the guest OSes, hence the need for segmentation. See and  for more details.

As glibc uses wrap-around segments, this is a very expensive operation under xen. To make glibc not cause performance issues, the nosegneg version of glibc should be used. Under Debian, this is accomplished by installing the libc6-xen package. In Gentoo, add "-mno-tls-direct-seg-refs" to the CFLAGS in /etc/make.conf, and rebuild glibc. See the Gentoo wiki page for more information.

Other
Most of the following will depend on what kernel the VM is booted with. It will require one compiled with the Xen patch set; Portage currently has xen-sources up to 2.6.16. But our Xen hosts run Debian, it's easier to use one of Debian's prebuilt xen kernels, though this leads to a few glitches with Gentoo.


 * Debian's initrd apparently mounts /proc and /sys. Gentoo is hardcoded to do this itself (during "rc sysinit", as called from /etc/inittab). My quick solution was to comment out the relevant lines in /sbin/rc (which is a bash script). I've submitted a gentoo bug about this.
 * The next difficulty comes in getting the OpenAFS kernel module rebuild for the new kernel. Since we're using a Debian kernel, the best way is to copy /usr/src/linux-headers-blah from a Debian VM with the kernel-headers-blah package installed. Of course update the /usr/src/linux symlink. This is almost enough to coax the openafs-kernel ebuild into working.
 * One more glitch -- Debian's kernel header scripts somewhere have "gcc-4.1" hardcoded, but that doesn't currently exist on gentoo (August 2007 -- there's either gcc, or gcc-4.1.2). The solution is to change the offending line in /usr/src/linux-headers-blah-xen-686/.kernelvariables
 * Once the AFS module is build, you will want to add /lib/modules to the exclude list for updating the VM image, as well as whatever other tweaks you've made.


 * A few more minor changes I made to the image, all relevant files excluded from imaging:
 * /etc/conf.d/net: setup for static IP
 * /etc/issue: remove the pretty TJ logo
 * /etc/inittab: remove the fun stuff running on tty 10-11
 * removed from runlevels: alsasound, gpm, ntp-client, ntpd