Gentoo Server Install

Summary
This article will cover the basic process used to install Gentoo Linux onto a CSL 64bit Server System using LVM for disk space management and 802.3ad link aggregation for network redundancy.

Backups, Backups, Backups
Make sure that you have a current backup of all data that is stored on the server. In particular, make sure you have a copy of the SSH keys (/etc/ssh/ssh_host_*) as losing these is BAD(R). The only reason for skipping this step is when you are setting up a brand new server.

Necessary Information
You will also need access to the following information at some point during the install (the values below are what the rest of this guide will be using as example values, be sure to substitute your own in): * Processor Architecture: x86_64/amd64/64 bit * Hostname: fiordland * Domain: csl.tjhsst.edu * FQDN: fiordland.csl.tjhsst.edu * Server VLAN: 1600 * IPv4 Address/Netmask: 198.38.17.42/23 * IPv4 Gateway: 198.38.17.254 * IPv6 Address Assignment: Stateless Address Autoconfiguration * IPv6 Gateway Assignment: Stateless Address Autoconfiguration * DNS Server IPs: 198.38.16.40, 198.38.16.41, and 151.188.14.2

Network Hardware Configuration
If the server is currently setup to use 802.3ad bonding or 802.1Q VLAN tagging, you will need to disable these temporarily.

First we need to find out which switchports the server is connected to. On core0 run: TJHSST-4500MSL#sh run | inc Fiordland description Port8 Fiordland-1,2 description Gi2/3 Fiordland-1 description Gi2/4 Fiordland-2 description Fa5/19 Fiordland-ilo

Then we look at the configuration of the individual switchports.

TJHSST-4500MSL#sh run inter Gi2/3 Building configuration... Current configuration : 305 bytes ! interface GigabitEthernet2/3 description Gi2/3 Fiordland-1 switchport access vlan 1600 switchport trunk encapsulation dot1q switchport trunk allowed vlan 16,1600 switchport mode trunk ! <--This line indicates that vlan tagging is enabled logging event link-status channel-protocol lacp channel-group 8 mode active ! <--This line indicates that bonding is enabled spanning-tree portfast trunk end

Disable bonding and VLAN tagging on at least one interface:

TJHSST-4500MSL#conf t Enter configuration commands, one per line. End with CNTL/Z. TJHSST-4500MSL(config)#inter Gi2/3 TJHSST-4500MSL(config-if)#no channel-group 8 mode active TJHSST-4500MSL(config-if)#switchport mode access TJHSST-4500MSL(config-if)#switchport access vlan 1600 ! <--replace with server's VLAN TJHSST-4500MSL(config-if)#spanning-tree portfast %Warning: portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc... to this interface when portfast is enabled, can cause temporary bridging loops. Use with CAUTION %Portfast has been configured on GigabitEthernet2/3 but will only have effect when the interface is in a non-trunking mode. TJHSST-4500MSL(config-if)#exit TJHSST-4500MSL(config)#exit TJHSST-4500MSL#wr mem

The switchport configuration should now look something like this:

TJHSST-4500MSL#sh run inter Gi2/3 Building configuration... Current configuration : 271 bytes ! interface GigabitEthernet2/3 description Gi2/3 Fiordland-1 switchport access vlan 1600 switchport trunk encapsulation dot1q switchport trunk allowed vlan 16,1600 switchport mode access logging event link-status channel-protocol lacp spanning-tree portfast end

RAID Controller Configuration
If the system you are installing has a hardware RAID controller and you wish to make any changes to the array configuration, do that now. You will need to reboot the server to access the controller's BIOS (F8 on the HP Servers). From the controller's BIOS, you can change the RAID level and number of drives in each array.

Booting the Server
Boot the server to the Gentoo install media. In most cases, you will be doing this via either a USB drive or a CD-ROM. In the CSL, you also have the option of using the Netboot server off of Steeltoe. If you would like to be able to remove the media after the server has finished booting, make sure to specify the docache argument. If you are installing over a serial console (such as HP's iLO or Sun's LOM), make sure to specify the console= argument with the appropriate serial device and settings (when in doubt, try console=ttyS0,9600,8,n,1 or console=ttyS1,9600,8,n,1; if the output looks garbled, try changing the baud rate from 9600 to 115200).

Take the defaults for any questions the live media asks until you reach the red livecd prompt.

Configuring Network Access
The first step in the installation is configuring network access. Execute the following commands to bring up networking.

ip link set eth0 up ip addr add 198.38.17.42/23 dev eth0 ip route add default via 198.38.17.254 dev eth0 echo "nameserver 198.38.16.40" > /etc/resolv.conf ping -c1 google.com

You should receive output similar to the following from the ping command:

PING google.com (74.125.228.101) 56(84) bytes of data. 64 bytes from iad23s08-in-f5.1e100.net (74.125.228.101): icmp_seq=1 ttl=55 time=17.1 ms --- google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 17.163/17.163/17.163/0.000 ms

Setup SSH Access
While serial consoles are great for emergency access, working on them for long periods of time can become very annoying in a hurry. For that reasons, we are going to setup SSH access to our server via the LiveCD environment.

To start, unless this is a new install, you should restore a copy of the server's SSH keys to the LiveCD environment. This will prevent you from having to work around messy SSH errors.

cd /etc/ssh/ scp :/path/to/sshkeys.

Next you need to set a root password on the LiveCD. This should be something relatively secure as it is the only security you have against your install being compromised remotely.

passwd

Finally, start sshd and reconnect to the server via SSH:

/etc/init.d/sshd start

Date and Time
Verify that the date and time are correct:

date

Note that the date and time are given in UTC which is 5 hours ahead of Eastern Standard Time (4 hours ahead during Daylight Savings Time). If the date and time are not correct, adjust them accordingly:

date MMDDhhmmYYYY #MM month, DD day, hh hours, mm minutes, YYYY year

Again, make sure to use UTC when setting the date and time. Be especially careful of date wrap-around if installing a server at night.

Disk Partitioning
You now need to identify the disk(s) on which you will be installing Gentoo Linux. On an older HP Server, the hardware RAID arrays generally appear as /dev/cciss/cXdY where X and Y are the controller and array number respectively. For newer HP Servers, the hardware RAID arrays will appear as /dev/sdX.

For Servers without a hardware RAID controller, the individual drives will usually appear as /dev/sd{a,b,c,etc}. In this case, you will likely want to setup mdadm software RAID for data redundancy and server availability.

You can use fdisk -l /path/to/disk to get some additional information (like capacity) about a connected drive.

For the rest of this article, we will assume that you will be installing linux to /dev/cciss/c0d0.

Run the following command to start fdisk targeted at your install drive:

fdisk /dev/cciss/c0d0

Use p to print the current partition layout and then use d to delete any existing partitions. NOTE: if you see the following message when you start fdisk, use o to create a new DOS partition table before you proceed:

WARNING: GPT (GUID Partition Table) detected on '/dev/cciss/c0d0'! The util fdisk doesn't support GPT. Use GNU Parted.

Use n to create the following partitions

Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-26460, default 1): Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-26460, default 26460): +100M Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 2 First cylinder (15-26460, default 15): Using default value 15 Last cylinder, +cylinders or +size{K,M,G} (15-26460, default 26460): +5G Command (m for help): n Command action e  extended p  primary partition (1-4) p Partition number (1-4): 3 First cylinder (666-26460, default 666): Using default value 666 Last cylinder, +cylinders or +size{K,M,G} (666-26460, default 26460): Using default value 26460

Use t to change the type on partition 3 to 8e (Linux LVM):

Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): 8e Changed system type of partition 3 to 8e (Linux LVM)

Use p to verify that the partition table looks similar to the following, then use w to write the changes to disk and close fdisk:

Command (m for help): p Disk /dev/cciss/c0d0: 218.5 GB, 218501038080 bytes 256 heads, 63 sectors/track, 26460 cylinders Units = cylinders of 16128 * 512 = 8257536 bytes Disk identifier: 0x00000000 Device Boot     Start         End      Blocks   Id  System /dev/cciss/c0d0p1              1          14      112864+  83  Linux /dev/cciss/c0d0p2             15         665     5249664   83  Linux /dev/cciss/c0d0p3            666       26460   208010880   8e  Linux LVM Command (m for help): w The partition table has been altered! Calling ioctl to re-read partition table. Syncing disks.

LVM Setup
We will be using LVM to dynamically manage the bulk of our disk space. LVM allows disk space to be reallocated among different partitions without having to repartition the physical harddisk.

Run the following commands to create our LVM volgroup:

pvcreate /dev/cciss/c0d0p3 vgcreate vg{hostname} /dev/cciss/c0d0p3

Create the following logical volumes to separate out important parts of the OS onto separate partitions:

lvcreate -L 5G -n usr vg{hostname} lvcreate -L 4G -n var vg{hostname} lvcreate -L 1G -n swap vg{hostname}

Filesystem Creation
We will be using ext4 for most of our filesystems; it is a well-tested and stable filesystem with a decent feature set.

First we create an ext2 filesystem for our /boot partition. We use ext2 for the /boot partition because it is very small and infrequently modified.

mkfs.ext2 /dev/cciss/c0d0p1

Make a swap filesystem on our swap partition and then activate it:

mkswap /dev/vg{hostname}/swap swapon /dev/vg{hostname}/swap

Make ext4 filesystems on the rest of our partitions:

mkfs.ext4 /dev/cciss/c0d0p2 mkfs.ext4 /dev/vg{hostname}/usr mkfs.ext4 /dev/vg{hostname}/var

Mounting the Filesystems
We will be mounting all of our filesystems with /mnt/gentoo/ as the root of our new installation:

mount /dev/cciss/c0d0p2 /mnt/gentoo cd /mnt/gentoo/ mkdir boot usr var mount /dev/cciss/c0d0p1 /mnt/gentoo/boot mount /dev/vg{hostname}/usr /mnt/gentoo/usr mount /dev/vg{hostname}/var /mnt/gentoo/var

Download Base Files
Grab the latest versions of the appropriate stage3 tarball for the server's architecture as well as the latest portage snapshot. NOTE - the below stage3 URL is most likely out of date; check your local gentoo mirror for the latest version.

cd /mnt/gentoo/ wget http://mirror.tjhsst.edu/gentoo/releases/amd64/current-stage3/stage3-amd64-20121013.tar.bz2 wget http://mirror.tjhsst.edu/gentoo/snapshots/portage-latest.tar.bz2

Unpack Base Files
Unpack the stage3 tarball and the portage snapshot:

cd /mnt/gentoo/ tar -xvf stage3-amd64-20121013.tar.bz2 #Now would be a good time to get some coffee tar -xvf portage-latest.tar.bz2 -C /mnt/gentoo/usr/ #Aaaand time for another coffee break... rm stage3-amd64-20121013.tar.bz2 portage-latest.tar.bz2

Preparing chroot environment
Copy the server template make.conf from the CSL Default Configs to /etc/portage/make.conf then edit it to set appropriate USE and MAKEOPTS values.

Make the overlay directory

mkdir -p /mnt/gentoo/usr/local/portage/overlay

Copy the resolv.conf file and the SSH host keys from the livecd environment to the chroot environment.

cp /etc/resolv.conf /mnt/gentoo/etc/ cp -av /etc/ssh/ssh_host_* /mnt/gentoo/etc/ssh/

Mount additional filesystems:

mount -t proc none /mnt/gentoo/proc mount -o bind /dev /mnt/gentoo/dev mount -o bind /sys /mnt/gentoo/sys

Chroot
chroot into the installation environment:

chroot /mnt/gentoo /bin/bash env-update source /etc/profile export PS1="(chroot)$PS1"

Edit /etc/locale.gen and uncomment the two en_US locales, then run the following command to update the generated locales.

locale-gen

Set the timezone in the installation environment

cp /usr/share/zoneinfo/America/New_York /etc/localtime

Copy over the CSL Overlay, then update the portage tree:

rsync -rv rsync://haimageserver.csl.tjhsst.edu/overlay/ /usr/local/portage/overlay/ emerge --sync

Fetching the Kernel Sources
Emerge git

emerge -a git

Clone the kernel sources

cd /usr/src/ git clone git://haimageserver.csl.tjhsst.edu/linux.git linux.git eselect kernel set linux.git cd linux git checkout v3.7

Configuring the Kernel
You can either use a CSL stock kernel configuration or build your own kernel configuration. To use a stock kernel configuration, copy the appropriate config file to /usr/src/linux/.config

If you are building your own kernel configuration, check the wiki page for your server model and planned roles for recommendations on driver and other configuration choices.

Building and Installing the Kernel
Execute the following command to build the kernel; adjust 5 to be equal to the value you set for MAKEOPTS in /etc/portage/make.conf

make -j5

Once again now's a good time to take a break while the kernel compiles. When it is done, run the following commands to install the kernel and modules. NOTE: the suffix for the config and the kernel files below should match the module directory name in /lib/modules/

make modules_install cp .config /boot/config-3.7.0-kvm cp arch/x86/boot/bzImage /boot/kernel-3.7.0-kvm

Building and Installing the Initramfs
We need to build an initramfs to support our separate /usr partition. The easiest way to do this is using the latest version of Genkernel. First, we need to install it:

mkdir -p /etc/portage/package.keywords echo "sys-kernel/genkernel" >> /etc/portage/package.keywords/genkernel emerge -a genkernel

Then we just need to tell genkernel to build an initramfs. Note, if you are using mdadm, you will need to add --mdadm to the genkernel arguments.

genkernel --lvm initramfs

Symlinking the Kernel and initramfs
We create easy-to-remember symlinks for the kernel and the initramfs:

cd /boot/ ln -snf kernel-3.7.0-kvm gentoo ln -snf initramfs-genkernel-x86_64-3.2.0-kvm gentoo-initramfs

Essential System Software
There are a few applications that are essential to the functionality of our new system. We should install them now. They include:


 * grub - Boot loader
 * syslog-ng - System logger
 * vixie-cron - cron daemon for scheduled tasks
 * logrotate - rotates system logs
 * iproute2 - needed for configuring network interfaces
 * lvm2 - needed if you're using LVM for disk management
 * mdadm - needed if you're using MDADM for software RAID
 * ifenslave - needed for bonding network interfaces
 * vconfig - needed for configuring VLAN tagging

Install an appropriate selection from the above list using a command similar to the following:

emerge -a grub syslog-ng vixie-cron logrotate iproute2 lvm2 ifenslave

System Configuration
We need to update a number of configuration files before our system can be considered operational.

/etc/fstab
Add / edit the following lines in /etc/fstab:

/dev/cciss/c0d0p1      /boot           ext2            noatime         1 2 /dev/cciss/c0d0p2      /               ext4            noatime         0 1 /dev/vg{hostname}/usr   /usr            ext4            noatime         0 2 /dev/vg{hostname}/var   /var            ext4            noatime         0 2 /dev/vg{hostname}/swap  none            swap            sw              0 0

/etc/conf.d/hostname
edit /etc/conf.d/hostname and set the system's hostname:

hostname="fiordland"

/etc/conf.d/hwclock
edit /etc/conf.d/hwclock and uncomment the following two lines and set both to YES:

clock_hctosys="YES" clock_systohc="YES"

/etc/timezone
Preserve the timezone across updates of sys-libs/timezone-data with the following command:

echo "America/New_York" > /etc/timezone

/etc/conf.d/net
Setup the system's networking configuration. You may want to consult the server's livedoc page as well as any role pages (VM server, storage server) for detailed information on networking configuration for a particular server. The below configuration is for redundant access (bonding/etherchannel) to a single network or VLAN:

slaves_bond0="eth0 eth1" config_bond0="198.38.17.42/23" routes_bond0="default via 198.38.17.254" dns_servers_bond0="198.38.16.40 198.38.16.41 151.188.14.2" dns_search_bond0="csl.tjhsst.edu tjhsst.edu sun.tjhsst.edu"

Remember to create any needed networking scripts:

cd /etc/init.d/ ln -snf net.lo net.bond0

/etc/inittab
Find and uncomment the following line to enable a console on the iLO Virtual Serial Port:

s0:12345:respawn:/sbin/agetty 9600 ttyS0 vt100

/etc/securetty
Make sure the following line is in /etc/securetty to allow root logins on the iLO Virtual Serial Port:

ttyS0

Boot Services
Setup various services to start at boot:

rc-update add lvm boot rc-update add net.bond0 default rc-update add sshd default rc-update add syslog-ng default rc-update add vixie-cron default

Root Password
Set the root password (You'll be really sorry if you forget this step...)

passwd

/boot/grub/grub.conf
Add / edit the following lines in /boot/grub/grub.conf

default 0 timeout 30 title Gentoo Linux / KVM root (hd0,0) kernel /boot/gentoo real_root=/dev/cciss/c0d0p2 dolvm bonding.mode=4 bonding.miimon=100 console=tty0 console=ttyS0,9600,8,n,1 initrd /boot/gentoo-initramfs

Grub MBR Installation
Run the following commands to install Grub into the MBR:

grep -v rootfs /proc/mounts > /etc/mtab grub-install --no-floppy /dev/cciss/c0d0

Note in some cases grub-install doesn't function, particularly when using mdadm. In this case, you may have to manually install Grub into the MBR via the grub shell.

grub grub> root (hd0,0) grub> setup (hd0) grub> quit

If using mdadm, you will want to repeat the above steps to install Grub into the MBR on each drive so that the system remains bootable in the event of a drive loss.

Exit chroot and Reboot
Exit the chroot, unmount all filesystems, and reboot to your new installation:

exit cd umount /mnt/gentoo/boot umount /mnt/gentoo/dev umount /mnt/gentoo/proc umount /mnt/gentoo/sys umount /mnt/gentoo/usr umount /mnt/gentoo/var umount /mnt/gentoo sync sync reboot

Note that most failures to reboot successfully are due to kernel issues, particularly missing drivers. If your system starts to boot the kernel but then crashes, make sure you have included all the necessary drivers, particularly HDD drivers in the kernel (not as modules). Also if you are using MDADM or LVM, make sure you have the appropriate drivers selected and applications installed.

Postinstall Networking
If your networking configuration specifies bonding, you will need to re-enable bonding on the switch before you can access the server over the network. First we assign both interfaces to a channel-group:

TJHSST-4500MSL#conf t Enter configuration commands, one per line. End with CNTL/Z. TJHSST-4500MSL(config)#inter Gi2/3 TJHSST-4500MSL(config-if)#channel-protocol lacp TJHSST-4500MSL(config-if)#channel-group 8 mode active TJHSST-4500MSL(config-if)#switchport mode access TJHSST-4500MSL(config-if)#switchport access vlan 1600 TJHSST-4500MSL(config-if)#spanning-tree portfast TJHSST-4500MSL(config-if)#exit TJHSST-4500MSL(config)#inter Gi2/4 TJHSST-4500MSL(config-if)#channel-protocol lacp TJHSST-4500MSL(config-if)#channel-group 8 mode active TJHSST-4500MSL(config-if)#switchport mode access TJHSST-4500MSL(config-if)#switchport access vlan 1600 TJHSST-4500MSL(config-if)#spanning-tree portfast TJHSST-4500MSL(config-if)#exit

Next we configure the channel-group interface:

TJHSST-4500MSL(config)#inter Port 8 TJHSST-4500MSL(config-if)#switchport mode access TJHSST-4500MSL(config-if)#switchport access vlan 1600 TJHSST-4500MSL(config-if)#spanning-tree portfast TJHSST-4500MSL(config-if)#exit TJHSST-4500MSL(config)#exit

You can verify the status of the Etherchannel with the following command:

TJHSST-4500MSL#show etherchannel 8 sum Flags: D - down        P - bundled in port-channel I - stand-alone s - suspended R - Layer3     S - Layer2 U - in use     f - failed to allocate aggregator M - not in use, minimum links not met u - unsuitable for bundling w - waiting to be aggregated d - default port Number of channel-groups in use: 19 Number of aggregators:          19 Group Port-channel  Protocol    Ports --+-+---+--- 8      Po8(SU)         LACP      Gi2/3(P)    Gi2/4(P)

Additional Software
There are a number of additional software packages that are required to make our system fully operational. Start by copying the portage configurations from the CSL Default Configs or from a backup.

rsync -rv hostname:/path/to/package.keywords /etc/portage/ rsync -rv hostname:/path/to/package.use /etc/portage/

Next update portage:

emerge -a1 portage

Make sure nano gets kept around (it is an editor that is available without /usr):

emerge --noreplace nano

New Software
Install some new packages to provide debugging, monitoring, and useful functionality:


 * eix - easily search for packages
 * genlop - gentoo log parser
 * gentoolkit - a selection of useful portage utilities
 * nss_ldap - access user account information stored in LDAP
 * heimdal - an implementation of Kerberos
 * pam_krb5 - use Kerberos for password validation
 * app-misc/screen - terminal multiplexer
 * iftop - graph network traffic
 * nrpe - nagios remote plugin executor
 * nsca - nagios service check acceptor
 * nagios-plugins - nagios monitoring plugins
 * nmap - network mapping utility
 * bind-tools - a collection of DNS utilities
 * ntp - implementation of the network time protocol
 * pciutils - list information on PCI cards
 * usbutils - list informaton on USB devices
 * lsof - manage file accesses
 * htop - interactive process viewer

emerge -a eix genlop gentoolkit nss_ldap heimdal pam_krb5 app-misc/screen iftop nrpe nsca nagios-plugins nmap bind-tools ntp pciutils usbutils lsof htop

Software Updates
Start a screen and begin a world update to apply any software updates that have been released since the stage3 was built.

screen #now inside the screen emerge -auND @world emerge -a @preserved-rebuild emerge -a --depclean revdep-rebuild -- -a

Configuration Files
Copy the following files/directories from the CSL Default Configs

/etc/krb5.conf /etc/ldap.conf /etc/ntp.conf /etc/conf.d/ntp-client /etc/portage/postsync.d/ /etc/nsswitch.conf /etc/ssh/ssh_config /etc/ssh/sshd_config /etc/pam.d/system-auth /etc/issue /etc/nagios/nrpe.cfg /etc/nagios/send_nsca.cfg /root/.k5login /root/scripts/

/etc/issue
Edit /etc/issue to reflect the system's proper FQDN

/etc/nagios/nrpe.cfg
Edit /etc/nagios/nrpe.cfg, set the system's IP. Adjust check values if needed (eg, adjust user limits higher on a remote access server).

/etc/security/access.groups
Create /etc/security/access.groups with the appropriate access groups, one per line (should be at least the hostname group and the allaccess group).

allaccess fiordland

/root/.k5login
Add any needed /root principals to /root/.k5login to grant them root access via ksu.

/etc/krb5.keytab
Restore /etc/krb5.keytab if you have a backup; otherwise use the following command to generate a new one:

ktutil get -p ahamilto/admin host/fiordland.csl.tjhsst.edu

Verify that the permissions on /etc/krb5.keytab are root:root and 0600

chown root:root /etc/krb5.keytab chmod 0600 /etc/krb5.keytab