beowulf-fstab

Name

/etc/beowulf/fstab – ClusterWare compute node filesystem control table

Description

The /etc/beowulf/fstab file on the master node contains a list of filesystems to be mounted on compute nodes at boot time. Its purpose, format, and contents are similar to the traditional /etc/fstab, plus a few additional cluster-specific features.

The ClusterWare fstab system is designed to keep all configuration information on a master node. The /etc/beowulf/fstab file is the default for all compute nodes. Any optional node-specific /etc/beowulf/fstab.N file overrides this default file for node number N.

The root filesystem on each compute node is a tmpfs filesystem that is automatically sized for the available RAM. In earlier versions of Scyld ClusterWare, this root filesystem was explicitly declared in fstab, but this is no longer done.

The compute node’s root filesystem is used to dynamically cache binaries and libraries from the master, to provide space for /tmp and /var/tmp, to provide mountpoints for NFS mounts, etc. Although ClusterWare does not require a harddrive on a Scyld compute node, some clusters employ harddrive(s) for node-local persistent storage, for “scratch” storage to avoid having /tmp and /var/tmp consume tmpfs RAM, or for swap space to expand the available virtual memory space and thus reduce Out-of-Memory conditions.

The ClusterWare fstab interacts with the mkfs and fsck directives in the /etc/beowulf/config file (see man beowulf-config) to control automatic creation or boot-time checking (and potentially repairing) of compute node filesystems on node-local harddrives.

A directive mkfs always specifies to rebuild at boot time every harddrive partition specified in /etc/beowulf/fstab, and thus should be used with great care so as to not automatically rebuild a partition and thus destroy data that is expected to survive across compute node reboots. Normally the default directive mkfs never is used.

A directive fsck full specifies to check and potentially repair at boot time every harddrive partition. Alternatively, fsck safe specifies to perform an fsck, but to not attempt any repairs; after boot, the cluster administrator may manually perform repairs as needed. A directive fsck never is the default, which specifies that no checking be done at boot time.

Syntax

The syntax and layout is identical to the master node’s /etc/fstab file. The file contents are processed line by line. All blank lines and lines that begin with a “#” are ignored. All other lines should have six fields, separated by tabs or spaces.

The first field is the device to mount. For filesystems on local harddrives, this should point to a /dev entry, such as /dev/hda2. If mounting an NFS filesystem, the device should be specified as hostname:directory, where hostname is the IP address of the NFS server, and directory is the path on the NFS server you want to mount. If the NFS server is the master node, you can use “$MASTER” as the hostname. Currently, hostname cannot be an actual alphanumeric host name because /etc/beowulf/fstab is evaluated at boot time before the compute node’s name service is initialized. For some special filesystems, such as proc and devpts, the hostname can be set to “none”.

The second field is the mount point. For a swap partition, this should be “swap”, but for all other filesystems, this must be a path that begins with “/”. Any paths that you specify as mount points will be automatically created by the node_up script before it tries to mount the filesystem. Ensure that you do not specify the same mount point on more than one line, because this can cause problems. You can have multiple lines that use “swap” as the mount point, but that is the only exception to the rule.

The third field is the filesystem type. This should be “swap” for swap partitions, or a standard Linux filesystem type (e.g., “ext2”, “ext3”, “xfs”), or “nfs” for an NFS file system, or particular pseudo filesystem types (e.g., “proc” for the proc filesystem, “devpts” for the devpts filesystem). Any filesystem that can normally be used by Linux can also be specified here, but you must also take steps to create the harddrive filesystems on the compute nodes before attempting to mount them.

The fourth field lists the mount options for the filesystem. All options should be comma-separated with no spaces. If you do not know of any specific options to use, then you should use the “defaults” keyword.

In addition to the mount options normally supported by Linux, one additional option is supported by ClusterWare: “nonfatal”. Normally, any mount failure results in an immediate abort of the node boot, and the node state transitions from “boot” to “error”. Adding “nonfatal” to the options overrides this behavior and allows the node boot to continue, potentially to a node “up” state. However, because filesystem mounts have in fact failed, the node may not actually have full functionality. When using the “nonfatal” option, the cluster administrator is encouraged to view the ClusterWare boot log files found in directory /var/log/beowulf/ to discover potential mount failures and other warnings or soft error conditions. The “nonfatal” option is useful for harddrive filesystems when not all compute nodes share the same number and partitioning of drives, or when NFS mounts might fail because an NFS server is temporarily unavailable or the specified filesystem is not currently exported.

The fifth and sixth fields are left there for compatibility with the standard fstab format. These fields are not used at the moment, but are required to be there. We recommend they both be set to “0”.

Examples

# This file is the fstab for nodes.
# One difference is that we allow for shell variable expansions...
#
# Variables that will get substituted:
#  MASTER = IP address of the master node.  (good for doing NFS mounts)

# This is the default setup from beofdisk, once you setup your disks.
#/dev/hda2  swap        swap    defaults,nonfatal   0 0
#/dev/hda3  /       ext2    defaults,nonfatal   0 0

# These should always be added
none        /proc       proc    defaults    0 0
none        /dev/pts    devpts  gid=5,mode=620  0 0

# NFS (for example and default friendliness)
$MASTER:/home   /home       nfs nolock,nonfatal     0 0

Files

/etc/beowulf/fstab, /etc/beowulf/fstab., /etc/beowulf/config