Managing Non-Scyld Nodes

A ClusterWare cluster typically consists of a Scyld master node and one or more Scyld compute nodes, integrated and communicating across the private cluster network interface. However, ClusterWare also supports additional devices and nodes that may reside on that private cluster network. This section describes how these Scyld and non-Scyld nodes are configured using entries in the /etc/beowulf/config file.

DHCP IP address assignment to devices

The private cluster network may have one or more devices attached to it that issue a DHCP request to obtain a dynamic IP address, vs. the device being configured with a static IP address. Typically, only the master node (or nodes - see Managing Multiple Master Nodes) owns a static IP address.


Care must be taken with static IP addresses to guarantee there are no address collisions.

Examples of such devices are managed switches and storage servers. The beoserv DHCP service for such devices is configured using the host directive, together with an associated hostname directive. For example,

nodes 32
iprange   # IPaddr range of compute nodes
hostrange    # IPaddr range of devices for DHCP
hostrange   # IPaddr range of PDUs for DHCP
host 00:A0:D1:E9:87:CA smartswitch
host 00:A0:D1:E3:FC:E2 pdu1
host 00:A0:D1:E3:FD:4A pdu2

The host keyword affects both the beoserv DHCP server and how the ClusterWare NSS responds to hostname lookups. The host keyword associates a non-cluster entity, identified by its MAC address, to an IP address that should be delivered to that client entity, if and when it makes a DHCP request to the master node, together with one or more optional hostnames to be associated with this IP address.

If the hostname is provided, then normal NSS functionality is available. Using the above example, then:

[user1@cluster ~] $ getent hosts smartswitch

returns: smartswitch


[user1@cluster ~] $ getent ethers 00:A0:D1:E9:87:CA


00:a0:d1:e9:87:ca smartswitch

Each host IP address must fall within a defined hostrange range of IP addresses. Moreover, each of the potentially multiple hostrange ranges must not overlap any other range, must not overlap the cluster compute nodes range that is defined by the iprange directive, and must not collide with IP address(es) of master node(s) on the private network.

Simple provisioning using PXE

A default node entry, such as:

node 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

or an explicitly numbered node entry, such as one for node15:

node 15 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

is assumed to be a Scyld node, and a PXE request from one of these MAC addresses results in beoserv provisioning the node with the kernel image, initrd image, and kernel command-line arguments that are specified in /etc/beowulf/config file entries, e.g.:

kernelimage /boot/vmlinuz-2.6.18-164.2.1.el5.540g0000
initrdimage /var/beowulf/boot/computenode.initrd
kernelcommandline rw root=/dev/ram0 image=/var/beowulf/boot/computenode.rootfs

ClusterWare automatically maintains the config file’s default kernelimage to specify the same kernel that currently executes on the master node. A Scyld node integrates into the BProc unified process space.

Enhanced syntax allows for custom booting of different kernel and initrd images. For example, specific nodes can boot a standalone RAM memory test in lieu of booting a full Linux kernel:

kernelimage 15 /var/beowulf/boot/memtest86+-4.00.bin
initrdimage 15 none
kernelcommandline 15 none

Thus when node15 makes a PXE request, it gets provisioned with the specified binary image that performs a memory test. In the above example, the initrdimage of none means that no initrd image is provisioned to the node because that particular memory test binary doesn’t need an initrd. Moreover, the node number specifier of 15 can be a range of node numbers, each of which would be provisioned with the same memory test.

Simple provisioning using the class directive

An optional config file class directive assigns a name to a set of image and kernel command-line arguments. The previous example can be alternatively accomplished with:

class memtest kernelimage /var/beowulf/boot/memtest86+-4.00.bin
class memtest initrdimage none
class memtest kernelcommandline none
node 15 memtest 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

which results in the same memory test provisioning of node15 as seen earlier.

Similarly, the default Scyld node provisioning can be expressed as:

class scyld kernelimage /boot/vmlinuz-2.6.18-164.2.1.el5.540g0000
class scyld initrdimage /var/beowulf/boot/computenode.initrd
class scyld kernelcommandline rw root=/dev/ram0 image=/var/beowulf/boot/computenode.rootfs
node scyld pxe pxe 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

The first pxe is termed the boot-sequence, and the second pxe is termed the boot-stage. The boot-stage describes how beoserv should respond to a node’s PXE request. In the example above, the boot-stage of pxe instructs beoserv to respond to the node’s first PXE request with the kernel image, initrd image, and kernel command-line specified in the class scyld.

Booting a node from the local harddrive

The node entry’s boot-sequence and boot-stage have more powerful capabilities. For example, suppose node15 is installed with a full distribution of CentOS 4.8 on a local harddrive, and suppose the master node’s config file contains entries:

class genericboot kernelimage none
class genericboot initrdimage none
class genericboot kernelcommandline none
node 15 genericboot pxe+local local 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

When node15 boots, it first makes a DHCP request to join the private cluster network, then it attempts to boot, abiding by the specific sequence of boot devices named in its BIOS. ClusterWare expects that the first boot device is PXE over Ethernet, and the second boot device is a local harddrive. When node15 initiates its PXE request to the master node, beoserv sees the boot-stage of local and thus directs node15 to “boot next”, i.e., to boot from the local harddrive.

Provisioning a non-Scyld node

In the previous example, we assumed that node15 already had a functioning, bootable operating system already installed on the node. Having a preexisting installation is not a requirement. Suppose the config file contains entries:

class centos5u4 kernelimage /var/beowulf/boot/vmlinuz-centos5u4_amd64
class centos5u4 initrdimage /var/beowulf/boot/initrd-centos5u4_amd64.img
class centos5u4 kernelcommandline initrd=initrd-centos5u4_amd64.img
                  ks=nfs: ksdevice=eth0
node 15 centos5u4 pxe+local pxe 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

(where the kernelcommandline has been broken into two lines for readability, although in reality it must be a single line in the config file). This time node15’s PXE request arrives, and the boot-stage of pxe directs beoserv to respond with the class centos5u4 kernel image, initrd image, and kernel command-line arguments. The latter’s ks arguments informs node15’s kernel to initiate a kickstart operation, which is a Red Hat functionality that provisions the requester with rpms and other configuration settings as specified in the /home/os/kickstarts/n5-ks.cfg kickstart configuration file found on the master node. It is the responsibility of the cluster administrator to create this kickstart file. See Special Directories, Configuration Files, and Scripts for a sample configuration file.

After this initial PXE response (i.e., the pxe step of the pxe+local boot-sequence), beoserv rewrites the node entry to change the boot-stage to the local second step of the pxe+local boot-sequence. For example,

node 15 centos5u4 pxe+local pxe 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

gets automatically changed to:

node 15 centos5u4 pxe+local local 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

What this accomplishes is: the first PXE request is met with a directive to boot a kernel on node15 that initiates the kickstart provisioning, and then any subsequent PXE request from node15 (presumably from a now-fully provisioned node) results in a beoserv directive to node15 to “boot next”, i.e., to boot from the local harddrive.

If the cluster administrator wishes to reprovision the node and start fresh, then simply change the boot-stage from local back to pxe, and execute systemctl reload clusterware to instruct beoserv to re-read the config file to see your manual changes.

If you want the node to kickstart reprovision on every boot (albeit an unlikely scenario, but presented here for completeness), then you would configure this using:

node 15 centos5u4 pxe pxe 00:A0:D1:E5:C4:6E 00:A0:D1:E5:C4:6F

Integrating a non-Scyld node into the cluster

A non-Scyld node that locally boots a full distribution operating system environment may have an assigned IP address in the private cluster network iprange, but it is initially invisible to the master node’s monitoring tools and job manager. The bpstat tool only knows about Scyld nodes, and the more general beostatus is ignorant of the non-Scyld node’s presence in the cluster. The non-Scyld node is itself ignorant about the names and IP addresses of other nodes in the cluster, whether they be Scyld or non-Scyld nodes, until and unless the cluster administrator adds each and every node into the non-Scyld node’s local /etc/hosts file.

This shortcoming can be remedied by installing two special ClusterWare packages onto the non-Scyld node: beostat-sendstats and beonss-kickbackclient. These packages contain the client-side pieces of beostat and beonss. They are available in the standard ClusterWare yum repository and are compatible with non-Scyld RHEL and CentOS distributions - and perhaps with other distributions. One way to judge compatibility is to determine what libraries the ClusterWare daemons need to find on the non-Scyld compute node. (The daemons are known to execute in recent RHEL and CentOS environments.) Examine the daemons that were installed on the master node when ClusterWare was installed:

ldd /usr/sbin/sendstats
ldd /usr/sbin/kickbackproxy

and then determine if the libraries that these binaries employ are present on the target non-Scyld node. If the libraries do so exist, then the special ClusterWare packages can be downloaded and installed on a non-Scyld node.

First, you should download the packages from the ClusterWare yum repo. A useful downloader is the /usr/bin/yumdownloader utility, which can be installed from the CentOS extras yum repository if it is not already installed on your master node:

[root@cluster ~] # yum install yum-utils

Then use the utility to download the special Penguin ClusterWare rpms:

[root@cluster ~] # yumdownloader --destdir=<localdir> beostat-sendstats beonss-kickbackclient

retrieves the rpms and stores them into the directory <localdir>, e.g., /var/www/html or /etc/beowulf/nonscyld.

These special packages can be installed manually on the non-Scyld node, or can be installed as part of the kickstart procedure (see Provisioning a non-Scyld node). Each package includes a /etc/init.d/ script that must be edited by the cluster administrator. Examine /etc/init.d/beostat-sendstats and /etc/init.d/beonss-kickbackclient, which contain comments that instruct the administrator about how to configure each script. Additionally, the non-Scyld node’s /etc/nsswitch.conf must be configured to invoke the kickback service for the databases that the administrator wishes to involve beonss and the master node. See the master node’s /etc/beowulf/nsswitch.conf for a guide to which databases are supported, e.g., hosts, passwd, shadow, and group. Finally, on the non-Scyld node, enable the scripts to start at node startup:

[root@cluster ~] # chkconfig beostat-sendstats on
[root@cluster ~] # chkconfig beonss-kickbackclient on