Installation and Upgrade of Scyld ClusterWare

The Scyld ClusterWare scyld-install script installs the necessary packages from the ClusterWare yum repositories, and installs dependency packages as needed from the base distribution (e.g., Red Hat RHEL or CentOS) yum repositories.

Important

Do not install ClusterWare as an upgrade to an existing ClusterWare 6 or 7 installation. Instead, install Scyld ClusterWare on a non-ClusterWare system that ideally is a virtual machine. (See Required and Recommended Components.)

Important

The head node(s) must use a Red Hat RHEL- or CentOS-equivalent base distribution release 7.6 or later environment, due to dependencies on newer libvirt and selinux packages.

Note

Clusters commonly employ multiple head nodes. The instructions in this section describe installing ClusterWare on the first head node. To later install ClusterWare on additional head nodes, see Managing Multiple Head Nodes.

scyld-install anticipates being potentially executed by a non-root user, so ensure that your userid can execute sudo. Additionally, if using sudo behind a proxy, then because sudo clears certain environment variables for security purposes, the cluster administrator should consider adding several lines to /etc/sudoers:

Defaults    env_keep += "HTTP_PROXY http_proxy"
Defaults    env_keep += "HTTPS_PROXY https_proxy"
Defaults    env_keep += "NO_PROXY no_proxy"

Important

Various commands that manipulate images execute as user root, thereby requiring that the commands internally use sudo and requiring that user root must have access to the administrator's workspace which contains the administrator's images. Typically the per-user workspace is ~/.scyldcw/workspace/. If that directory is not accessible to the command executing as root, then another accessible directory can be employed, and the administrator can identify that alternative pathname by adding a modimg.workspace setting to ~/.scyldcw/settings.ini.

Important

scyld-install uses the yum command to access Scyld ClusterWare and potentially various other repositories (e.g., Red Hat RHEL or CentOS) that by default normally reside on Internet websites. However, if the head node(s) do not have Internet access, then the required repositories must reside on local storage that is accessible by the head node(s). See Appendix: Creating Local Repositories without Internet.

Execute the ClusterWare install script

If /etc/yum.repos.d/clusterware.repo exists, then scyld-install's subsequent invocations of yum will employ that configuration file. If /etc/yum.repos.d/clusterware.repo does not exist, then scyld-install prompts the user for an appropriate authentication token and uses that to build a /etc/yum.repos.d/clusterware.repo that is customized to your cluster.

scyld-install accepts an optional argument specifying a cluster configuration file that contains information necessary to set up the DHCP server. For example:

cat <<-EOF >/tmp/cluster-conf
interface enp0s9            # names the private cluster interface
nodes 4                     # max number of compute nodes
iprange 10.10.32.45         # starting IP address of node 0
node 08:00:27:f0:44:35      # node 0 MAC address
node 08:00:27:f0:44:45      # node 1 MAC address
node 08:00:27:f0:44:55      # node 2 MAC address
node 08:00:27:f0:44:65      # node 3 MAC address
EOF

where the syntax of this cluster configuration file is:

domain <DOMAIN_NAME>

Optional. Defaults to "cluster.local".

interface <INTERFACE_NAME>

Optional. Specifies the name of head node's interface to the private cluster network, although that can be determined from the specification of the <FIRST_IP> in the iprange line.

nodes <MAX_COUNT>

Optional. Specifies the max number of compute nodes, although that can be determined from the iprange if both the <FIRST_IP> and <LAST_IP> are present. The max will also adjust as-needed if and when additional nodes are defined. For example, see Node Creation with Known MAC address(es).

iprange <FIRST_IP> [<LAST_IP>]

Specifies the IP address of the first node (which defaults to n0) and optionally the IP address of the last node. The <LAST_IP> can be deduced from the <FIRST_IP> and the nodes <MAX_COUNT>. The <FIRST_IP> can include an optional netmask via a suffix of /<BIT_COUNT> (e.g., /24) or a mask (e.g., /255.255.255.0).

<FIRST_INDEX> <FIRST_IP> [<LAST_IP>] [via <FROM_IP>] [gw <GATEWAY_IP>]

This is a more elaborate specification of a range of IP addresses, and it is common when using DHCP relays or multiple subnets. <FIRST_INDEX> specifies that the first node in this range is node n<FIRST_INDEX> and is assigned IP address <FIRST_IP>; optionally specifies that the range of nodes make DHCP client requests that arrive on the interface that contains <FROM_IP>; optionally specifies that each DHCP'ing node be told to use <GATEWAY_IP> as their gateway, which otherwise defaults to the IP address (on the private cluster network) of the head node.

For example: 128 10.10.24.30/24 10.10.24.100 via 192.168.65.2 gw 10.10.24.254 defines a DHCP range of 71 addresses, the first starting with 10.10.24.30, and assigns the first node in the range as n128; watches for DHCP requests arriving on the interface containing 192.168.65.2; and tells these nodes to use 10.10.24.254 as the their gateway.

node [<INDEX>] <MAC> [<MAC>]

One compute node per line, and commonly consisting of multiple node lines, where each DHCP'ing node is recognized by its unique MAC address and is assigned an IP address using the configuration file specifications described above. Currently only the first <MAC> is used. An optional <INDEX> is the index number of the node that overrides the default of sequentially increasing node number indices and thereby creates a gap of unassigned indices. For example, a series of eight node lines without an <INDEX> that is followed by node 32 52:54:00:c4:f7:1e creates a gap of unassigned indices n8 to n31 and assigns this node as n32.

Note

ClusterWare yum repositories contain RPMs that duplicate various Red Hat EPEL RPMs, and these ClusterWare RPMs get installed or updated in preference to their EPEL equivalents, even if /etc/yum.repos.d/ contains an EPEL .conf file.

Note

ClusterWare employs userid/groupid 539 to simplify communication between the head node(s) and the backend shared storage where it stores node image files, kernels, and initramfs files. If the scyld-install script detects that this uid/gid is already in use by other software, then the script issues a warning and chooses an alternative new random uid/gid. The cluster administrator needs to set the appropriate permissions on that shared storage to allow all head nodes to read and write all files.

The ClusterWare database is stored as JSON content within a replicated document store distributed among the ClusterWare head nodes. This structure protects against the failure of any single head node.

For example, using the cluster-config created above, install ClusterWare from a yum repo:

scyld-install --config /tmp/cluster-conf

By default scyld-install creates the DefaultImage that contains a kernel and rootfs software from the same base distribution installed on the head node, although if the head node executes RHEL8, then no DefaultImage and DefaultBoot are created.

Alternatively, for more flexibility (especially with a RHEL8 head node), execute the installer with an additional option that identifies the base distribution to be used for the DefaultImage:

scyld-install --config /tmp/cluster-conf --os-iso <ISO-file>

where <ISO-file> is either a pathname to an ISO file or a URL of an ISO file. That ISO can match the head node's distribution or can be any supported distribution.

scyld-install unpacks an embedded compressed payload and performs the following steps:

  • Checks for a possible newer version of the clusterware-installer RPM. If one is found, then the script will update the local RPM installation and execute the newer scyld-install script with the same arguments. An optional argument --skip-version-check bypasses this check.

  • An optional argument --yum-repo /tmp/clusterware.repo re-installs a yum repo file to /etc/yum.repos.d/clusterware.repo. This is unnecessary if /etc/yum.repos.d/clusterware.repo already exists and is adequate.

  • Checks whether the clusterware RPM is installed.

  • Confirms the system meets various minimum requirements.

  • Installs the clusterware RPM and its supporting RPMs.

  • Copies a customized Telegraf configuration file to /etc/telegraf/telegraf.conf

  • Enables the tftpd service in xinetd for PXE booting.

  • Randomizes assorted security-related values in /opt/scyld/clusterware/conf/base.ini

  • Sets the current user account as a ClusterWare administrator in /opt/scyld/clusterware/conf/base.ini. If this is intended to be a production cluster, then the system administrator should create additional ClusterWare administrator accounts and clear this variable. For details on this and other security related settings, including adding ssh keys to compute nodes, please see the Installation & Administrator Guide section Securing the Cluster.

  • Modifies /etc/yum.repos.d/clusterware.repo to change enabled=1 to enabled=0. Subsequent executions of scyld-install to update ClusterWare will temporarily (and silently) re-enable the ClusterWare repo for the duration of that command. This is done to avoid inadvertent updates of ClusterWare packages if and when the clusterware administrator executes a more general yum install or yum update intending to add or update the base distribution packages.

Then scyld-install uses systemd to enable and start firewalld, and opens ports for communication between head nodes as required by etcd. See Services, Ports, Protocols for details.

Once the ports are open, scyld-install initializes the ClusterWare database and enables and starts the following services:

  • httpd: The Apache HTTP daemon that runs the ClusterWare service and proxies Grafana.

  • xinetd: Provides network access to tftp for PXE booting.

  • Telegraf: Collects head node performance data and feeds into InfluxDB.

  • InfluxDB: Stores node performance and status data for visualization in Grafana.

  • Grafana: Displays the head node and compute node status data through a web interface.

The script then:

  • Opens ports in firewalld for public access to HTTP, HTTPS, TFTP, iSCSI, and incoming Telegraf UDP messages.

  • Installs and configures the cluster administrator's clusterware-tools package (unless it was executed with the --no_tools option).

  • Configures the cluster administrator's ~/.scyldcw/settings.ini to access the newly installed ClusterWare service using the scyld-tool-config tool.

  • Creates an initial simple boot image DefaultImage, boot config DefaultBoot, and attributes DefaultAttribs using the scyld-add-boot-config tool.

  • Loads the cluster configuration specified on the command line using the scyld-cluster-conf load command.

  • Restarts the httpd service to apply the loaded cluster configuration.

Important

See the Node Images and Boot Configurations for details about how to modify existing boot images, create new boot images, and associate specific boot images and attributes with specific compute nodes. We strongly recommend not modifying or removing the initial DefaultImage, but rather cloning that basic image into a new image that gets modified further, or just creating new images from scratch.

Important

If you wish to ensure that the latest packages are installed in the image after the scyld-install, then execute scyld-modimg -i DefaultImage --update --overwrite --upload.

Important

See Common Additional Configuration for additional optional cluster configuration procedures, e.g., installing and configuring a job scheduler, installing and configuring one of the MPI family software stacks.

Important

If this initial scyld-install does not complete successfully, or if you want to begin the installation anew, then when/if you re-run the script, you should cleanse the partial, potentially flawed installation by adding the --clear argument, e.g., scyld-install --clear --config /tmp/cluster-conf. If that still isn't sufficient, then scyld-install --clear-all --config /tmp/cluster-conf does a more complete clearing, then reinstalls all the ClusterWare packages.

Due to licensing restrictions, when running on a Red Hat RHEL system, the installer will still initially create a CentOS compute node image as the DefaultImage. If after this initial installation a cluster administrator wishes to instead create compute node images based on RHEL, then use the scyld-clusterctl repos tool as described in Appendix: Creating Arbitrary RHEL Images, and create a new image (e.g., DefaultRHELimage) to use as a new default.

Configure additional cluster administrators

The ClusterWare administrator's command-line tools are found in the clusterware-tools package. which is installed by default on the head node by scyld-install. It can be additionally installed on any system that has HTTP (or HTTPS, see Securing the Cluster) access to a ClusterWare head node in the cluster.

To install these tools on a machine other than the head node, login to that other system, copy /etc/yum.repos.d/clusterware.repo from a head node to the same location on this system, then execute:

sudo yum install clusterware-tools

Once the tools are installed, each administrator must configure a connection to the ClusterWare service, which is controlled by variables in the user's ~/.scyldcw/settings.ini file. The scyld-tool-config tool script is provided by the clusterware-tools package. The contents of the settings.ini file are discussed in the Reference Guide. Running that tool and answering the on-screen questions will generate a settings.ini file, although administrators of more advanced cluster configurations may need to manually add or edit additional variables.

Once the settings.ini is created, you can test your connection by running a simple node query:

scyld-nodectl ls

This query may complain at this time that no nodes exist or no nodes are selected, although such a complaint does verify that the requesting node can properly communicate with a head node database. However, if you see an error resembling the one below, check your settings.ini contents and your network configuration:

Failed to connect to the ClusterWare service.  Please check that the
service is running and your base_url is set correctly in
/home/adminuser/.scyldcw/settings.ini or on the command line.

The connection URL and username can also be overridden for an individual program execution using the --base-url and --user options available for all scyld-* commands. The settings.ini file generated by scyld-install will contain a blank client.authpass variable. This is provided for convenience during installation, though for production clusters the system administrator will want to enforce authentication restrictions. See details in Securing the Cluster.

Updating Base Distribution Software

The decision about if and when to update RHEL-clone base distribution software is complex and needs to be made by a local cluster administrator, ranging from never updating anything on the cluster to updating frequently.

Production clusters in constant use commonly have regular update schedules, typically ranging from weekly to quarterly. The cluster administrator should track the RHEL-clone release notifications as well as the ClusterWare release notifications to determine which security fixes, bug fixes, and feature enhancements merit disrupting normal cluster operations in order to perform an update or a group of updates.

Updating Scyld ClusterWare Software

From time to time, Scyld will release updates and add-ons to Scyld ClusterWare. Customers on active support plans for Scyld software products can access these updates on the Penguin Computing website. Visit https://www.penguinsolutions.com/computing/support/technical-support/ for details. That website offers answers to common technical questions and provides access to application notes, software updates, and product documentation.

ClusterWare release versions follow the traditional three dot-separated numbers format: <major>.<minor>.<patch>. Updating to a newer major release should be done with care, especially updating ClusterWare 7 (or 6) to 12. See Scyld ClusterWare Migration Guide for details. Updating ClusterWare 11 to ClusterWare 12 requires an awareness of specific issues that are discussed later in this section.

The Release Notes contains brief notes about the latest release, and the Changelog & Known Issues provides a history of significant changes for each software release and a list of Known Issues And Workarounds.

Updating head nodes

The scyld-install tool is used to update Scyld ClusterWare software on a head node, just as it was used to perform the initial installation. This tool first determines if a newer clusterware-installer package is available, and if so will update clusterware-installer and then restart scyld-install.

Important

A simple yum update will not update Scyld ClusterWare packages on a head node, as the scyld-install tool has disabled /etc/yum.repos.d/clusterware.repo in order to prevent yum update from inadvertently updating Scyld ClusterWare. Instead, Penguin Computing strongly recommends using the scyld-install tool to perform updates of the basic Scyld ClusterWare packages that were originally installed by scyld-install. To install or update any optional Scyld ClusterWare packages described in Additional Software, you must use sudo yum <install-or-update>--enablerepo=scyld* <packages>.

Important

scyld-install uses the yum command to access Scyld ClusterWare and potentially various other repositories (e.g., Red Hat RHEL or CentOS) that by default normally reside on Internet websites. However, if the head node(s) do not have Internet access, then the required repositories must reside on local storage that is accessible by the head node(s). See Appendix: Creating Local Repositories without Internet.

Note

Executing scyld-install with no arguments presupposes that ClusterWare is not yet installed. If ClusterWare is currently installed, then the tool asks for positive confirmation that the user does intend to update existing software. You can avoid this interaction by providing the -u` or --update arg. That same degree of caution occurs if executing scyld-install --update on a server that does not currently have ClusterWare already installed: the tool asks for positive confirmation that the user does intend to install ClusterWare as a fresh install.

The scyld-install tool only updates basic ClusterWare head node software that was previously installed by the tool, plus any other dependency packages. After ClusterWare is updated, you can execute yum check-update --enablerepo=scyld* | grep scyld to view the optional ClusterWare packages that were previously installed using yum install --enablerepo=scyld*, and then use sudo yum update --enablerepo=scyld* <PACKAGES> to update (or not) as appropriate for your local head node.

You can also execute yum check-update to view the non-ClusterWare installed packages that have available updates, and then use sudo yum update <PACKAGES> to selectively update (or not) as appropriate for your local head node.

Alternatively, scyld-install --clear-all empties the database and clears the current installation. Just like during an initial installation, after a --clear-all the database should be primed with a cluster configuration. The cluster configuration can be loaded at the same time as the --clear-all using the --config /path/to/cluster-conf argument. This will use the scyld-cluster-conf tool to load the cluster configuration's initial declaration of private cluster interface, max number of nodes, starting IP address, and MAC address(es), as described in Execute the ClusterWare install script. For more details of the scyld-cluster-conf tool please refer to the Reference Guide.

Similar to using scyld-install on a non-ClusterWare server to perform a fresh install or to join another head node on an existing cluster, executing scyld-install --clear-all --config /path/to/cluster-conf> will invoke the scyld-add-boot-config script to create a new default boot image.

Updating compute nodes

A compute node can be dynamically updated using a simple yum update, which will use the local /etc/yum.repos.d/*repo file(s). If the compute node is executing a ClusterWare created image, then these changes (and any other changes) can be made persistent across reboots using scyld-modimg and performing the yum install and yum update operations inside the chroot. See Modifying PXEboot Images for details.

Updating ClusterWare 11 to ClusterWare 12

ClusterWare version 11 updates cleanly to version 12, albeit retaining the CW11-built boot configurations and images.

Important

A cluster using the ClusterWare Couchbase database must first switch that database to etcd. See Appendix: Switching Between Databases for details.

Important

You must examine /etc/yum.repos.d/clusterware.repo and potentially edit that file to reference the ClusterWare version 12 repos. If the baseurl= contains the string clusterware/11/, then change 11 to 12. If the gpgkey contains RPM-GPG-KEY-PenguinComputing, then change PenguinComputing to scyld-clusterware.

CW11-based compute nodes are compatible with CW12 parent head nodes. However, to make use of the full additional functionality of CW12, after updating the CW11 head node(s) you should also update CW11 images to CW12 with at least the newest version of clusterware-node. See Updating compute nodes, above.

Updating Base Distribution Software

Base distribution software updates on a schedule managed by the distributor (e.g., Red Hat). RHEL-clone versioning consists of two dot-separated numbers that define a major release and minor release. See Supported Distributions and Features for details about what Scyld ClusterWare supports on head nodes and compute nodes.

Various package releases can occur for a given major.minor release, and those patch releases are generally compatible with other software in the same major.minor release, which means a node can generally update patch release packages as desired. However, a kernel or device driver update may potentially require relinking of 3rd-party software. These patch releases typically include security fixes, bug fixes, and backward-compatible feature enhancements. See the distributors' documentation for details.

A minor release update generally entails a larger number of packages, and those packages need to be updated as a group in order to guarantee interoperability. Before updating to a new minor release the cluster administrator should confirm that the 3rd-party software intended to be in use is compatible with that particular major.minor base distribution. A minor release commonly includes a new kernel with substantial changes, and that commonly requires 3rd-party device software to be relinked. See the distributors' documentation for details.

A major release update always entails a large number of packages and usually changes of some degree to user and application interfaces. Commonly there is no simple updating from one major release to another, and an administrator commonly needs to perform a fresh install of the new distribution. Before updating to a new major release the cluster administrator should confirm that the intended 3rd-party software is supported by that major release. See the distributors' documentation for details.

Important

The Scyld ClusterWare build version (e.g., el7, el8, or el9, combined with x86_64 or aarch64) must match the base distribution's major release and hardware platform.

Updating Firmware

A cluster contains hardware components that employ writeable firmware, such as the BIOS on a server, device controller firmware, and "smart switch" firmware. The management of cluster firmware and the identification of a need to update that firmware is beyond the scope of this document. Contact Penguin Computing Support for guidance.

Services, Ports, Protocols

Apache

Apache serves the ClusterWare REST API via HTTP on port 80 using mod_wsgi through the httpd systemd service aliased as clusterware. HTTPS Encryption over port 443 can be enabled through standard Apache and operating system procedures. Apache is Open Source, and Penguin Computing contributes the REST API. The log files are /var/log/clusterware/api_access_log and /var/log/clusterware/api_error_log.

The ClusterWare GUI is also served through Apache from the /var/www/clusterware/front/ directory.

Chrony

Chrony is used to keep time synchronized across the cluster, including synchronization to upstream network time-servers and to all nodes within the cluster itself. The systemd service name is chronyd and it uses port 123 for its time-keeping communications, and port 323 for receiving commands from the chronyc management tool. This service is configured, started, and stopped by the ClusterWare service based on the cluster configuration. The configuration file is generated from a template located at /opt/scyld/clusterware-chrony/chrony.conf.template.

Couchbase (deprecated)

The replicated configuration key/value store Couchbase has the systemd service name couchbase-server. Log files are found in /opt/couchbase/var/lib/couchbase/logs/ but may also be accessed through the Couchbase console on port 8091 of any head node.

The Couchbase ports:

4369/tcp  # erlang port mapper (for Couchbase join)
8091/tcp  # Couchbase GUI and command interface (for Couchbase join)
8092/tcp  # Couchbase JSON interface (for Couchbase queries)
21101/tcp # otp port (for Couchbase join)
11209/tcp # memcached (for Couchbase data synchronization)
11210/tcp # Also used for Couchbase data synchronization

DHCP

DHCP provides dynamic host configuration, with a systemd service name clusterware-dhcpd and using port 68. The log file is var/log/clusterware/isc-dhcpd.log. This service is configured, started, and stopped by the ClusterWare service based on the cluster configuration. The configuration file is generated from a template located at /opt/scyld/clusterware-iscdhcp/dhcpd.conf.template.

DNS

DNS provides name- and ip-address-lookup services, with a systemd service name clusterware-dnsmasq and using port 53. This service is configured, started, and stopped by the ClusterWare service based on the cluster configuration. The configuration file is generated from a template located at /opt/scyld/clusterware-dnsmasq/dnsmasq.conf.template.

etcd

The replicated configuration key/value store etcd has the systemd service name clusterware-etcd. Log files are found in /var/log/clusterware/. etcd uses port 52380 to communicate with other head nodes.

iSCSI

iSCSI optionally serves root filesystems to compute nodes and uses port 3260. Serving root file systems via iSCSI is configured by the ClusterWare service using the targetcli command line tool.

OpenSSH

OpenSSH provides services to remotely execute programs and to transfer files, with a systemd service name sshd and using port 22. Encryption is SSH. The log file is /var/log/messages.

Telegraf / InfluxDB

Telegraf and InfluxDB communicate and store compute node performance data, with a systemd service name telegraf and using port 8094. Encryption is HTTPS. The log files are found in /var/log/telegraf/.

TFTP

The TFTP Server provides downloads for early iPXE boot files, with a systemd service name xinetd and using port 69. This service can be replaced by appropriate network card firmware. The log file is /var/log/messages.