Securing the Cluster

This Installation & Administrator Guide section discusses cluster security issues that are exclusive to Scyld ClusterWare. We assume that the cluster administrator is familiar with security issues that are not solely related to ClusterWare, such as securing the cluster from outside access, optionally enabling various Red Hat RHEL/CentOS functionalities for logging and auditing access to nodes and storage and for managing SELinux.

Authentication

The cluster administrator authentication method is controlled in the /opt/scyld/clusterware/conf/base.ini file by the plugins.auth variable and is initially set to "dummy". This plugin is the least secure and accepts any password for a known administrator, providing very little security. The initial list of known administrators is stored in the same file in the auth.tmpadmins variable. The scyld-install installation will (unless passed the --no-tools argument) add the current user to that comma separated list of user names.

Any administrator can add additional administrators through the scyld-adminctl command whose arguments match the other scyld-*ctl commands as described in the Reference Guide. We suggest that administrators add accounts for themselves through this tool, and thereafter clear the auth.tmpadmins variable. This variable is only intended to be used during early installation, for small experimental clusters, or when recovering from some sort of failure.

When deploying ClusterWare, the plugins.auth variable will be set to "appauth". This plugin executes the command defined in the appauth.app_path variable as user root. The default implementation of that command is provided by /opt/scyld/clusterware/bin/pam_authenticator. This implementation interfaces with the PAM authentication system using the /etc/pam.d/cw_check_user configuration file. The contents of this file initially use local system authentication, although this can be modified to authenticate against any mechanism available through the PAM system. Please see PAM documentation provided by your distro as well as the main PAM project. See the Red Hat https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system-level_authentication_guide/pluggable_authentication_modules documentation.

Administrators can provide authentication methods beyond PAM by implementing a script or application and providing it via the appauth.app_path variable. Appropriate applications should start with no arguments, read a username and password separated by a newline from stdin , and reply with either yes or no followed by a newline on stdout. For example, a test run of pam_authenticator looks like:

[example@head ~] sudo /opt/scyld/clusterware/bin/pam_authenticator
tester
not_the_password
no

Role-Based Access Controls

Note

ClusterWare supports the Keycloak authentication system that may be helpful in linking to existing enterprise identity management systems. For more info, see Appendix: Integrating Keycloak with ClusterWare for RBAC

Prior to version 12.2.0, ClusterWare only had the concept of an "admin" who could perform any action on any of the ClusterWare database entries. ClusterWare now has Role-Based Access Controls (RBAC) to allow different "classes" of administrators, some of whom may have signicantly reduced sets of actions that they may perform. There are 6 system-provided Roles plus a "No Access" pseudo-role, described below. A given admin user can be assigned one or more Roles, with each Role granting a specific set of Permissions (roles are additive; if one role grants a permission, then, in general, another role cannot remove it).

By default, the first admin user added to the system will be assigned to the "Full Admin" Role and thereby gain permission to take any action within the system. When creating additional admin users, by default, they will inherit the same roles as their creator. This means that a Full Admin will create other Full Admins - the same as the default on prior versions of ClusterWare.

The current roles established in ClusterWare are:

  • Authenticated User

    • Any admin account will have this role which simply grants read-only access to the system. Auth-Users can read nearly any object in the ClusterWare database, with the exception of reserved attributes. This role may be useful for admins who need to report on cluster status and configuration, or perhaps for "power users" who may have some knowledge of cluster operations but should not be able to modify or change the configuration.

  • Onsite Engineer

    • The Onsite Engineer role is aimed at technical staff who may need to interact with the machines physically, e.g. to power on/off the machines, or to rack/unrack the machines. In addition to the “read” permissions from Auth-User, the Onsite Engineer is able to issue power-control commands through scyld-nodectl, and to write/update node information.

  • Imaging Engineer

    • The Imaging Engineer is primarily responsible for the creation of new images or modification of existing ones. To better assist them in that role, they are also given permissions to read reserved attributes (to see what nodes are booting into what images).

  • Production Engineer

    • For most day-to-day operations, the Production Engineer role might be the most useful. It grants several permissions: to update/modify nodes, including power-control functionality; to update regular and reserved attributes; to create/modify images and boot configs; to create/modify naming pools; and to create/modify git repositories.

  • Manager

    • In addition to the Auth-User (read-only) permissions, a Manager can create new admins in the system and view reserved attributes. This role may be useful in larger organizations where the creation and removal of admin accounts could be managed by less technical staff.

  • Full Admin

    • The Full Admin role grants all possible permissions to the admin so they can read/write any object, do full power-control on nodes, update, or delete any object, etc.

    • Permissions that are only granted to Full Admins: control over distros and repos; control over dynamic groups and state sets; control over head nodes; control over hostnames and networks; and control over the cluster configuration itself.

  • No Access

    • While not a "role" per se, ClusterWare uses the No Access pseudo-role to enforce a block on an account. If an admin has the No Access role, then they will not be able to perform any actions, regardless of any other role they may have.

    • It is certainly best practice to remove such blocked accounts from ClusterWare entirely, and to also remove them from any authentication system that feeds ClusterWare (e.g. Keycloak). However, it is sometimes helpful to have "helper" accounts that can be enabled and disabled quickly. E.g. in larger sites, there may be contractors or vendor support staff that need access to a system to help with a problem; but when the problem is resolved, those accounts can be deactivated with the "No Access" role but left in-place in case another support event arises later.

To assign roles when a new account is created, use the scyld-adminctl tool with a comma-separated list of roles:

scyld-adminctl create name=charlie roles=ImagingEngineer,OnsiteEngineer

To assign roles after an account has been created, simply update the admin record with the new list of roles:

scyld-adminctl -isally update roles=Manager

Note

For organizations not wishing to utilize RBAC, simply assign every admin user to the "Full Admin" Role; this is the default and will then mimic the behavior of previous ClusterWare versions.

For more information, including a full description of the Roles and Permissions, see Role-Based Access Control System.

Changing the Database Password

The scyld-install installation configures the ClusterWare database with a randomly generated password. This password is used when joining a new head node to the cluster and must be provided either through a command line or on request during the installation of the new head node. This password is stored in the database.admin_pass variable in the /opt/scyld/clusterware/conf/base.ini file. The details of changing this password depend on the specific database the cluster is using.

Important

Once this password is changed within the database, change the database.admin_pass variable in base.ini and restart the clusterware service on each head node.

etcd

Use the etcdctl tool (provided in the clusterware-etcd package) to change the etcd password:

/opt/scyld/clusterware-etcd/bin/etcdctl user passwd root

Compute Node Remote Access

By default, remote access to compute nodes is provided through SSH using key-based authentication, although administrators may also enable password-based SSH in the compute node image by configuring a password for the root user. Every head node generates a public/private key pair and places these files in directory /opt/scyld/clusterware/.ssh/ using the names id_rsa.clusterware and id_rsa.clusterware.pub. These keys are used by the head nodes to execute commands on the compute nodes. All head node public keys are downloaded by compute nodes at boot time by the update_keys.sh script and appended to /root/.ssh/authorized_keys. This allows any head node to execute a command on any compute node. The /opt/scyld/clusterware/.ssh/id_rsa.clusterware key can be used by system administrators as an "automation" key for tasks like cron jobs. It is also useful in recovery situations where an administrator may need to use this private key to directly access compute nodes

This same script that downloads the head node public keys will also download the public keys attached to every cluster administrator account. These accounts are created using the scyld-adminctl tool as follows:

scyld-adminctl create name=admin keys=@~/.ssh/id_rsa.pub

This would allow anyone with the corresponding id_rsa to SSH into the root account on any compute node booted after the key was added. The key can also be added as a string or updated for an existing administrator. For example,

scyld-adminctl -i admin update keys='ssh-rsa AAAAB3NzaC1yc2EAAAADA....'

Cluster administrators are also welcome to add SSH keys to compute node images in small private clusters, although adding administrator accounts with public keys simplifies management of larger clusters with multiple node images or cluster administrators. Note that administrator accounts stored in the database or listed in the base.ini both use the same authentication mechanisms described in the previous section.

Important

We urge cluster administrators to create their own administrator accounts and remove their usernames from the base.ini file after cluster installation.

Compute Node Host Keys

In most computer systems the SSH sshd daemon uses unique host keys to identify itself to clients, and host keys are not created during image creation. This means that each compute node will generate its own host keys during boot. Since the compute node changes are discarded on reboot, a new set of keys will be generated with each boot.

In an appropriately protected cluster, some administrators prefer for all compute nodes to share host keys. This can be achieved by storing host keys in the compute node image. For example, to generate host keys and repack the DefaultImage, an administrator can run:

scyld-modimg -i DefaultImage --exec sshd-keygen --overwrite --upload

All nodes that boot using this image after this change will use identical host keys, so ideally you should reboot the nodes with each node's updated image. To remove the host keys from an image, an administrator needs to delete the /etc/ssh/ssh_host_* files from the compute node image.

Encrypting Communications

By default the administrative tools communicate with the head node via HTTP, although they can also use HTTPS if appropriate certificates are configured on the head node's Apache web server. Please refer to documentation provided by your distro about how to properly enable HTTPS on the Apache server. Apache configuration files are located in /opt/scyld/clusterware/conf/httpd/. The Apache VirtualHost definition can be found in vhost.conf, and the proxy definition in that file will need to be included into the HTTPS VirtualHost.

Once HTTPS is enabled, the ~/.scyldcw/settings.ini file of any existing ClusterWare tool installation should be updated. In that file the protocol of the client.base_url variable will need to be updated. It should be safe to leave HTTP enabled for localhost-only access, and in that case local tool installations can continue to use the original localhost-based URL.

Security-Enhanced Linux (SELinux)

Security-Enhanced Linux (SELinux) is a set of patches to the Linux kernel and various utilities that provide mandatory access control to major subsystems of a node. See https://en.wikipedia.org/wiki/Security-Enhanced_Linux for general discussion of SELinux.

ClusterWare supports SELinux on the head nodes and compute nodes.

SELinux On Compute Nodes

For Red Hat RHEL and CentOS compute nodes, the root file systems created by the scyld-modimg tool include SELinux support as part of the installation of the @core yum group. During the boot process the mount_rootfs script will, like the standard dracut based initramfs, load the SELinux policy before switching root. Note that the default cmdline in the boot configurations created through scyld-add-boot-config (including the DefaultBoot configuration) will contain enforcing=0, thereby placing all compute nodes in SELinux "permissive" mode. Only remove this option once you have completed testing to confirm that your applications will run as expected with SELinux in "enforcing" mode.

SELinux on compute nodes may be disabled in the standard ways through command line arguments or by changing the contents of the node's /etc/selinux/config configuration file. For details please refer to appropriate distro-provided documentation.

In addition to the default "targeted" SELinux policy provided by RHEL and CentOS, ClusterWare also supports the Multi-Level Security (MLS) policy for compute nodes. Enabling the MLS policy inside an image is done the same way as it would be done on a locally installed system. After entering the image chroot using scyld-modimg, first install the selinux-policy-mls package, and then modify the /etc/selinux/config file to reference the newly installed policy. Because the clusterware-node SELinux policy module is installed at image creation time, it may need to be re-installed after switching to the MLS policy:

semodule --install /opt/scyld/clusterware-node/clusterware-node.pp.bz2

The semodule command can also be used to check if the policy is loaded:

semodule --list | grep clusterware

When exiting the chroot, ClusterWare automatically relabels the file system based on the policy referenced in /etc/selinux/config.

Important

Fully configuring a cluster for MLS requires significant effort, including labeling objects on shared storage and defining additional policy around user workflows and tools. Please refer to your operating system documentation, as such details are beyond the scope of this document. Note that Scyld ClusterWare-provided schedulers, MPI implementations, and 3rd party applications may need additional custom permissions not covered here in order to configure a functional MLS cluster.

When creating boot configuration for an MLS enabled image, please be aware that the MLS policy, by default, does not allow the root user to log into the compute node via ssh. Because ssh is used by the ClusterWare soft power commands, please either enable the root login functionality or use the _remote_user node attribute to configure login as a user with sudo shutdown permission. The root login permission can be enabled through the setsebool command, and the boolean is named ssh_sysadm_login.

SELinux On Head Nodes

On head nodes, SELinux is detected to be in "enforcing" mode at both installation and service run time. To switch SELinux from "enforcing" to "permissive" mode, please see the documentation for your operating system. If this switch is made while the ClusterWare service is running, please restart that service:

sudo systemctl restart clusterware

MLS Policy On Head Nodes

For head nodes enforcing the MLS policy, the SELinux user sysadm_u should be used to install ClusterWare and run administrative tools.

To map a Linux user to the sysadm_u SELinux user, you can run:

sudo semanage login --add linux_user --seuser sysadm_u

By default, the sysadm_u user should run with the sysadm_t domain.

Security Technical Implementation Guides (STIG)

STIG security hardening implements compliance with the Defense Information Systems Agency (DISA) guidelines described in the Security Technical Implementation Guides (STIGs) (https://csrc.nist.gov/glossary/term/security_technical_implementation_guide). Certain high-security clusters may require STIG compliance.

ClusterWare provides basic STIG support for kickstarted nodes by adding the following snippet to your kickstart *.ks file:

%addon org_fedora_oscap
    content-type = scap-security-guide
    profile = xccdf_org.ssgproject.content_profile_stig
%end

To configure a STIG head node, add the snippet to your kickstart config file and reboot the node using that *.ks file to enable STIG. Then install ClusterWare (Installation and Upgrade of Scyld ClusterWare) on the STIG-enabled node in the usual way.

ClusterWare provides an example file /opt/scyld/clusterware/kickstarts/basic-stig.ks with that snippet appended for administrators who would like to kickstart infrastructure nodes or additional head nodes with that STIG applied at install time.