Scyld Cloud Controller

The Scyld Cloud Controller is directly responsible for the creation and manipulation of VMs through OpenStack. This means defining VMs via Nova, creating their boot volumes in Cinder, and attaching public IP addresses as necessary for external access. The Cloud Controller maintains a mapping of users to VMs, and makes sure that only allowed users are able to access each VM.

The Cloud Controller also has APIs to manage users, groups, and storage volumes, though these are not always used depending on the deployment specifics. Each API resource is described in detail below.

server-instance

The server-instance resource is for manipulating VMs. In SCM, a server-instance is a VM, and a server-instance directly maps to a VM in OpenStack. Each VM defined in SCM receives a universally unique identifier (UUID) that uniquely identifies the VM.

The UUID used by SCM is not the same as the UUID used by OpenStack. This lets SCM present a persistent VM UUID to a user across multiple power state changes. When a VM is powered off via SCM, the OpenStack VM is actually deleted. All of the VM’s configuration information is stored on its boot disk, which is persistent. When the VM is powered back on via SCM, a brand new OpenStack VM is created, but booted from the pre-existing boot disk. In this way the VM appears persistent, but without SCM creating its own UUID the OpenStack UUID would change every time the VM is powered off.

server-flavor

The server-flavor resource retrieves the list of available flavors defined in OpenStack Nova. These flavors are listed in the Cloud Portal as instance-types, and represent the combination of RAM, CPU cores, and boot disk size that the VM will be allocated. This resource is read-only as presented by SCM. New flavors are not defined via the SCM API, but by OpenStack command-line interface (CLI) commands such as openstack flavor create, nova flavor-create, or through the OpenStack Horizon dashboard.

server-image

The server-image resource retrieves a list of available images defined in OpenStack Glance. More accurately, the API returns the list of images that are available for use by the querying user. It is possible to make images in Glance private to a specific user. When this is the case, the superuser sees all images all the time, but a regular user only sees the images that they are allowed access to. This is also a read-only API. New images are defined/uploaded directly using OpenStack CLI commands such as glance image-create, openstack image create, or through the OpenStack Horizon dashboard.

user

The user resource is used to establish a user account with the Cloud Controller. This is necessary in order to create VMs using the Cloud Controller because SCM uses OpenStack’s Keystone identity management system. When a user is created in the Cloud Controller, the Cloud Controller creates a user with the same name in Keystone.

Unless a user is using the APIs directly, user creation is a step that is automated by the Cloud Portal. The ultimate result of user creation within the Cloud Controller depends on the back-end user driver configured, which is chosen based on whether LDAP/AD integration is enabled, whether NIS is to be used to make users available to a cluster, or whether local user entries are to be made.

In the case of LDAP/AD integration, creating a user in the Cloud Controller only results in the creation of a user in Keystone. Additionally, the user is associated with the corresponding Cloud Auth user.

When local user creation is enabled, including NIS, creating a user in the Cloud Controller results in a call to useradd, such that the user is created in the appropriate /etc/passwd file. Using NIS, this entry can then be made globally available to other nodes, such as NFS storage servers, and ClusterWare head nodes and compute nodes.

group

The group resource is used to create groups in the local /etc/group file. This only makes sense when using the local user and NIS backends, and the group modification APIs are disabled when LDAP/AD integration is enabled.

The Cloud Controller only allows users in the same account owner domain to be in the same group. For example, if Account Owner A has Managed Users B and C, then only users A, B, and C can be added to the group. Account Owner D would not be allowed to be added to the group.

storage-volume

The storage-volume resource is designed to be used in conjunction with HPC clusters, such that a created storage-volume maps directly to a user’s $HOME directory within the cluster. If storage volumes for user $HOME directories do not need to be created or managed by SCM, this API can be disabled.

When enabled, having an active storage-volume is a prerequisite for VM creation. The rationale behind this is that a user cannot do any meaningful work without a $HOME directory. Storage-volume creation is often the most custom part of an SCM deployment. Many storage environments exist, and the steps needed to create, mount, and export a storage volume can vary greatly. Custom routines can be added to address these steps, and to add in support for things like setting quotas.

user-storage

Rather than individual storage volumes, SCM can manage a storage pool shared between all users of the cluster. In such a setup, the user’s storage quota can be set and usage recorded and summarized by the Scyld Cloud Accountant.

The Scyld Cloud Controller server will need to mount the storage pool.

Scyld Cloud Controller Setup

Configuration settings for the Scyld Cloud Controller are in the cloudcontroller.ini file. Important configuration settings are:

  • virtualization.boot_timeout - Time in seconds a VM has to boot into ‘ACTIVE’ state from a powered off state or creation before it is declared ‘ERROR’ and configuration is aborted. Defaults to 180.
  • virtualization.reboot_timeout - Time in seconds a VM has to boot into ‘ACTIVE’ state from a powered off state or creation before it is declared ‘ERROR’ and configuration is aborted. Reboots are generally quicker than a first boot. Defaults to 240.
  • virtualization.auto_assign_public_ip - This boolean flag controls whether SCM will assign a floating IP to the VM once it leaves the ‘BOOT’ state. Defaults to True.
  • virtualization.flavor_caps - A dictionary of per-user limits on flavors. Example: {“1”: 1} means each user can run 1 server of flavor ID 1.
  • virtualization.hostname_prefix - The prefix added to the VM’s shortened SCC UUID to create the hostname. Defaults to ‘vm’.
  • virtualization.hostname_uuid_uniq_chars - How many characters of the VM’s SCC UUID to use when creating the hostname. Defaults to 7.
  • virtualization.image_config_routines - A map of image IDs to Python classes to perform additional VM configuration after booting.
  • openstack.allow_rdp - Boolean flag controlling whether to add security group rules allowing RDP (TCP/UDP Port 3389) to the VM. Defaults to False.
  • openstack.autodelete_bootvol - This boolean flag controls whether the Cinder volume used to boot a VM is automatically deleted when a VM is deleted. Defaults to False.
  • openstack.compute_node_ranges - Comma seperated list of CIDR ranges to allow interactive qsub traffic from (TCP Port 1024-65535). Example: 10.20.32.0/23,172.16.0.0/16
  • openstack.floating_ip_pool_name - The name of the public OpenStack network used to create floating IPs.
  • openstack.grow_instances - Boolean flag controlling whether restarting stopped VMs will grow their root volume to match the VM’s flavor’s current disk size. Defaults to False.
  • openstack.private_network_name - The name of the OpenStack network to be used as the primary private network.
  • openstack.secondary_networks - The name of additional OpenStack networks to be added to the VM as additional interfaces.
  • storage.[mkdir/lustre].options - The octal file permission for newly created user storage directories. Defaults to ‘750’.
  • storage.[mkdir/lustre].root - The base directory where per-user storage directories will be created.
  • storage.[mkdir/lustre].skel - The location of skeleton files to be copied to all user storage directories.
  • storage.lustre.default_quota - The default Lustre quota for user storage in MB. Defaults to 0.
  • storage.lustre.max_quota - The maximum Lustre quota in MB. Defaults to 10e6.
  • storage.lustre.timeout - The timeout for Lustre operations. Defaults to ‘3s’.
  • storage.nfs.servers - For storage drivers that reach out to NFS servers, this is a comma separated list of their IP addresses.
  • storage.nfs.use_exports_d - Boolean flag. When True, NFS exports are created as seperate files in /etc/exports.d/. Otherwise, NFS exports are handled by the NFS storage agent modifying /etc/exports. Defaults to True.
  • storage.nfs.export.default_options - For NFS exports created by the storage driver, this is the default set of export options. Defaults to ‘rw,no_subtree_check’.
  • storage.nfs.export.auto_exports - A dictionary of CIDR:exportoption pairs that serve as a list of exports that should be created for every storage-volume. For example, exporting a newly created $HOME to cluster compute nodes.
  • storage.nfs.export.auto_mounts - A dictionary of mountpoint:mountoption pairs that should be mounted automatically in every VM created by SCM.
  • storage.nfs.export.auto_clients - A list of IP addresses that identify nodes that should automatically mount newly created exports. This is typically used for nodes that are not login nodes.
  • userauth.manage_sshd_acls - This boolean flag causes AllowUsers lines to be written to the VM’s /etc/ssh/sshd_config file when the VM starts. If the instance image’s property is_linux is True (the default), then one line is written for each user account that’s permitted to access the instance, and the VM’s ssh server is restarted. Defaults to True.
  • userauth.autocreate_storage - Boolean flag, if True a storage volume will be created during user creation. Defaults to False.

User Storage Usage Recording

If the shared pool user storage driver is enabled, per-user usage needs to be recorded by using the import script after activating the cloud controller’s Python environment.

# source /var/www/wsgi/cloudcontroller/env/bin/activate
# record_user_storage /var/www/wsgi/cloudcontroller/cloudcontroller.ini

Alternatively, if the storage driver is disabled, the cloud controller can still records the user storage usage of a storage pool by passing the pool’s mount path directly.

# record_user_storage /var/www/wsgi/cloudcontroller/cloudcontroller.ini lustre /mnt/lustre

To have this operation performed regularly, this action is usually handled as part of a cron script.