Appendix: Using Singularity

Singularity is available in Scyld ClusterWare by installing the singularity-scyld RPM, which is built from source developed by Sylabs Inc., or by installing the singularity RPM found in the EPEL yum repository. See https://www.sylabs.io/docs for their extensive documentation.

The following example creates a Singularity container openmpi.sif containing openmpi3.1, and placing that container in a bootable image.

First create the openmpi.def Singularity definition file, then use that file to create the container:

# Use quoted "EOF" for bash to avoid % and $ expansions; just EOF for sh.
cat <<-"EOF" >openmpi.def
BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum

%files
    /etc/yum.repos.d/clusterware.repo /etc/yum.repos.d/clusterware.repo

%environment
    PATH=/opt/scyld/openmpi/3.1.3/gnu/bin:$PATH
    LD_LIBRARY_PATH=/opt/scyld/openmpi/3.1.3/gnu/lib:/opt/scyld/slurm/lib64:$LD_LIBRARY_PATH
    MPI_HOME=/opt/scyld/openmpi/3.1.3/gnu
    MPI_LIB=/opt/scyld/openmpi/3.1.3/gnu/lib
    MPI_INCLUDE=/opt/scyld/openmpi/3.1.3/gnu/include
    MPI_SYSCONFIG=/opt/scyld/openmpi/3.1.3/gnu/etc

%post
    # IMPORTANT:
    #   If instead using "OSVersion: 6" instead of "OSVersion 7" above,
    #   then for any subsequent `rpm` or `yum`, add:
    # rpm --rebuilddb
    echo "Installing openmpi3.1-gnu rpm"
    yum -y install openmpi3.1-gnu
    exit 0
EOF

# Create the Singularity chroot "/tmp/openmpi" in which updates can be made.
sudo singularity build --sandbox /tmp/openmpi openmpi.def

# Make a sample update: build an openmpi test program inside the chroot.
sudo singularity exec -w /tmp/openmpi \
     mpicc -o /usr/bin/ring /opt/scyld/openmpi/3.1.3/gnu/examples/ring_c.c

# Finalize the sandbox chroot into the Singularity container "openmpi.sif".
sudo singularity build openmpi.sif /tmp/openmpi

Create a bootable image that hosts the Singularity container and can execute openmpi applications:

# Clone a new image instead of modifying an existing image.
scyld-imgctl -i DefaultImage clone name=SingularityImage

# Install needed packages inside the new image.
scyld-modimg -i SingularityImage --freshen --overwrite --no-discard \
             --install singularity-scyld,openmpi3.1-gnu --upload

# Now get into the chroot of the Singularity image.
scyld-modimg -i SingularityImage --chroot --overwrite --upload --no-discard

# Inside the root, add your userid (e.g., "myuserid") if necessary, which
#   creates a /home/myuserid/ directory, and import the Singularity container file.
useradd myuserid
scp myuserid@localhost:/home/myuserid/openmpi.sif /home/myuserid/
exit

Boot nodes n0 and n1 with SingularityImage:

scyld-bootctl -i DefaultBoot clone name=SingularityBoot
scyld-bootctl -i SingularityBoot update image=SingularityImage
scyld-nodectl -i n[0-1] set _boot_config=SingularityBoot
# Now reboot nodes n0 and n1
scyld-nodectl -i n[0-1] reboot

When the nodes are up, then initialize passphrase-less key-based access, as described in OpenMPI, MPICH, and/or MVAPICH under Additional Software.

Now you can run the ring program from n0 (or n1):

# logged into n0, or using a job scheduler
mpirun -np 2 --host n0,n1 singularity exec openmpi.sif /usr/bin/ring

Or from the head node:

# If not already installed
sudo yum install singularity-scyld openmpi3.1-gnu --enablerepo=scyld*

module load openmpi/gnu/3.1.3
mpirun -np 2 --host n0,n1 singularity exec openmpi.sif /usr/bin/ring