Appendix: Using SingularityΒΆ
Singularity is available in Scyld ClusterWare by installing the singularity-scyld RPM, which is built from source developed by Sylabs Inc., or by installing the singularity RPM found in the EPEL yum repository. See https://www.sylabs.io/docs for their extensive documentation.
The following example creates a Singularity container openmpi.sif
containing openmpi3.1,
and placing that container in a bootable image.
First create the openmpi.def
Singularity definition file,
then use that file to create the container:
# Use quoted "EOF" for bash to avoid % and $ expansions; just EOF for sh.
cat <<-"EOF" >openmpi.def
BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum
%files
/etc/yum.repos.d/clusterware.repo /etc/yum.repos.d/clusterware.repo
%environment
PATH=/opt/scyld/openmpi/3.1.3/gnu/bin:$PATH
LD_LIBRARY_PATH=/opt/scyld/openmpi/3.1.3/gnu/lib:/opt/scyld/slurm/lib64:$LD_LIBRARY_PATH
MPI_HOME=/opt/scyld/openmpi/3.1.3/gnu
MPI_LIB=/opt/scyld/openmpi/3.1.3/gnu/lib
MPI_INCLUDE=/opt/scyld/openmpi/3.1.3/gnu/include
MPI_SYSCONFIG=/opt/scyld/openmpi/3.1.3/gnu/etc
%post
# IMPORTANT:
# If instead using "OSVersion: 6" instead of "OSVersion 7" above,
# then for any subsequent `rpm` or `yum`, add:
# rpm --rebuilddb
echo "Installing openmpi3.1-gnu rpm"
yum -y install openmpi3.1-gnu
exit 0
EOF
# Create the Singularity chroot "/tmp/openmpi" in which updates can be made.
sudo singularity build --sandbox /tmp/openmpi openmpi.def
# Make a sample update: build an openmpi test program inside the chroot.
sudo singularity exec -w /tmp/openmpi \
mpicc -o /usr/bin/ring /opt/scyld/openmpi/3.1.3/gnu/examples/ring_c.c
# Finalize the sandbox chroot into the Singularity container "openmpi.sif".
sudo singularity build openmpi.sif /tmp/openmpi
Create a bootable image that hosts the Singularity container and can execute openmpi applications:
# Clone a new image instead of modifying an existing image.
scyld-imgctl -i DefaultImage clone name=SingularityImage
# Install needed packages inside the new image.
scyld-modimg -i SingularityImage --freshen --overwrite --no-discard \
--install singularity-scyld,openmpi3.1-gnu --upload
# Now get into the chroot of the Singularity image.
scyld-modimg -i SingularityImage --chroot --overwrite --upload --no-discard
# Inside the root, add your userid (e.g., "myuserid") if necessary, which
# creates a /home/myuserid/ directory, and import the Singularity container file.
useradd myuserid
scp myuserid@localhost:/home/myuserid/openmpi.sif /home/myuserid/
exit
Boot nodes n0 and n1 with SingularityImage:
scyld-bootctl -i DefaultBoot clone name=SingularityBoot
scyld-bootctl -i SingularityBoot update image=SingularityImage
scyld-nodectl -i n[0-1] set _boot_config=SingularityBoot
# Now reboot nodes n0 and n1
scyld-nodectl -i n[0-1] reboot
When the nodes are up, then initialize passphrase-less key-based access, as described in OpenMPI, MPICH, and/or MVAPICH under Additional Software.
Now you can run the ring
program from n0 (or n1):
# logged into n0, or using a job scheduler
mpirun -np 2 --host n0,n1 singularity exec openmpi.sif /usr/bin/ring
Or from the head node:
# If not already installed
sudo yum install singularity-scyld openmpi3.1-gnu --enablerepo=scyld*
module load openmpi/gnu/3.1.3
mpirun -np 2 --host n0,n1 singularity exec openmpi.sif /usr/bin/ring