Welcome to Scyld ClusterWare’s documentation!

• Release Notes
  • Release Notes: Scyld ClusterWare Release v7.9.19-7919g0000
    • About This Release
      • Important: Recommend using /usr/sbin/install-scyld script
      • Important for clusters using 3rd-party drivers or applications
      • Important for clusters using Panasas storage
    • First Installation of Scyld ClusterWare 7 On A Server
    • Upgrading An Earlier Release of Scyld ClusterWare 7 to 7.9
    • Post-Installation Configuration Issues
      • Resolve *.rpmnew and *.rpmsave configuration file differences
      • Disable SELinux and NetworkManager
      • Edit /etc/beowulf/conf.d/sysctl.conf as needed
      • Optionally reduce size of /usr/lib/locale/locale-archive
      • Optionally configure and enable compute node CPU speed/power management
      • Optionally install a different TORQUE package
      • Optionally enable job manager
      • Optionally enable TORQUE scheduler
      • Optionally enable Ganglia monitoring tool
      • Optionally enable NFS locking
      • Optionally adjust the size limit for locked memory
      • Optionally increase the max number of processes per user
      • Optionally enable SSHD on compute nodes
      • Optionally allow IP Forwarding
      • Optionally increase the nf_conntrack table size
      • Optionally configure vm.zone_reclaim_mode on compute nodes
      • Optionally configure automount on compute nodes
      • Optionally reconfigure node names
    • Post-Installation Configuration Issues For Large Clusters
      • Optionally increase the number of nfsd threads
      • Optionally increase the max number of processID values
      • Optionally increase the max number of open files
      • Issues with Ganglia
    • Post-Installation Release of Updated Packages
    • Notable Feature Enhancements And Bug Fixes
      • v7.9.19 - March 17, 2023
      • v7.9.18 - February 6, 2023
      • v7.9.17 - November 17, 2022
      • v7.9.16 - August 19, 2022
      • v7.9.15 - July 8, 2022
      • v7.9.14 - June 1, 2022
      • v7.9.13 - April 13, 2022
      • v7.9.12 - March 2, 2022
      • v7.9.11 - January 20, 2022
      • v7.9.10 - December 6, 2021
      • v7.9.9 - October 20, 2021
      • v7.9.8 - September 15, 2021
      • v7.9.7 - July 27, 2021
      • v7.9.6 - June 21, 2021
      • v7.9.5 - May 26, 2021
      • v7.9.4 - April 27, 2021
      • v7.9.3 - April 7, 2021
      • v7.9.2 - February 11, 2021
      • v7.9.1 - January 6, 2021
      • v7.9.0 - December 9, 2020
      • v7.8.5 - September 22, 2020
      • v7.8.4 - August 5, 2020
      • v7.8.3 - July 7, 2020
      • v7.8.2 - June 17, 2020
      • v7.8.1 - May 27, 2020
      • v7.8.0 - May 6, 2020
      • v7.7.5 - April 1, 2020
      • v7.7.4 - February 20, 2020
      • v7.7.3 - December 13, 2019
      • v7.7.2 - November 27, 2019
      • v7.7.1 - October 18, 2019
      • v7.7.0 - September 24, 2019
      • v7.6.4 - August 16, 2019
      • v7.6.3 - July 10, 2019
      • v7.6.2 - June 11, 2019
      • v7.6.1 - April 8, 2019
      • v7.6.0 - December 14, 2018
      • v7.5.4 - October 2, 2018
      • v7.5.3 - July 27, 2018
      • v7.5.2 - July 14, 2018
      • v7.5.1 - June 4, 2018
      • v7.5.0 - June 4, 2018
      • v7.4.6 - October 3, 2018
      • v7.4.5 - March 23, 2018
      • v7.4.4 - February 8, 2018
      • v7.4.3 - January 11, 2018
      • v7.4.2 - December 26, 2017
      • v7.4.1 - October 31, 2017
      • v7.4.0 - October 11, 2017
      • v7.4.0 - October 6, 2017
      • v7.3.7 - October 13, 2017
      • v7.3.6 - August 7, 2017
      • v7.3.5 - June 28, 2017
      • v7.3.4 - June 16, 2017
      • v7.3.3 - April 27, 2017
      • v7.3.2 - April 3, 2017
      • v7.3.1 - March 8, 2017
      • v7.3.0 - January 20, 2017
      • v7.2.0 - November 14, 2016
    • Known Issues And Workarounds
      • Issues with bootmodule firmware
      • Kernel panic using non-invpcid old Intel nodes
      • Managing environment modules .version files
      • Installing and managing concurrent versions of packages
      • Issues with OpenMPI
      • Issues with Scyld ClusterWare process migration in heterogeneous clusters
      • Issues with MVAPICH2 and mpirun_rsh or mpispawn
      • Issues with Singularity
      • Issues with ptrace
      • Issues with IP Forwarding
      • Issues with kernel modules
      • Issues with port numbers
      • Issues with Spanning Tree Protocol and portfast
      • Issues with Gdk
      • Caution when modifying Scyld ClusterWare scripts
      • Caution using tools that modify config files touched by Scyld ClusterWare
      • Running nscd service on master node may cause kickbackdaemon to misbehave
      • Beofdisk does not support local disks without partition tables
      • Issues with bproc and the getpid() syscall
• Installation Guide
  • Preface
  • Scyld ClusterWare System Overview
  • Recommended Components
  • Assembling the Cluster
  • Software Components
  • Quick Start Installation
    • Introduction
    • Network Interface Configuration
      • Cluster Public Network Interface
      • Cluster Private Network Interface
    • Network Security Settings
    • Red Hat RHEL7 or CentOS7 Installation
    • Scyld ClusterWare Installation
      • Configure Yum To Support ClusterWare
      • Install ClusterWare
    • Trusted Devices
    • Compute Nodes
  • Detailed Installation Instructions
    • Red Hat Installation Specifics
      • Network Interface Configuration
      • Network Security Settings
      • Package Group Selection
    • Updating Red Hat or CentOs Installation
      • Updating Using Yum
      • Updating Using Media
    • Scyld ClusterWare Installation
      • Configure Yum To Support ClusterWare
      • Install ClusterWare
    • Trusted Devices
    • Enabling Access to External License Servers
    • Post-Installation Configuration
    • Scyld ClusterWare Updates
      • Updating ClusterWare
  • Cluster Verification Procedures
    • Monitoring Cluster Status
      • bpstat
      • BeoStatus
    • Running Jobs Across the Cluster
      • Directed Execution with bpsh
      • Dynamic Execution with beorun and mpprun
  • Troubleshooting ClusterWare
    • Failing PXE Network Boot
    • Mixed Uni-Processor and SMP Cluster Nodes
    • IP Forwarding
    • SSH Traffic
    • Device Driver Updates
    • Finding Further Information
  • Compute Node Disk Partitioning
    • Architectural Overview
    • Operational Overview
    • Disk Partitioning Procedures
      • Typical Partitioning
      • Default Partitioning
      • Generalized, User-Specified Partitions
      • Unique Partitions
    • Mapping Compute Node Partitions
  • Changes to Configuration Files
    • Changes to Red Hat Configuration Files
    • Possible Changes to ClusterWare Configuration Files
• Eval Installation Guide
  • Evaluation Installation Instructions
  • Scyld ClusterWare System Overview
  • Hardware Requirements
  • BIOS Requirements
  • Software Requirements
  • Install Yum Configuration File and License File
  • Update Base Distribution and Install Scyld Clusterware Software
  • Configure the Private and Public Networks
  • Start Cluster Operations
  • Documentation and Support
  • Purchasing Scyld ClusterWare
  • Uninstalling Scyld ClusterWare
• Administrator’s Guide
  • Preface
  • Scyld ClusterWare Design Overview
    • System Architecture
      • System Hardware Context
      • Network Topologies
      • System Data Flow
      • System Software Context
      • System Level Files
    • Technical Description
      • Compute Node Boot Procedure
      • BProc Distributed Process Space
      • Compute Node Categories
      • Compute Node States
      • Miscellaneous Components
    • Software Components
      • BeoBoot Tools
      • BProc Daemons
      • BProc Clients
      • ClusterWare Utilities
  • Monitoring the Status of the Cluster
    • Monitoring Utilities
      • Cluster Monitoring Interfaces
      • Monitoring Daemons
    • Using the Data
    • beostatus
      • beostatus File Menu
      • beostatus Modes
    • beostat
    • bpstat
    • Ganglia
    • beoweb
  • Configuring the Cluster
    • Configuring the Cluster Manually
      • Configuration Files
      • Command Line Tools
      • Configuring CPU speed/power for Compute Nodes
      • Adding New Kernel Modules
      • Accessing External License Servers
      • Configuring SSH for Remote Job Execution
      • Interconnects
  • Remote Administration and Monitoring
    • Command Line Tools
    • X Forwarding
  • Managing Users on the Cluster
    • Managing User Accounts
      • Adding New Users
      • Removing Users
    • Managing User Groups
      • Creating a Group
      • Adding a User to a Group
      • Removing a Group
    • Controlling Access to Cluster Resources
      • What Node Ownership Means
      • Checking Node Ownership
      • Setting Node Ownership
  • Job Batching
    • Enabling TORQUE or Slurm
  • Managing Non-Scyld Nodes
    • DHCP IP address assignment to devices
    • Simple provisioning using PXE
    • Simple provisioning using the class directive
    • Booting a node from the local harddrive
    • Provisioning a non-Scyld node
    • Integrating a non-Scyld node into the cluster
  • Managing Multiple Master Nodes
    • Active-Passive Masters
    • Active-Active Masters
  • Managing Node Failures
    • Protecting an Application from Node Failure
    • Compute Node Failure
      • When Compute Nodes Fail
      • Compute Node Data
    • Master Node Failure
      • When Master Nodes Fail - Without Run-to-Completion
      • When Master Nodes Fail - With Run-to-Completion
  • Compute Node Boot Options
    • Compute Node Boot Media
      • PXE
      • Local Disk
      • Linux BIOS
      • Flash Disk
    • Changing Boot Settings
      • Adding Steps to the node_up Script
      • Per-Node Parameters
      • Other Per-Node Config Options
    • Error Logs
  • Disk Partitioning
    • Disk Partitioning Concepts
    • Disk Partitioning with ClusterWare
      • Master Node
      • Compute Nodes
    • Default Partitioning
      • Master Node
      • Compute Nodes
    • Partitioning Scenarios
      • Applying the Default Partitioning
      • Specifying Manual Partitioning
  • File Systems
    • File Systems on a Cluster
      • Local File Systems
      • Network/Cluster File Systems
    • NFS
      • NFS on Clusters
      • Configuration of NFS
    • ROMIO
      • Reasons to Use ROMIO
      • Installation and Configuration of ROMIO
    • Other Cluster File Systems
  • Load Balancing
    • Load Balancing in a Scyld Cluster
      • Mapping Policy
      • Queuing Policy
    • Implementing a Scheduling Policy
  • IPMI
    • IPMITool
  • Updating Software On Your Cluster
    • What Can’t Be Updated
  • Special Directories, Configuration Files, and Scripts
    • What Resides on the Master Node
      • /etc/beowulf/ directory
      • /usr/lib/beoboot directory
      • /var/beowulf directory
      • /var/log/beowulf directory
    • What Gets Put on the Compute Nodes at Boot Time
    • /usr/lib/locale/locale-archive Internationalization
    • Site-Local Startup Scripts
    • Sample Kickstart Script
• User’s Guide
  • Preface
  • Scyld ClusterWare Overview
  • What Is a Beowulf Cluster?
    • A Brief History of the Beowulf
    • First-Generation Beowulf Clusters
    • Scyld ClusterWare: A New Generation of Beowulf
  • Scyld ClusterWare Technical Summary
    • Top-Level Features of Scyld ClusterWare
    • Process Space Migration Technology
    • Compute Node Provisioning
    • Compute Node Categories
    • Compute Node States
    • Major Software Components
  • Typical Applications of Scyld ClusterWare
  • Interacting With the System
    • Verifying the Availability of Nodes
    • Monitoring Node Status
      • The BeoStatus GUI Tool
      • The bpstat Command Line Tool
      • The beostat Command Line Tool
    • Issuing Commands
      • Commands on the Master Node
      • Commands on the Compute Node
    • Copying Data to the Compute Nodes
      • Sharing Data via NFS
      • Copying Data via bpcp
      • Programmatic Data Transfer
      • Data Transfer by Migration
    • Monitoring and Controlling Processes
  • Running Programs
    • Program Execution Concepts
      • Stand-Alone Computer vs. Scyld Cluster
      • Traditional Beowulf Cluster vs. Scyld Cluster
      • Program Execution Examples
    • Environment Modules
    • Running Programs That Are Not Parallelized
      • Starting and Migrating Programs to Compute Nodes (bpsh)
      • Copying Information to Compute Nodes (bpcp)
    • Running Parallel Programs
      • An Introduction to Parallel Programming APIs
      • Mapping Jobs to Compute Nodes
      • Running MPICH and MVAPICH Programs
      • Running OpenMPI Programs
      • Running MPICH2 and MVAPICH2 Programs
      • Running PVM-Aware Programs
      • Porting Other Parallelized Programs
    • Running Serial Programs in Parallel
      • mpprun
      • beorun
    • Job Batching
      • Job Batching Options for ClusterWare
      • Job Batching with TORQUE
      • Job Batching with POD Tools
    • Using Singularity
    • File Systems
  • Glossary of Parallel Computing Terms
  • TORQUE and Maui Release Information
  • OpenMPI Release Information
  • MPICH2 Release Information
  • MPICH-3 Release Information
    • CHANGELOG
    • Release Notes
  • MVAPICH2 Release Information
  • SLURM Release Information
• Programmer’s Guide
  • Under Revision
• Reference Guide
  • Preface
  • Scyld ClusterWare Commands
    • beoboot
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • beoconfig
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • beomap
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
      • Special Notes
    • beonpc
      • Name
      • Synopsis
      • Description
      • Options
      • Example
    • beorun
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • beosi
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • beostatus
      • Name
      • Synopsis
      • Description
      • Options
      • FILTERING OPTIONS
      • Examples
    • beostat
      • Name
      • Synopsis
      • Description
      • Options
    • bpcp
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • bpdate
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • bpsh
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • bpstat
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • mpprun
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
  • Scyld ClusterWare Maintenance Commands
    • beofdisk
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • beorsync
      • Name
      • Synopsis
      • Description
      • Errors
      • Examples
    • beoserv
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • bpctl
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
      • Return Values
    • bplib
      • Name
      • Synopsis
      • Description
      • Options
    • bpmaster
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • bpslave
      • Name
      • Synopsis
      • Description
      • Options
    • node_down
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • recvstats
      • Name
      • Synopsis
      • Description
      • Options
      • Examples
    • sendstats
      • Name
      • Synopsis
      • Description
      • Examples
  • Scyld ClusterWare Special Files
    • beowulf-config
      • Name
      • Description
      • Config File Format
      • Keywords
      • Examples
    • beowulf-fstab
      • Name
      • Description
      • Syntax
      • Examples
      • Files
    • beowulf-nsswitch.conf
      • Name
      • Description
      • See Also
  • Scyld ClusterWare Beostat Libraries
    • beostat_count_idle_cpus
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_count_idle_cpus_on_node
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_avail_nodes_by_id
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_cpu_count
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • beostat_get_cpu_percent
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_cpuinfo_x86
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_disk_usage
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_last_multicast
      • Name
      • Synopsis
      • Description
      • Return Value
      • Errors
    • beostat_get_loadavg
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_meminfo
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_MHz
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_name
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_net_dev
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_net_rate
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_stat_cpu
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_statfs_p
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_get_time
      • Name
      • Synopsis
      • Arguments
      • Description
      • Examples
      • Return Value
      • Errors
    • beostat_is_node_available
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
  • Scyld ClusterWare BProc Libraries
    • bproc_access
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_chown
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_currnode
      • Name
      • Synopsis
      • Return Value
      • Bugs
      • Errors
    • bproc_detach
      • Name
      • Synopsis
      • Description
      • Return Value
      • Errors
      • Notes
    • bproc_execmove
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_getnodebyname
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_masteraddr
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_move
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_nodeaddr
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_nodeinfo
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_nodenumber
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_nodestatus
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_numnodes
      • Name
      • Synopsis
      • Description
      • Return Value
      • Errors
    • bproc_pidnode
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Bugs
      • Errors
    • bproc_rexec
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_rfork
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
    • bproc_setnodestatus
      • Name
      • Synopsis
      • Arguments
      • Description
      • Return Value
      • Errors
• Feedback