Scyld Cloud Accountant

An SCM cluster has resources that are consumable by its users, such as allocated storage and core-hours consumed on a compute node. Resource usage is stored in a dedicated database and the billable usage can typically be summarized in daily units over a single metric (such as core-hours, GB, or time). The Scyld Cloud Accountant periodically polls these resource databases, calculates the single usage metric for that particular resource type, and stores it within the accountant database. Users and administrators can then retrieve daily resource usage summaries from the accountant and retrieve results in JavaScript Object Notation (JSON) format using HTTP/HTTPS requests.

Resource Families and Resource Types

Resource types are grouped into resource families if they describe similar resources. Each resource family is expected to have a specific summary metric. Three different resource families are currently supported by the Scyld Cloud Accountant:

  • Storage resource usage is defined by the amount of disk bytes allocated.
  • User Storage resource usage is defined by the amount of disk bytes used.
  • Server-Instance resource usage is defined by the amount of time that a server has been used.
  • Job resource usage is defined by the number of core-hours used across all jobs.

There are several types of Job resource types, based on the scheduler:

  • PBS Torque
  • SGE
  • UGE
  • Slurm Workload Manager

In addition to daily summary information, the Scyld Cloud Accountant also has the ability to provide more detailed usage information about individual scheduler jobs.

Access Permissions

Access to daily usage summaries is determined by Cloud Auth. While every user is allowed to see their own resource usage information, account owners are also able to query the accountant for resource usage information on their managed users.

Scyld Cloud Accountant Setup

The cloudaccountant must be configured to connect to each cloudcontroller instance in the SCM installation. In most cases, there is a single cloudcontroller. Each cloudcontroller instance has a unique id: the first one is id 1, the next is id 2, etc.

  • sqlalchemy.cloud.controller-id.resource-type.url - Specifies the database URL for the resource-type for a given cloud-container-id. Currently assumes MySQL databases. For example: To point to the server-instances resource-type of cloud-controller-id 1, one would set sqlalchemy.cloud.1.instances.url = url
  • sqlalchemy.cloud.controller-id.slurm.server - Specifies a SSH URL capable of running sacct for the Slurm job manager

Initializing the Scyld Accountant Database

Note

This only applies to MySQL databases that have not been initialized.

If the Scyld Cloud Accountant database is ever destroyed and needs to be recreated, you can do so by running the database initialization script. Begin by logging in as root and activating the python environment for the accountant:

Verify that sqlalchemy.accountant.url has been properly defined in cloudaccountant.ini.

Now run the initialization script:

# source /var/www/wsgi/cloudaccountant/env/bin/activate
# initialize_scyld_cloud_accountant_db /var/www/wsgi/cloudaccountant/cloudaccountant.ini

Importing from Resource Databases

To manually import data from the resource databases into the cloud accountant databases, first, activate the Python environment for the accountant, then run the import script:

# source /var/www/wsgi/cloudaccountant/env/bin/activate
# scyld_cloud_accountant_db_import /var/www/wsgi/cloudaccountant/cloudaccountant.ini <START DATE> <END DATE>

This will import data relevant to all resources with URLs listed in the cloudaccountant.ini file between the given <START DATE> and <END DATE>.

NOTE: Importing over the same dates repeatedly will not remove any existing entries or produce duplicate rows. Importing will only add or modify existing entries.

To have this operation performed regularly, this action is usually handled as part of a cron script.

Importing from Slurm Accountant

To manually import data for from the resource databases into the cloud accountant databases, first, activate the Python environment for the accountant, then run the import script:

# source /var/www/wsgi/cloudaccountant/env/bin/activate
# scyld_cloud_accountant_slurm_import /var/www/wsgi/cloudaccountant/cloudaccountant.ini -r <START DATE>:<END DATE>

This will import data from the Slurm servers listed in the cloudaccountant.ini file between the given <START DATE> and <END DATE>.

NOTE: Importing over the same dates repeatedly will not remove any existing entries or produce duplicate rows. Importing will only add or modify existing entries.

To have this operation performed regularly, this action is usually handled as part of a cron script.