Appendix: Creating New Plugins

Clusterware provides a flexible plugin system for collecting a variety of node information – status, hardware, health, and telegraf monitoring. There are 4 types of plugins:

  • Status Plugins

    • These are called every status cycle, usually every 10 sec; can return any datatype: e.g. free RAM is given as a float, distro is a string, etc.; one script can return multiple “sensor” readings;

  • Hardware Plugins

    • These called less often than “regular” status plugins, usually every 300 sec; can return any datatype; one script can return multiple “sensor” readings;

  • Health-Check Plugins

    • These are called less often than “regular” status plugins, usually every 300 sec; returns “healthy”, “unhealthy”, or a time-stamp (indiciating that the plugin is still calculating the value); one script can return multiple “sensor” readings;

  • Telegraf Plugins

    • These are really smaller, more granular Telegraf config files that ClusterWare can individually enable/disable; each script is a self-contained piece of Telegraf configuration which contains settings for a given Telegraf “input” module.

Note

While plugins do run as root on the compute nodes, they can still be restricted by SELinux.

Status Plugins

Broadly speaking, a status plugin is a shell-script that returns one or more JSON-like blobs of data for a named sensor reading:

“<sensor_name>”:  <JSON_data>

For a single plugin that emits multiple sensor readings, they should be separated by a newline:

“<sensor_name1>”:  <JSON_data>
“<sensor_name2>”:  <JSON_data>

The script runs as root and can use any tool, or sequence of tools, on the system in order to collect the information and format it properly.

While the script could reach across the network to run commands on other systems, this would potentially take significantly longer to process and so it is discouraged, especially in status plugins, although pinging other systems or doing more complicated communication checks may be appropriate in health checks.

The first line of the script should be a #! line giving the shell or interpreter to use. Every plugin is sent the path to the clusterWare-node package as the first argument since there is a useful “library” of routines in the functions.sh script. While less useful for the usually-dynamic status information, every script is sent a cache directory as the second argument, usually /opt/scyld/clusterware-node/etc/status.d. This directory can be used to store information for the next cycle rather than recomputing some lengthy calculation.

The sensor name can be any quoted string, though it is recommended to not have spaces or odd characters to avoid later problems when trying to use the data, we suggest using alphanumeric characters and underscores (A-Za-z0-9_). The JSON-data portion can be any JSON data: a string, number, boolean, list, or dictionary.

As the plugins are just scripts, to debug them, simply execute them manually and verify that the output is correct. Note that the plugin system does very little error-checking on the scripts, and since it is executing so often, it does not do a formal check of the JSON. If bad data is returned, it will cause failures in the status-update cycle. On the compute node, the failure will likely be silent (unless _status_debug=1) so it is best to check the head node logs to get more information on the error.

Note

Changes to _status_plugins will be processed on the next status-update cycle, usually every 10 seconds unless changed by the admin.

Hardware Plugins

Broadly speaking, a hardware plugin is also just a shell-script that returns one or more JSON-like blobs of data for a named sensor reading:

“<sensor_name>”:  <JSON_data>

For a single plugin that emits multiple sensor readings, they should be separated by a newline. As with status plugins, the script can run any tool, or sequence of tools, on the system.

The first line of the script should be a #! line giving the shell or interpreter to use.

Since hardware information is usually more static, it may make sense for hardware plugins to calculate information once and then cache it to a file on disk so as to reduce future computational effort needed. Every script is sent the cache directory as the second argument, usually /opt/scyld/clusterware-node/etc/hardware.d. Every hardware plugin is also sent the path to the clusterware-node package as the first argument since there is a useful “library” of routines in the functions.sh script.

The sensor name can be any quoted string, though it is recommended to not have spaces or odd characters to avoid later problems when trying to use the data, we suggest using alphanumeric characters and underscores (A-Za-z0-9_). The JSON-data portion can be any JSON data: a string, number, boolean, list, or dictionary.

As the hardware plugins are just scripts, to debug them, simply execute them manually and verify that the output is correct. As with the status plugins, there is very little error-checking done on the scripts.

Note

Changes to _hardware_plugins will likely be detected on the next status cycle, usually every 10 seconds, but will not be processed until the next hardware-update cycle, usually every 300 seconds unless changed by the admin.

Health-Check Plugins

A health plugin is a shell-script that returns one or more strings for a named sensor reading:

"<sensor_name>":  "<string>"

where the string is usually either “healthy” or “unhealthy”, but any string can be sent. If the first word in the string is “healthy”, then it is assumed that the health-check passed; any other string is considered unhealthy (not just the word “unhealthy”). One can optionally communicate more specific information in the string, e.g. any errors or issues that were detected. I.e. heathy: ping time was 0.164 ms would also indicate a healthy check; unhealthy: could not ping server or simply could not ping server would also indicate an unhealthy check (since it does not start with healthy) and also give a possible path to help triage the problem. If the plugin emits more than one sensor reading, they should be separated by newlines.

The sensor name can be any quoted string, though it is recommended to not have spaces or odd characters to avoid later problems when trying to use the data, we suggest using alphanumeric characters and underscores (A-Za-z0-9_).

As with status plugins, the health-check script can run any tool, or sequence of tools, on the system. The first line of the script should be a #! line giving the shell or interpreter to use.

Every script is given the cache directory as the second argument, usually /opt/scyld/clusterware-node/etc/health.d. Lengthier calculations can potentially be run once and stored there for future use. Every health-check plugin is also sent the path to the clusterware-node package as the first argument since there is a useful “library” of routines in the functions.sh script.

Prior to sending the data to the head node, the compute node will assess all the health-check values and assign the node a global "health" status in the node's _health attribute. If any health check reports unhealthy, then the node is considered unhealthy; if all checks report healthy, then the node is considered healthy.

As the health-check plugins are just scripts, to debug them, simply execute them manually and verify that the output is correct. As with the status plugins, there is very little error-checking done on the scripts.

Note

Changes to _health_plugins will likely be detected on the next status cycle, usually every 10 seconds, but will not be processed until the next health-update cycle, usually every 300 seconds unless changed by the admin.

Telegraf Plugins

Where the status plugins are small scripts that are run during the periodic status-update cycle, Telegraf plugins are small configuration files that can be enabled/disabled by the HPC-admin.

“Developing” Telegraf plugins for ClusterWare really means the creation of config files, often by splitting up one of the larger Telegraf sample configuration files into smaller, self-contained pieces. Developing brand-new Telegraf plugins is outside the scope of this document.

A list of Telegraf plugins can be found at: https://docs.influxdata.com/telegraf/v1/plugins/

Note

head nodes can only use the telegraf-enabled directory as there is no _telegraf_plugins attribute. Admins must run the /opt/scyld/clusterware-telegraf/bin/reconfig-telegraf.sh script manually to push any changes into production.

Examples

As an example, if a cluster was running the Ceph file system, then one might want to use the Telegraf “inputs.ceph” module:

https://github.com/influxdata/telegraf/blob/release-1.28/plugins/inputs/ceph/README.md

To do so, create a new plugin file: /opt/scyld/clusterware-telegraf/telegraf-available/ceph.conf with the following:

[[inputs.ceph]]
interval = '1m'
ceph_binary = "/usr/bin/ceph"
socket_dir = "/var/run/ceph"
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
mds_prefix = "ceph-mds"
rgw_prefix = "ceph-client"
socket_suffix = "asok"
ceph_user = "client.admin"
ceph_config = "/etc/ceph/ceph.conf"
gather_admin_socket_stats = true
gather_cluster_stats = false

With that file stored in telegraf-available, the admin can either sym-link it into telegraf-enabled or add it to the _telegraf_plugins attribute:

scyld-nodectl –all set _telegraf_plugins=ceph

Once the plugin is enabled through either method, the Telegraf daemon on the compute nodes will begin sending Ceph data to the head nodes.

Another example would be to ping several remote servers instead of just the parent head node. This could be useful, for example, to detect issues with a node's connection to the primary storage server and a network gateway, for example. In this case, we could edit the ping.conf file in telegraf-available and include the storage server name in the list of URLs to ping:

[[inputs.ping]]
## List of urls to ping
urls = ["parent-head-node", “storage-server”, “gateway-server”]
## number of pings to send per collection (ping -c <COUNT>)
count = 1

The documentation on the Ping plugin:

https://github.com/influxdata/telegraf/blob/release-1.28/plugins/inputs/ping/README.md

shows a number of other options that admins may find useful to configure. If an existing file is modified, a restart of Telegraf will be necessary to have the system re-read the configuration file (systemctl restart telegraf).

Note that Telegraf's exec and execd modules allow for arbitrary scripts or executables to be run, with the output being ingested into Telegraf. This can be a very powerful tool for admins to write custom scripts that might be very specific to their cluster.

See the Influxdata documentation for more information: