scyld-nodectl

NAME

scyld-nodectl -- Query and modify nodes for the cluster.

USAGE

scyld-nodectl

[-h] [-v] [-q] [[-c | --config] CONFIG] [--base-url URL] [[-u | --user] USER[:PASSWD]] [--human | --json | --csv | --table] [--pretty | --no-pretty] [--show-uids] [-a | -i NODES] | --up | --down | --booting] {clear, clone,cp, create,mk, delete,rm, exec, hardware, join, leave, list,ls, ping, power, reboot, replace,re, scp, script, set, shutdown, sol, ssh, status, update,up, waitfor}

OPTIONAL ARGUMENTS

-h, --help

Print usage message and exit. Ignore trailing args, parse and ignore preceding args.

-v, --verbose

Increase verbosity.

-q, --quiet

Decrease verbosity.

-c, --config CONFIG

Specify a client configuration file CONFIG.

--show-uids

Do not try to make the output more human readable.

-a, --all

Interact with all nodes (default for list).

-i, --ids NODES

A comma-separated list of nodes or an admin-defined group of nodes to act upon.

--up

Interact with all "up" nodes.

--down

Interact with all "down" nodes.

--booting

Interact with all "booting" nodes.

ARGUMENTS TO OVERRIDE BASIC CONFIGURATION DETAILS

--base-url URL

Specify the base URL of the ClusterWare REST API.

-u, --user USER[:PASSWD]

Masquerade as user USER with optional colon-separated password PASSWD.

FORMATTING ARGUMENTS

--human

Format the output for readability (default).

--json

Format the output as JSON.

--csv

Format the output as CSV.

--table

Format the output as a table.

--pretty

Indent JSON or XML output, and substitute human readable output for other formats.

--no-pretty

Opposite of --pretty.

ACTIONS

clear [-a | --all | NAME ... ]

Delete attribute name(s) and their value(s).

-a, --all

Delete all attributes.

clone (cp) [--content JSON | INI_FILE] [NAME=VALUE ...]

Copy node with new NAME/VALUE identifier pairs.

--content JSON | INI_FILE

Overwrite fields in the cloned node.

create (mk) [--content JSON | INI_FILE ] [NAME=VALUE ...]

Add a node, commonly by specifying its MAC address (e.g., mac=MACaddr, that assigns the next available node number and associated IP address).

--content JSON | INI_FILE

Load this content into the database as a node.

delete (rm)

Delete node(s).

exec [--grouped] [--in-order] [--label] [--stdin IN] [--binary] [--stdout OUT] [--stderr ERR] CMD

Execute the CMD (double-quotes are optional) on node(s). The scyld-nodectl exec command passes its current stdin, stdout, and stderr to the remote command, or uses the --stdin, --stdout, and/or --stderr arguments to override the default(s) with a file.

When run via an ssh command (e.g. ssh cwhead scyld-nodectl --up exec uptime), that stdin should be provided and closed with Ctrl-d, or ssh should be passed the -t argument to force tty allocation. Otherwise the command will detect stdin is a pipe and wait for end-of-file.

Commands executed on multiple nodes will execute in parallel. The degree of fan-out can be controlled through the ssh_runner.fanout configuration variable in base.ini. Because these commands execute in parallel, their output may be interleaved or not in node index order. Override this with grouped or --in-order arguments.

For sshpass functionality, see _remote_pass in the Reserved Attributes section of the Reference Guide.

--grouped

Results are locally buffered and printed grouped by node.

--in-order

Output is printed in node index order, implies --grouped.

--label

Force output labeling, even if a single node is selected.

--stdin IN

Provide @file or input string as stdin for the CMD.

--binary

Treat CMD output as binary data.

--stdout OUT

Provide a filename OUT for the CMD stdout output. Any {} in the filename gets translated to the node name (see EXAMPLES).

--stderr ERR

Provide a filename ERR for the CMD stderr output. Any {} in the filename gets translated to the node name (see EXAMPLES). An ERR value consisting of the string STDOUT will merge stderr into stdout.

hardware

Show the "hardware" information subset of scyld-nodectl ls -L.

join GROUP ...

Append GROUP(S) to the node group lists.

leave [-a | --all | GROUP ...]

Remove GROUP(S) from the node group lists.

-a, --all

Remove node(s) from all groups (other than the global default).

list (ls) [--long | --long-long | --raw]

Show information about nodes.

-l, --long

Show a subset of all optional information for each node.

-L, --long-long

Show all optional information for each node.

--raw

Display the raw JSON content from the database.

ping [COUNT]

ping the specified node(s) with COUNT packets (default 1).

power {on | off | cycle | status | setnext BOOTDEV}

Display or control the node power state through the plugin defined by the node's power_uri, usually ipmi. The options on, off, cycle, and status correspond to ipmitool actions.

The option setnext specifies the boot device or method to use for the next node boot. BOOTDEV choices are none, pxe, disk, and bios.

reboot [--soft | --hard] [--kexec] [--force] [--timeout SECS]

Reboot node(s) using either "soft" (using ssh) or "hard" (using ipmi) or kexec methods. If none is specified, then the default behavior is to initially attempt a "soft" reboot; and if after a short delay (default 5 seconds) the node does not appear to begin a reboot, then perform a "hard" power cycle. Ignore the reboot if the node's _no_boot is set to one, unless an overriding --force argument is supplied.

--soft

Reboot node(s) using ssh methods.

--hard

Reboot node(s) using ipmi methods.

--kexec

Boot directly into a new kernel without a full reboot which would include Power On Self Test (POST) and hardware initialization. See man kexec for details. This is implemented on a compute node using the ClusterWare reboot-kexec tool which installs from the clusterware-node package.

--force

Override the node's _no_reboot attribute value when set to 1.

--timeout SECS

Wait a non-default SECS seconds between "soft" and "hard" methods.

replace (re) [--content JSON | INI_FILE ] [ NAME=VALUE ] ...

Replace all node fields.

--content JSON | INI_FILE

Replace all fields with the specified content.

scp

See scyld-nodectl exec in EXAMPLES, below.

scp

Copy files to or from node(s).

script SCRIPT

Execute the specified ClusterWare SCRIPT (distributed in the clusterware-node package) on the specified compute node(s). The script name list (or ls) displays names of the available scripts, which generally execute automatically at boot time to facilitate various node initializations and have limited usefulness for later execution by a cluster administrator. However, the scripts fetch_hosts (re-download the list of head nodes) and update_keys (update SSH keys) may be useful in rare circumstances for a booted node.

set [--content JSON | INI_FILE ] [ NAME=VALUE ] ...

Set attribute value(s).

--content JSON | INI_FILE

Import the NAME/VALUE pairs from the file into the node attributes.

shutdown [--soft | --hard] [--timeout SECS]

Shutdown node(s) using either soft (using ssh) or hard (using ipmi) methods. If neither --soft nor --hard is specified, then the default behavior is to first attempt a soft shutdown; if after a short delay the node does not appear to begin a shutdown, then perform a hard power off.

--hard

Shutdown node(s) using ipmi methods.

--soft

Shutdown node(s) using ssh methods.

--timeout SECS

Wait SECS seconds between "soft" and "hard" methods.

sol [--enable ID] [--steal]

Start a serial-over-lan connection using the local ipmitool.

--enable ID

If SOL payload is disabled, then attempt to enable for ID and retry.

--steal

If an SOL session is currently active for that node, then deactivate that session and retry.

ssh [--pubkey FILE]

Create an SSH connection to the specified node as the user root. This is done using a local SSH key that is temporarily copied to the compute node through the head node and removed after the command completes. The user can provide their own public key, or one will be generated and stored in ~/.scyldcw/tempauth.key.

--pubkey FILE

Specify a file containing a public key to use for this connection.

status [--long] [--long-long] [--health]

Show node status.

--health

Show status based on _health attribute.

-l, --long

Show a subset of all optional information for each node.

-L, --long-long

Show all optional information for each node.

--raw

Display the raw JSON content from the database.

--refresh

Show basic node states, refreshing for any state change.

update (up) [--content JSON | INI_FILE ] [ NAME=VALUE ] ...

Modify node NAME field(s) with new value(s).

--content JSON | INI_FILE

Overwrite this content into the database for a node.

waitfor [Options] COND

Complete when one or more of the specified nodes meet the condition COND, which is either an expression or a '@'-prefixed file name. If no nodes are specified, then defaults to --all.

--failure COND

Also complete if the failure condition becomes true.

--timeout SECS

Complete after SECS seconds if condition(s) never become true.

--name NAME

Use the currently defined COND state known as NAME, or define a new COND and remember it as NAME.

--load-only

Just save the state sets into the database.

--delete NAME

Delete an existing state set NAME.

--show [NAME]

Show a list of all state sets, or optionally just the details of one.

--stream

Stream back ongoing results instead of returning the first result and exiting.

--skip

Do not use or print the initial node states.

--one-per

Stream node state changes with one node per line.

--this-head

Only return state changes handled by the current head.

EXAMPLES

scyld-nodectl list

List all node names.

scyld-nodectl status

Shows the basic state of each node.

scyld-nodectl status

Shows the basic state of node n5.

scyld-nodectl -i n5 ls -L

Shows full information available for node n5.

scyld-nodectl -i %groupx ls -l

Shows an expanded information available for each node joined to the admin-defined group groupx.

scyld-nodectl create mac=00:25:90:0C:D9:3C

Add a new node to the end of the current list of nodes.

scyld-nodectl create mac=00:25:90:0C:D9:3C index=10

Add a new node beyond the end of the current list of nodes as node n10.

scyld-nodectl -i n3 update mac=40:25:88:0C:B9:2C

Replace the current MAC address for node n5 with a new MAC address.

scyld-nodectl -i n20 update power_uri=ipmi:///admin:passwd@10.2.255.37

Replace the current power_uri (defaults to "none") to an ipmitool authentication and BMC IP address.

scyld-nodectl -in2 ssh

Use ssh to open a shell on node n2.

scyld-nodectl -i n2 exec ls /var/log

Execute ls /var/log on node n2, directing stdout and stderr to scyld-nodectl's stdout and stderr, respectively.

scyld-nodectl -i n2 exec --stdout /tmp/n2.var.log ls /var/log

Execute ls /var/log on node n2, directing stdout to the head node file /tmp/n2.var.log.

scyld-nodectl -i n[2-4] exec --stderr STDOUT --stdout /tmp/{}.var.log ls /var/log

Execute ls /var/log on nodes n2, n3, and n4, directing both stderr and stdout to the head node files /tmp/n2.var.log, /tmp/n3.var.log, and /tmp/n4.var.log, respectively.

scyld-nodectl --up exec --stderr STDOUT --stdout /tmp/{}.var.log ls /var/log

Perform the same action as above, although this time for all the "up" nodes.

scyld-nodectl -in5 exec --stdout /tmp/n5-log.tar.gz tar -czf- /var/log

Execute tar -czf- /var/log on node n5, directing the stdout of the packed result into the head node file /tmp/n5-log.tar.gz.

scyld-nodectl -in5 exec --stdin=@/tmp/n5-log.tar.gz tar -C /root -xzf-

Send the local file /tmp/n5-log.tar.gz as the stdin to node n5 as it executes tar -C /root -xzf- to unpack the stdin contents at /root.

scyld-nodectl -in3 scp check-health.sh r:/opt/scyld/clusterware-node/bin/check-health.sh

Copy the local check-health.sh file to node n3 as file /opt/scyld/clusterware-node/bin/check-health.sh.

scyld-nodectl -in3 scp check-health.sh r:/opt/scyld/clusterware-node/bin/

Copy the local check-health.sh file to node n3 directory /opt/scyld/clusterware-node/bin/. The trailing / in the remote path is mandatory to differentiate copying a file vs. copying a directory.

scyld-nodectl -in3 scp r:/opt/scyld/clusterware-node/bin/check-health.sh /tmp/

Copy the remote n3 check-health.sh file to the head node's /tmp/ directory.

scyld-nodectl -in3 scp /opt/scyld/clusterware-tools/examples/ r:/tmp/

Copy the directory /opt/scyld/clusterware-tools/examples to node n3 directory /tmp/. The trailing / in the remote path is mandatory to differentiate copying a file vs. copying a directory.

scyld-nodectl -i n4 reboot ; scyld-nodectl -i n4 waitfor 's[state] == "up"'

Reboot node n4, then wait until the node returns to the "up" state.

scyld-nodectl -i n4 reboot ; scyld-nodectl -i n4 waitfor up

Reboot node n4, then wait until the node returns to the "up" state. Another supported shorthand is the conditional "down".

scyld-nodectl -i n0 waitfor @/opt/scyld/clusterware-tools/examples/node-states.ini

For node n0 establish a waitfor state condition described in that specified examples file, in which the state condition is named status. If no -i <NODE(s) is specified, then defaults to --all.

scyld-nodectl waitfor --name status

For all nodes re-establish a waitfor state condition for the previously defined state named status. When the condition is true for any node, write the state to stdout and exit.

scyld-nodectl waitfor --name status --stream

For all nodes re-establish a waitfor state condition for the previously defined state named status. When the condition is true for any node, write the state change to stdout and continue executing.

scyld-nodectl -i n0 reboot then waitfor up then exec uname -r

Initiate a reboot of node n0, wait for the node to return to an "up" state, and then execute uname -r on the node.

scyld-nodectl -i n0 ssh

Start a ssh session on node n0 as user root (by default) or whatever user is specified in the node's _remote_user attribute.

RETURN VALUES

Upon successful completion, scyld-nodectl returns 0. On failure, an error message is printed to stderr and scyld-nodectl returns 1.