scyld-nodectl

NAME

scyld-nodectl -- Query and modify nodes for the cluster.

USAGE

scyld-nodectl

[-h] [-v] [-q] [[-c | --config] CONFIG] [--base-url URL] [[-u | --user] USER[:PASSWD]] [--human | --json | --csv | --table] [--pretty | --no-pretty] [--show-uids] [-a | -i NODES] | --up | --down | --booting] {list,ls, create,mk, clone,cp, update,up, replace,re, del,rm, set,clear, join,leave, status, hardware, reboot, shutdown, power, sol, ssh, script, exec, ping, waitfor, scp}

OPTIONAL ARGUMENTS

-h, --help

Print usage message and exit. Ignore trailing args, parse and ignore preceding args.

-v, --verbose

Increase verbosity.

-q, --quiet

Decrease verbosity.

-c, --config CONFIG

Specify a client configuration file CONFIG.

--show-uids

Do not try to make the output more human readable.

-a, --all

Interact with all nodes (default for list).

-i, --ids NODES

A comma-separated list of nodes or an admin-defined group of nodes to act upon.

--up

Interact with all "up" nodes.

--down

Interact with all "down" nodes.

--booting

Interact with all "booting" nodes.

ARGUMENTS TO OVERRIDE BASIC CONFIGURATION DETAILS

--base-url URL

Specify the base URL of the ClusterWare REST API.

-u, --user USER[:PASSWD]

Masquerade as user USER with optional colon-separated password PASSWD.

FORMATTING ARGUMENTS

--human

Format the output for readability (default).

--json

Format the output as JSON.

--csv

Format the output as CSV.

--table

Format the output as a table.

--pretty

Indent JSON or XML output, and substitute human readable output for other formats.

--no-pretty

Opposite of --pretty.

ACTIONS

list (ls) [--long | --long-long | --raw]

Show information about nodes.

-l, --long

Show a subset of all optional information for each node.

-L, --long-long

Show all optional information for each node.

--raw

Display the raw JSON content from the database.

hardware

Show the "hardware" information subset of scyld-nodectl ls -L.

status [--long] [--long-long] [--health]

Show node status.

--health

Show status based on _health attribute.

-l, --long

Show a subset of all optional information for each node.

-L, --long-long

Show all optional information for each node.

--raw

Display the raw JSON content from the database.

--refresh

Show basic node states, refreshing for any state change.

create (mk) [--content JSON | INI_FILE ] [NAME=VALUE ...]

Add a node, commonly by specifying its MAC address (e.g., mac=MACaddr, that assigns the next available node number and associated IP address).

--content JSON | INI_FILE

Load this content into the database as a node.

clone (cp) [--content JSON | INI_FILE] [NAME=VALUE ...]

Copy node with new NAME/VALUE identifier pairs.

--content JSON | INI_FILE

Overwrite fields in the cloned node.

delete (rm)

Delete node(s).

update (up) [--content JSON | INI_FILE ] [ NAME=VALUE ] ...

Modify node NAME field(s) with new value(s).

--content JSON | INI_FILE

Overwrite this content into the database for a node.

replace (re) [--content JSON | INI_FILE ] [ NAME=VALUE ] ...

Replace all node fields.

--content JSON | INI_FILE

Replace all fields with the specified content.

set [--content JSON | INI_FILE ] [ NAME=VALUE ] ...

Set attribute value(s).

--content JSON | INI_FILE

Import the NAME/VALUE pairs from the file into the node attributes.

clear [-a | --all | NAME ... ]

Delete attribute name(s) and their value(s).

-a, --all

Delete all attributes.

join GROUP ...

Append GROUP(S) to the node group lists.

leave [-a | --all | GROUP ...]

Remove GROUP(S) from the node group lists.

-a, --all

Remove node(s) from all groups (other than the global default).

reboot [--soft | --hard] [--kexec] [--force] [--timeout SECS]

Reboot node(s) using either "soft" (using ssh) or "hard" (using ipmi) or kexec methods. If none is specified, then the default behavior is to initially attempt a "soft" reboot; and if after a short delay (default 5 seconds) the node does not appear to begin a reboot, then perform a "hard" power cycle. Ignore the reboot if the node's _no_boot is set to one, unless an overriding --force argument is supplied.

--soft

Reboot node(s) using ssh methods.

--hard

Reboot node(s) using ipmi methods.

--kexec

Boot directly into a new kernel without a full reboot which would include Power On Self Test (POST) and hardware initialization. See man kexec for details. This is implemented on a compute node using the ClusterWare reboot-kexec tool which installs from the clusterware-node package.

--force

Override the node's _no_reboot attribute value when set to 1.

--timeout SECS

Wait a non-default SECS seconds between "soft" and "hard" methods.

shutdown [--soft | --hard] [--timeout SECS]

Shutdown node(s) using either soft (using ssh) or hard (using ipmi) methods. If neither --soft nor --hard is specified, then the default behavior is to first attempt a soft shutdown; if after a short delay the node does not appear to begin a shutdown, then perform a hard power off.

--hard

Shutdown node(s) using ipmi methods.

--soft

Shutdown node(s) using ssh methods.

--timeout SECS

Wait SECS seconds between "soft" and "hard" methods.

power {on | off | cycle | status | setnext BOOTDEV}

Display or control the node power state through the plugin defined by the node's power_uri, usually ipmi. The options on, off, cycle, and status correspond to ipmitool actions.

The option setnext specifies the boot device or method to use for the next node boot. BOOTDEV choices are none, pxe, disk, and bios.

sol [--enable ID] [--steal]

Start a serial-over-lan connection using the local ipmitool.

--enable ID

If SOL payload is disabled, then attempt to enable for ID and retry.

--steal

If an SOL session is currently active for that node, then deactivate that session and retry.

scp

See scyld-nodectl exec in EXAMPLES, below.

ssh [--pubkey FILE]

Create an SSH connection to the specified node as the user root. This is done using a local SSH key that is temporarily copied to the compute node through the head node and removed after the command completes. The user can provide their own public key, or one will be generated and stored in ~/.scyldcw/tempauth.key.

--pubkey FILE

Specify a file containing a public key to use for this connection.

script SCRIPT

Execute the specified ClusterWare SCRIPT (distributed in the clusterware-node package) on the specified compute node(s). The script name list (or ls) displays names of the available scripts, which generally execute automatically at boot time to facilitate various node initializations and have limited usefulness for later execution by a cluster administrator. However, the scripts fetch_hosts (re-download the list of head nodes) and update_keys (update SSH keys) may be useful in rare circumstances for a booted node.

exec [--grouped] [--in-order] [--stdin IN] [--binary] [--stdout OUT] [--stderr ERR] CMD

Execute the CMD (double-quotes are optional) string on node(s). The scyld-nodectl exec command passes its current stdin, stdout, and stderr to the remote command, or uses the --stdin, --stdout, and/or --stderr arguments to override the default(s) with a file.

When run via an ssh command (e.g. ssh cwhead scyld-nodectl --up exec uptime), that stdin should be provided and closed with Ctrl-d, or ssh should be passed the -t argument to force tty allocation. Otherwise the command will detect stdin is a pipe and wait for end-of-file.

Commands executed on multiple nodes will execute in parallel. The degree of fan-out can be controlled through the ssh_runner.fanout configuration variable in base.ini. Because these commands execute in parallel, their output may be interleaved or not in node index order. Override this with grouped or --in-order arguments.

For sshpass functionality, see _remote_pass in the Reserved Attributes section of the Reference Guide.

--grouped

Results are locally buffered and printed grouped by node.

--in-order

Output is printed in node index order, implies --grouped.

--stdin IN

Optional @file or input string provided to CMD executing on the node(s) as stdin.

--stdout OUT

Optionally write stdout to the file OUT. Any {} in the filename gets translated to the node name (see EXAMPLES).

--binary

Treat CMD output as binary data.

--stderr ERR

Optionally write stderr to the file ERR. Any {} in the filename gets translated to the node name (see EXAMPLES). An ERR value consisting of the string STDOUT will merge stderr into stdout.

ping [COUNT]

ping the specified node(s) with COUNT packets (default 1).

waitfor [Options] COND

Complete when one or more of the specified nodes meet the condition COND, which is either an expression or a '@'-prefixed file name. If no nodes are specified, then defaults to --all.

--failure COND

Also complete if the failure condition becomes true.

--timeout SECS

Complete after SECS seconds if condition(s) never become true.

--name NAME

Use the currently defined COND state known as NAME, or define a new COND and remember it as NAME.

--load-only

Just save the state sets into the database.

--delete NAME

Delete an existing state set NAME.

--show [NAME]

Show a list of all state sets, or optionally just the details of one.

--stream

Stream back ongoing results instead of returning the first result and exiting.

--skip

Do not use or print the initial node states.

--one-per

Stream node state changes with one node per line.

--this-head

Only return state changes handled by the current head.

EXAMPLES

scyld-nodectl list

List all node names.

scyld-nodectl status

Shows the basic state of each node.

scyld-nodectl status

Shows the basic state of node n5.

scyld-nodectl -i n5 ls -L

Shows full information available for node n5.

scyld-nodectl -i %groupx ls -l

Shows an expanded information available for each node joined to the admin-defined group groupx.

scyld-nodectl create mac=00:25:90:0C:D9:3C

Add a new node to the end of the current list of nodes.

scyld-nodectl create mac=00:25:90:0C:D9:3C index=10

Add a new node beyond the end of the current list of nodes as node n10.

scyld-nodectl -i n3 update mac=40:25:88:0C:B9:2C

Replace the current MAC address for node n5 with a new MAC address.

scyld-nodectl -i n20 update power_uri=ipmi:///admin:passwd@10.2.255.37

Replace the current power_uri (defaults to "none") to an ipmitool authentication and BMC IP address.

scyld-nodectl -in2 ssh

Use ssh to open a shell on node n2.

scyld-nodectl -i n2 exec ls /var/log

Execute ls /var/log on node n2, directing stdout and stderr to scyld-nodectl's stdout and stderr, respectively.

scyld-nodectl -i n2 exec --stdout /tmp/n2.var.log ls /var/log

Execute ls /var/log on node n2, directing stdout to the head node file /tmp/n2.var.log.

scyld-nodectl -i n[2-4] exec --stderr STDOUT --stdout /tmp/{}.var.log ls /var/log

Execute ls /var/log on nodes n2, n3, and n4, directing both stderr and stdout to the head node files /tmp/n2.var.log, /tmp/n3.var.log, and /tmp/n4.var.log, respectively.

scyld-nodectl --up exec --stderr STDOUT --stdout /tmp/{}.var.log ls /var/log

Perform the same action as above, although this time for all the "up" nodes.

scyld-nodectl -in5 exec --stdout /tmp/n5-log.tar.gz tar -czf- /var/log

Execute tar -czf- /var/log on node n5, directing the stdout of the packed result into the head node file /tmp/n5-log.tar.gz.

scyld-nodectl -in5 exec --stdin=@/tmp/n5-log.tar.gz tar -C /root -xzf-

Send the local file /tmp/n5-log.tar.gz as the stdin to node n5 as it executes tar -C /root -xzf- to unpack the stdin contents at /root.

scyld-nodectl -i n4 reboot ; scyld-nodectl -i n4 waitfor 's[state] == "up"'

Reboot node n4, then wait until the node returns to the "up" state.

scyld-nodectl -i n4 reboot ; scyld-nodectl -i n4 waitfor up

Reboot node n4, then wait until the node returns to the "up" state. Another supported shorthand is the conditional "down".

scyld-nodectl -i n0 waitfor @/opt/scyld/clusterware-tools/examples/node-states.ini

For node n0 establish a waitfor state condition described in that specified examples file, in which the state condition is named status. If no -i <NODE(s) is specified, then defaults to --all.

scyld-nodectl waitfor --name status

For all nodes re-establish a waitfor state condition for the previously defined state named status. When the condition is true for any node, write the state to stdout and exit.

scyld-nodectl waitfor --name status --stream

For all nodes re-establish a waitfor state condition for the previously defined state named status. When the condition is true for any node, write the state change to stdout and continue executing.

scyld-nodectl -i n0 reboot then waitfor up then exec uname -r

Initiate a reboot of node n0, wait for the node to return to an "up" state, and then execute uname -r on the node.

RETURN VALUES

Upon successful completion, scyld-nodectl returns 0. On failure, an error message is printed to stderr and scyld-nodectl returns 1.