Appendix: Integrating Keycloak with ClusterWare for RBAC

Installation

Keycloak is a powerful, flexible authentication and authorization system with the ability to directly store user credentials and to link to other authentication providers. ClusterWare’s use of Keycloak does not necessarily use all of its features, and the full configuration process for all of Keycloak is beyond the scope of this document. For more in-depth information, admins can refer to the Keycloak website and, particularly, its documentation link:

https://www.keycloak.org/documentation

Assuming Keycloak is installed and operational, the process to integrate it with ClusterWare involves selecting or creating a realm, creating a new client within that realm, adding users to the realm, assigning or creating roles that will map to ClusterWare roles, and finally, configuring and restarting ClusterWare.

Installation instructions can be found in Keycloak’s documentation. This includes how-to guides for several different types of installations: bare metal, Docker, Podman, Kubernetes, and OpenShift.

https://www.keycloak.org/guides#getting-started https://www.keycloak.org/operator/basic-deployment

Select a Realm

In Keycloak, a "realm" is an administrative domain that includes users, groups, roles, and client application information. A fresh installation of Keycloak will only have one realm, called "master", and it is highly recommended that admins create at least one other realm to use in production. The "master" realm should not be used for daily operations. In an enterprise organization, there may already be a suitable realm available. Since client-specific information can be compartmentalized even within a realm, there shouldn’t be any concern about ClusterWare’s use of that realm interfering with other operations.

Note

When creating a new realm, the name that is given will be used as the unique identifier for the realm, and will show

up in the URL.

More information can be found in Keycloak’s Server Administration documentation:

https://www.keycloak.org/docs/latest/server_admin/index.html

Configuring realms:

https://www.keycloak.org/docs/latest/server_admin/index.html#configuring-realms

Create a New Client

Inside the selected realm, admins should create a new client – a client is an administrative container for information related to a specific application or use-case within this realm. In this case, ClusterWare will be given its own client-container so that it does not interfere with any other applications that may be using the same Keycloak instance.

When creating the client, note that the "client ID" acts as the unique identifier and as a "short name" for the client - it will be displayed on many of the Keycloak webpages and it will show up in the URL.

Specific settings to be verified:

  • The "Client authentication" and "Authorization" checkboxes must be enabled.

  • In the "Authentication flow" section, the "Standard flow" and "Direct access grants" checkboxes must be enabled.

  • The "Use refresh tokens" checkbox should be enabled (in the web-UI, this is under the "Advanced" tab).

Take note of the following information as it will be needed later during the ClusterWare configuration process:

  • The server connection information, including the server address or name and any port information

  • The realm name and client-ID

  • The "client secret" – in the web-UI, this can be found in the Credentials tab; the "Client Authenticator" should be "Client Id and Secret", and under it should be "Client Secret"

More information can be found in Keycloak’s Server Administration documentation:

https://www.keycloak.org/docs/latest/server_admin/index.html https://www.keycloak.org/docs/latest/server_admin/index.html#_oidc_clients

Add Users

After setting up these basic "containers" for interacting with ClusterWare, the next step is to create any users that will need ClusterWare admin roles. For enterprise organizations, these accounts may already be created and can be re-used for ClusterWare.

The user creation process is straightforward – each user needs a unique username (unique within the realm), and can optionally be given first/last names and an email address. If desired, users can be configured to be required to change their password at first login, or required to verify their email address, etc. If Keycloak was configured for other organizational uses, it may require additional or different user information, e.g. a physical address or an employee ID number.

It may be worth considering Keycloak’s "Groups" feature to categorize and organize users to simplify user-to-role mapping. E.g. a group could be defined for "HPC-Admins" and that group can then be granted a role of "hpc.fulladmin"; any user added to that group will thus become a Full Admin. This can greatly simplify the process of adding roles to large numbers of users.

More details can be found in Keycloak’s Server Administration documentation:

https://www.keycloak.org/docs/latest/server_admin/index.html

User management:

https://www.keycloak.org/docs/latest/server_admin/index.html#assembly-managing-users_server_administration_guide

Select or Create Roles

For a given user, ClusterWare will look at the Keycloak-provided list of roles to determine what that user is or is not allowed to do. The configuration process (see below) will need to map each of the ClusterWare roles to a Keycloak-provided role. For enterprise organizations, there may be existing roles that can be re-used for ClusterWare, but otherwise, admins will have to create any needed roles.

The ClusterWare roles are: * On-site Engineer * Imaging Engineer * Production Engineer * Manager * Full Admin * Authenticated User (this role is also granted to anyone in any of the other roles)

Once the roles are defined, users can then be assigned to those roles either directly or through a group.

Note

Admins may choose to not provide a mapping for a given ClusterWare role. In that case, then that role will simply be ignored. E.g. if there is no mapping for the Manager role, then no one will ever be able to be assigned that role.

More information can be found in Keycloak’s Server Administration documentation:

https://www.keycloak.org/docs/latest/server_admin/index.html

Roles and Groups:

https://www.keycloak.org/docs/latest/server_admin/index.html#assigning-permissions-using-roles-and-groups

Configuring ClusterWare

With the Keycloak configuration from above, admins can now configure ClusterWare to match. In particular, ClusterWare will need to know the basic configuration and the role-mapping details.

The Keycloak configuration info is stored in the ClusterWare database with the 'authctl' tool, found at '/opt/scyld/clusterware/bin/authctl'. When using 'authctl', replace the values with info from the actual Keycloak instance as noted above:

authctl set keycloak base_url https://192.168.122.33:8080 authctl set keycloak client clusterware authctl set keycloak realm penguin authctl set keycloak secret abc123def456ghi789

Next, the role-mapping needs to be configured. The mapping is given as a comma-separated list of "keycloak_role=clusterware_role" settings. For example:

authctl set keycloak role_mapping "kc_fulladmin=role.fulladmin,kc_manager=role.manager"

Note that the 'authctl' tool can also be used to double-check the data in the Clusterware database:

authctl show

Finally, the main ClusterWare configuration needs to be modified to swap out the standard authentication system (the appauth module) with the Keycloak module. Edit '/opt/scyld/clusterware/conf/base.ini' and look for the "plugins.auth" section:

plugins.auth = appauth

Change this to:

plugins.auth = keycloakauth

Note that it may be helpful to add a user or two to the 'auth.tmpadmins' list in 'base.ini' as well. If anything goes wrong with the Keycloak integration process, most users will not be able to authenticate – only the tmpadmins will be allowed into the system, and they will have full admin privileges.

When those file modifications are complete, save the file and restart ClusterWare:

systemctl restart clusterware

Important

For multi-head clusters, the base.ini file should be modified on all heads, then the service on each head node should be restarted.

Production Operations

Once the steps in the Installation process have been completed and ClusterWare has been restarted, the cluster should now be looking to Keycloak for authentication credentials and roles. For production operations, there are several important changes to how admins interact with the system.

User Management

Adding new users and modifying roles must now be done in both Keycloak and ClusterWare. The instructions for adding a new user to Keycloak are shown in the "Add Users" and "Select or Create Roles" sections above. Separately, the user must be added to ClusterWare through the 'scyld-adminctl' command:

scyld-adminctl create name=username

Note

When using Keycloak, no roles should be assigned in ClusterWare – all roles must be assigned through Keycloak.

Logging and Auditing

Keycloak includes configurable logging and auditing settings that admins can use to directly track user behavior on the cluster, or alternatively, to feed into another system for intrusion detection or log analysis.

By default, the log file will be 'keycloak.log' and will be found in the root directory where keycloak was installed, i.e. the '<kc-install-root>/data/log' directory. It should include all log messages except DEBUG level messages (so INFO, WARNING, ERROR, etc. should all be logged). The file follows standard Unix/Linux syslog formatting and should be easily interpretable by other tools. The documentation links below include examples of forwarding the data to Graylog and ELK log aggregation systems.

There are several sections of the Keycloak documentation that may be of interest:

https://www.keycloak.org/server/logging https://www.keycloak.org/docs/latest/server_admin/index.html#configuring-auditing-to-track-events

Access Token Lifespan

When an admin logs in to ClusterWare, the ClusterWare server will pass the information to Keycloak, which will verify the credentials and return a set of tokens (JWT, Java Web Tokens). Those tokens will be cached in each admin’s '.scyldcw' directory, in the 'auth_token.hdr' file. That file will include the current "access token" that can be used to perform actions in ClusterWare, as well as a "refresh token" that can be used to refresh the access token if or when it expires. If the refresh token does expire, the user will be prompted for their password, which will grant them a new set of tokens. The ClusterWare tools will automatically use the tokens in a way that should reduce the number of times the admin has to enter their password.

Current best practice is to limit the access token to a lifetime of approximately 5-10 minutes while allowing the refresh token to have a longer duration, such as 30-60 minutes. If a credential is stolen (or the file copied), then the Keycloak admins can potentially invalidate the refresh token and limit exposure to the compromised tokens. Token expiration settings for both access and refresh tokens will now be managed in Keycloak, so admins may re-configure those settings to reduce the frequency that passwords need to be re-entered.

Appendix: Integrating FreeIPA with ClusterWare

Installation

FreeIPA is a powerful, open-source identity management system that can export an LDAP directory of user credentials which can then be ingested into Keycloak and other authentication systems. ClusterWare's use of FreeIPA does not necessarily use all of its features, and the full configuration process for all of FreeIPA is beyond the scope of this document. For more in-depth information, admins can refer to the FreeIPA website and, particularly, its documentation link:

https://www.freeipa.org/page/Documentation.html

Assuming FreeIPA is installed and operational, the process to integrate it with Keycloak involves creating a new "User Federation" connection, including an optional "User LDAP Filter" to reduce the user accounts to only those in a certain group. Once configured, any users that match the filter will be available inside Keycloak and can then be assigned roles and added to ClusterWare.

Installation instructions can be found in FreeIPA's documentation. FreeIPA provides RPM and DEB packages, as well as container deployment options.

https://www.freeipa.org/page/Quick_Start_Guide https://www.freeipa.org/Downloads.html

Identify a Group for ClusterWare Users

In a larger enterprise, it may make sense to select or create a group that will be used to identify users who have administrative access to ClusterWare. This can help reduce the number of accounts that are being sync'ed between Keycloak and FreeIPA. If a group already exists for those users, the full DN for the group should be noted. Otherwise, a new group should be created and any ClusterWare admins should be added to that group.

By default, FreeIPA places user groups in a DN of the form:

cn=keycloak-allowed,cn=groups,cn=accounts,dc=<companyname>,dc=<com>

The DC components at the end will be dependent on the domain name as it is configured inside FreeIPA.

Identify an Admin Account for Keycloak

Keycloak will need to authenticate in order to access FreeIPA, so it will need an admin-level account to do so. If an account already exists, the full DN for the account should be noted. Otherwise, a new account should be created.

By default, FreeIPA places users in a DN of the form:

uid=<kcadmin>,cn=users,cn=accounts,dc=<companyname>,dc=<com>

Again, the DC components will reflect the domain name, and the kcadmin component will be the username.

Configure Keycloak

In Keycloak, switch to the realm that is being used by ClusterWare and look for the "User federation" tab in the left menu. When creating a new User-federation connection, select the "LDAP" option.

Several settings need to be configured:

  • For the "Vendor", select "Red Hat Directory Services"

  • Connection URL will be the URL to get to the FreeIPA server. Note that this should start with a prefix of ldap:// or ldaps://.

    • Use the "Test connection" button to verify that Keycloak can reach FreeIPA over the network.

If any problems show up, it may indicate firewall or other network issues.

  • Bind type is "simple", and use the DN for the admin user selected or created above. E.g. uid=<kcadmin>,cn=users,cn=accounts,dc=<companyname>,dc=<com>. Enter the admin password into the "Bind credential" text box.

    • Use the "Test authentication" button to verify that the username and password are working correctly, and that FreeIPA is responding properly to Keycloak's requests.

  • Edit mode should be "READ_ONLY"

  • For the default FreeIPA settings, the Users DN field should be cn=users,cn=accounts,dc=<companyname>,dc=<com>.

  • Username LDAP attribute and RDN LDAP attribute should both be set to uid.

  • UUID LDAP attribute should be uidNumber.

  • Although optional, admins are encouraged to set a User LDAP filter to reduce the number of user accounts that are downloaded and sync'ed from FreeIPA to Keycloak.

    • For a newly created group in a default FreeIPA system, a suitable filter might be: (memberOf=cn=<keycloak-allowed>,cn=groups,cn=accounts,dc=<companyname>,dc=<com>) where keycloak-group is the name of the group that contains all ClusterWare admins.

LDAP offers a power filtering syntax that can allow for one or more user-groups, or even selecting users by one or more roles. See https://ldap.com/ldap-filters/ for more information.

For all other settings, the defaults should work. Larger enterprises may want to think through the Periodic full sync and Periodic changed users sync settings to limit the load on the FreeIPA servers; note that those settings are defined in number of seconds between synchronization events.

Note that one must click the "Save" button at the bottom of the webpage or else any changes will be lost!

Verifying the Integration

Once the User federation connection is created, the connection can be verified by going to the Users page in the Keycloak web-UI. Since there is a federation defined, the Users webpage will only present a search box, not a list of known users. To see all users, simply search for *. Depending on the number of accounts that need to be downloaded or sync'ed, it may take a few seconds before results are rendered.

Once the page updates, all available users should be shown -- all users defined inside Keycloak as well as those defined in FreeIPA.

Note

Keycloak and FreeIPA accounts can co-exist, so it may be useful to create a few Keycloak-only users just in case the FreeIPA connection goes down.

In addition to the "Test" buttons in the Keycloak web-UI, one can always go directly to the Keycloak and FreeIPA web-UIs to do some "debugging".

  • The FreeIPA login page should be at `https://<freeipa-base-url/ipa/ui/ (be sure to logout from any current sessions). This can be useful in verifying that a given username and password work at all. If FreeIPA does not allow the account, then Keycloak will never see it and the problem is likely inside FreeIPA.

  • The Keycloak login page for a realm should be at: https://<keycloak-base-url>:8080/realms/<realm-name>/account/ (be sure to logout from any current sessions). If Keycloak does not allow the account, then it could be a problem inside Keycloak.

Note

When using Keycloak and FreeIPA, no roles should be assigned in ClusterWare – all roles must be assigned through Keycloak.