Using the Highstate workspace

The Highstate workspace is used to manage and monitor your system’s configuration. You can see a historical view of highstate runs across your infrastructure to get insights and trends related to configuration drift across all minions.

As a systems administrator, you are aware of the importance of keeping your system’s configuration in compliance, identifying any systems with failures or changes, and bringing those systems back into compliance as quickly as possible. The Highstate workspace provides valuable insight into your system’s configuration and includes tools that can help you manage configuration drift.

What is highstate

Instead of manually configuring each minion and each application one-by-one, you can use the Tanzu Salt state management system to create state files that you can apply to many minions simultaneously. These state files can include a set of instructions that tell Tanzu Salt which operations to run on the minion and in which order. The state management system also ensures each minion is configured properly and as efficiently as possible.

However, if you have hundreds of state files and thousands of minions, running each state file takes time. A faster, more intuitive way to apply state files to your minions is to run highstate. When you run highstate, the state.highstate function is called. The state.highstate function applies all Salt states outlined in a top.sls file that you create on your Salt master.

Running highstate can be useful in the following scenarios:

• You are setting up your infrastructure for the first time and need to standardize configurations across your environment.
• You have critical systems that must remain in compliance. The Schedules workspace in Tanzu Salt can be used to schedule a highstate job that runs once a week to ensure that your machines are configured properly.

The top.sls file maps Salt states to a group of targets and consists of the Salt environment, target criteria, and state files to run. The minions download the top.sls file and attempt to match the expressions in it. If a minion matches the expression, it runs the specified state files. For example, you might organize your top.sls file to apply certain state files to all minions, some to CentOS minions, and others to minions whose IDs contain the string web.

base:
  '*':
    - vim
    - scripts
  'os:CentOS':
    - match: grain
    - centos-config
  '*web*':
    - apache
    - django

While running highstate, if a configuration has drifted, Tanzu Salt puts minions and applications back into its compliant configuration state. If a configuration must change, Tanzu Salt quickly deploys those changes to the affected minions. You can then use the Highstate workspace to see what changes were made during the last highstate run. Charts and reports in the workspace also help visualize the changes across your infrastructure.

Consider testing your changes in a development environment or staging environment. If your states succeed, you can apply them in your production environment.

You can also run highstate in dry run mode. If you run highstate in dry run mode, Tanzu Salt generates a job return that shows what changes would be made on each minion if the states were applied. To learn more about dry run mode, see How do I create jobs.

Important:
You must create and configure the top.sls file on your Salt master before you run highstate for the first time. See Organizing states in the Salt user guide for more information about Salt states and the top.sls file.

Run highstate

After you configure the top.sls file, you can run your first highstate.

To run highstate:

1. Click the States tab in the Highstate workspace.
2. Click Run Highstate.
3. In the dialog box, select the target to run highstate on.
4. Click Run Highstate.

After the highstate run is finished, you can view the results of your first highstate run at several levels:

• Global level: see highstate returns across your environment
• Target level: see highstate returns by target group
• Minion level: see highstate returns by minion

Global level

You can see highstate compliance across your environment from the States tab and Minions tabs. This global level helps you assess the overall state of your system’s configuration since the latest highstate run.

Across all levels, the latest highstate data is pulled from each minion, whether the data is from a recent highstate run or from many highstate runs. You can use the Limit by drop-down menu in the States tab to ensure that the Highstate workspace only displays the most recent data.

The global level includes the following information for all states:

Term	Description
State	The ID declaration for the state, for example, common-centos-packages.
SLS file	The file that contains the state declaration, for example, centos-config.sls.
Minions	The numbers of minions that ran the state.
Errors	Indicates if the state run failed. The state fail for many reasons. For example: A required file does not exist or is inaccessible due to filesystem permissions. A network resource or service is unavailable. There is a package dependency conflict so the state cannot be applied. The Comments column provides more details about the cause of the state failure.
Succeeded	Indicates the overall percentage of states that succeeded.
Changed	Indicates the overall percentage of minions that reported changes for each state.
Changes	Indicates if the most recent highstate run resulted in changes. The Comments column provides more details about the changes that were made.
Avg Duration	Average duration for the state run in milliseconds (ms).
Environments	The Salt environment that contains the state file, for example, base or dev.
JID	The job identification number for the state.highstate run.

If you click a state from the States tab, you can view the most recent highstate run on each of your Salt minions.

The following information is provided for each state:

Term	Description
Minion ID	The ID for the minion that ran the state.
Status	Indicates if the state run succeeded or failed.
Changes	Indicates if the most recent highstate run resulted in changes. The Comments column provides more details about the changes that were made.
Environment	The Salt environment that contains the state file, for example, base or dev.
Cluster/Master	The Salt master associated with the minion.
Duration	Duration for the state run in milliseconds (ms).
Comment	A message that provides further details about the success or failure of the state run.
JID	The job identification number for the state.highstate run.

You can also run an ad-hoc highstate job at the global level:

1. In Tanzu Salt, open the Highstate workspace.
2. Click Run Highstate.
3. In the dialog box, select the target to run highstate on.
4. Click Run Highstate.

You can monitor the job’s progress in the Activity tab of the Highstate workspace. After the job is complete, you can view the updated reports and details of the highstate run at the global, target, or minion levels.

Target level

To view highstate returns at the target level, you can open the Targets workspace. If you click a target, you can see all previous state runs for all minions in the target group.

The target level provides the following information about each minion in the target group:

Term	Description
Minion ID	The ID of the minion.
State Runs	The number of state runs for a minion.
Errors	Indicates if the state run failed. The state might fail for many reasons. For example: A required file does not exist or is inaccessible due to filesystem permissions. A network resource or service is unavailable. There is a package dependency conflict so the state cannot be applied. The Comments column provides more details about the cause of the state failure.
Succeeded	Indicates the overall percentage of states that succeeded.
Changed	Indicates the overall percentage of states that resulted in changes.
Changes	Indicates if the most recent highstate run resulted in changes. The Comments column provides more details about the changes that were made.
Duration	Duration for the state run in milliseconds (ms).
Environments	The Salt environment that contains the state file, for example, base or dev.
JID	The job identification number for the state.highstate run.

To run an ad-hoc highstate job at the target level:

1. In Tanzu Salt, open the Targets workspace.
2. Select the target group you want to run highstate on.
3. Click the Highstate tab.
4. Click Run Highstate.
5. In the dialog box, confirm the target group you want to run highstate on.
6. Click Run Highstate.

Minion level

To view highstate returns at the minion level, you can open the Minions tab in the Highstate workspace.

The minion level provides the following information for all minions:

Term	Description
Minion ID	The ID of the minion.
State Runs	The number of state runs for a minion.
Errors	Indicates if the state run failed. The state might fail for many reasons. For example: A required file does not exist or is inaccessible due to filesystem permissions. A network resource or service is unavailable. There is a package dependency conflict so the state cannot be applied. The Comments column provides more details about the cause of the state failure.
Succeeded	Indicates the overall percentage of states that succeeded.
Changed	Indicates the overall percentage of states that resulted in changes for the minion.
Changes	Indicates if the most recent highstate run resulted in changes.

If you click a minion ID, you can view all previous state runs for a minion.

The following information is provided:

Term	Description
Succeeded	Indicates if the state run succeeded or failed.
Changes	Indicates if the most recent highstate run resulted in changes.
Environment	The Salt environment that contains the state file, for example, base or dev.
SLS file	The file that contains the state declaration, for example, centos-config.sls.
State	The ID declaration for the state, for example, common-centos-packages.
Duration	Duration for the state run in milliseconds (ms).
Comment	A message that provides further details about the success or failure of the state run.
JID	The job identification number for the state.highstate run.

To run an ad-hoc highstate job at the minion level:

1. In Tanzu Salt, open the Highstate workspace.
2. Select the minion or minions you want to run highstate on.
3. Click Run Highstate.
4. In the dialog box, click Run Highstate.

View highstate activity

The Activity tab in the Highstate workspace is used to monitor the status of highstate runs. The Activity tab gives visibility to many types of events and activities, such as scheduled jobs or ad-hoc jobs.

The interactive chart in the Activity tab includes several reports to provide an overview of previous highstate runs in your Tanzu Salt environment. You can view results from the past 24 hours up to the past 30 days for each report.

Report	Description
Minion: average duration	Displays the average duration for state runs across all minions, over time.
Minions: changes vs no changes	Displays the number of minions that reported changes versus minions that did not report changes, over time.
Minions: success vs failure	Displays the numbers of minions that reported successful state runs versus minions that reported failed state runs, over time.
States: changes vs no changes	Displays the number of states that resulted in changes versus states that did not result in changes, over time.
States: success vs failure	Displays the number of states that succeeded versus states that failed.

The Activity tab has three sections that sort jobs by status:

• Completed: used to monitor the status of completed highstate jobs.
• In Progress: used to monitor the status of highstate jobs that are currently running.
• Upcoming: used to monitor the status of upcoming highstate jobs.

The table in the Activity tab shows information about the job’s status, origin, schedule (if available), job, target group, and associated job ID (JID). You can filter the columns as needed.