The Highstate workspace is used to manage and monitor your system's configuration. You can see a historical view of highstate runs across your infrastructure to get insights and trends related to configuration drift across all minions.

As a systems administrator, you are aware of the importance of keeping your system's configuration in compliance, identifying any systems with failures or changes, and bringing those systems back into compliance as quickly as possible. The Highstate workspace provides valuable insight into your system's configuration and includes tools that can help you manage configuration drift.

What is highstate?

Instead of manually configuring each minion and each application one-by-one, you can use the Automation Config state management system to create state files that you can apply to many minions simultaneously. These state files can include a set of instructions that tell Automation Config which operations to run on the minion and in which order. The state management system also ensures each minion is configured properly and as efficiently as possible.

However, if you have hundreds of state files and thousands of minions, running each state file takes time. A faster, more intuitive way to apply state files to your minions is to run highstate. When you run highstate, the state.highstate function is called. The state.highstate function applies all Salt states outlined in a top.sls file that you create on your Salt master.

Running highstate can be useful in the following scenarios:
  • You are setting up your infrastructure for the first time and need to standardize configurations across your environment.
  • You have critical systems that must remain in compliance. The Schedules workspace in Automation Config can be used to schedule a highstate job that runs once a week to ensure that your machines are configured properly.

The top.sls file maps Salt states to a group of targets and consists of the Salt environment, target criteria, and state files to run. The minions download the top.sls file and attempt to match the expressions in it. If a minion matches the expression, it runs the specified state files. For example, you might organize your top.sls file to apply certain state files to all minions, some to CentOS minions, and others to minions whose IDs contain the string web.

base: 
  '*':
    - vim
    - scripts
  'os:CentOS': 
    - match: grain 
    - centos-config 
  '*web*':
    - apache
    - django

While running highstate, if a configuration has drifted, Automation Config puts minions and applications back into its compliant configuration state. If a configuration must change, Automation Config quickly deploys those changes to the affected minions. You can then use the Highstate workspace to see what changes were made during the last highstate run. Charts and reports in the workspace also help visualize the changes across your infrastructure.

Consider testing your changes in a development environment or staging environment. If your states succeed, you can apply them in your production environment.

You can also run highstate in dry run mode. If you run highstate in dry run mode, Automation Config generates a job return that shows what changes would be made on each minion if the states were applied. To learn more about dry run mode, see How do I create jobs.

Important:

You must create and configure the top.sls file on your Salt master before you run highstate for the first time. See Organizing states in the Salt user guide for more information about Salt states and the top.sls file.

Run highstate

After you configure the top.sls file, you can run your first highstate.

To run highstate:

  1. Click the States tab in the Highstate workspace.
  2. Click Run Highstate.
  3. In the dialog box, select the target to run highstate on.
  4. Click Run Highstate.

After the highstate run is finished, you can view the results of your first highstate run at several levels:

  • Global level: see highstate returns across your environment
  • Target level: see highstate returns by target group
  • Minion level: see highstate returns by minion

Global level

You can see highstate compliance across your environment from the States tab and Minions tabs. This global level helps you assess the overall state of your system's configuration since the latest highstate run.

The Highstate workspace in the SaltStack Config user interface after a recent highstate run

Across all levels, the latest highstate data is pulled from each minion, whether the data is from a recent highstate run or from many highstate runs. You can use the Limit by drop-down menu in the States tab to ensure that the Highstate workspace only displays the most recent data.

The global level includes the following information for all states:

Term Description
State The ID declaration for the state, for example, common-centos-packages.
SLS file The file that contains the state declaration, for example, centos-config.sls.
Minions The numbers of minions that ran the state.
Errors Indicates if the state run failed. The state fail for many reasons. For example:
  • A required file does not exist or is inaccessible due to filesystem permissions.
  • A network resource or service is unavailable.
  • There is a package dependency conflict so the state cannot be applied.

The Comments column provides more details about the cause of the state failure.

Succeeded Indicates the overall percentage of states that succeeded.
Changed Indicates the overall percentage of minions that reported changes for each state.
Changes Indicates if the most recent highstate run resulted in changes.

The Comments column provides more details about the changes that were made.

Avg Duration Average duration for the state run in milliseconds (ms).
Environments The Salt environment that contains the state file, for example, base or dev.
JID The job identification number for the state.highstate run.

If you click a state from the States tab, you can view the most recent highstate run on each of your Salt minions.

All previous highstate runs for the default html page state in SaltStack Config

The following information is provided for each state:

Term Description
Minion ID The ID for the minion that ran the state.
Status Indicates if the state run succeeded or failed.
Changes Indicates if the most recent highstate run resulted in changes.

The Comments column provides more details about the changes that were made.

Environment The Salt environment that contains the state file, for example, base or dev.
Cluster/Master The Salt master associated with the minion.
Duration Duration for the state run in milliseconds (ms).
Comment A message that provides further details about the success or failure of the state run.
JID The job identification number for the state.highstate run.

You can also run an ad-hoc highstate job at the global level:

  1. In Automation Config, open the Highstate workspace.
  2. Click Run Highstate.
  3. In the dialog box, select the target to run highstate on.
  4. Click Run Highstate.

You can monitor the job's progress in the Activity tab of the Highstate workspace. After the job is complete, you can view the updated reports and details of the highstate run at the global, target, or minion levels.

Target level

To view highstate returns at the target level, you can open the Targets workspace. If you click a target, you can see all previous state runs for all minions in the target group.

Previous highstate runs for a CentOS target group in SaltStack Config

The target level provides the following information about each minion in the target group:

Term Description
Minion ID The ID of the minion.
State Runs The number of state runs for a minion.
Errors Indicates if the state run failed. The state might fail for many reasons. For example:
  • A required file does not exist or is inaccessible due to filesystem permissions.
  • A network resource or service is unavailable.
  • There is a package dependency conflict so the state cannot be applied.

The Comments column provides more details about the cause of the state failure.

Succeeded Indicates the overall percentage of states that succeeded.
Changed Indicates the overall percentage of states that resulted in changes.
Changes Indicates if the most recent highstate run resulted in changes.

The Comments column provides more details about the changes that were made.

Duration Duration for the state run in milliseconds (ms).
Environments The Salt environment that contains the state file, for example, base or dev.
JID The job identification number for the state.highstate run.

To run an ad-hoc highstate job at the target level:

  1. In Automation Config, open the Targets workspace.
  2. Select the target group you want to run highstate on.
  3. Click the Highstate tab.
  4. Click Run Highstate.
  5. In the dialog box, confirm the target group you want to run highstate on.
  6. Click Run Highstate.

Minion level

To view highstate returns at the minion level, you can open the Minions tab in the Highstate workspace.

Highstate returns for all minions in a SaltStack Config environment

The minion level provides the following information for all minions:

Term Description
Minion ID The ID of the minion.
State Runs The number of state runs for a minion.
Errors Indicates if the state run failed. The state might fail for several reasons:
  • A required file does not exist or is inaccessible due to filesystem permissions.
  • A network resource or service is unavailable.
  • There is a package dependency conflict so the state cannot be applied.

The Comments column provides more details about the cause of the state failure.

Succeeded Indicates the overall percentage of states that succeeded.
Changed Indicates the overall percentage of states that resulted in changes for the minion.
Changes Indicates if the most recent highstate run resulted in changes.

If you click a minion ID, you can view all previous state runs for a minion.

All state runs for one Salt minion in the Highstate workspace

The following information is provided:

Term Description
Succeeded Indicates if the state run succeeded or failed.
Changes Indicates if the most recent highstate run resulted in changes.
Environment The Salt environment that contains the state file, for example, base or dev.
SLS file The file that contains the state declaration, for example, centos-config.sls.
State The ID declaration for the state, for example, common-centos-packages.
Duration Duration for the state run in milliseconds (ms).
Comment A message that provides further details about the success or failure of the state run.
JID The job identification number for the state.highstate run.

To run an ad-hoc highstate job at the minion level:

  1. In Automation Config, open the Highstate workspace.
  2. Select the minion or minions you want to run highstate on.
  3. Click Run Highstate.
  4. In the dialog box, click Run Highstate.

View highstate activity

The Activity tab in the Highstate workspace is used to monitor the status of highstate runs. The Activity tab gives visibility to many types of events and activities, such as scheduled jobs or ad-hoc jobs.

An interactive chart in the Highstate workspace that shows the number of minions with and without changes over the past 30 days

The interactive chart in the Activity tab includes several reports to provide an overview of previous highstate runs in your Automation Config environment. You can view results from the past 24 hours up to the past 30 days for each report.

Report Description
Minion: average duration Displays the average duration for state runs across all minions, over time.
Minions: changes vs no changes Displays the number of minions that reported changes versus minions that did not report changes, over time.
Minions: success vs failure Displays the numbers of minions that reported successful state runs versus minions that reported failed state runs, over time.
States: changes vs no changes Displays the number of states that resulted in changes versus states that did not result in changes, over time.
States: success vs failure Displays the number of states that succeeded versus states that failed.

The Activity tab has three sections that sort jobs by status:

  • Completed: used to monitor the status of completed highstate jobs.
  • In Progress: used to monitor the status of highstate jobs that are currently running.
  • Upcoming: used to monitor the status of upcoming highstate jobs.

The table in the Activity tab shows information about the job's status, origin, schedule (if available), job, target group, and associated job ID (JID). You can filter the columns as needed.