The Highstate workspace is used to manage and monitor your system's configuration. You can see a historical view of highstate runs across your infrastructure to get insights and trends related to configuration drift across all minions.
As a systems administrator, you are aware of the importance of keeping your system's configuration in compliance, identifying any systems with failures or changes, and bringing those systems back into compliance as quickly as possible. The Highstate workspace provides valuable insight into your system's configuration and includes tools that can help you manage configuration drift.
What is highstate?
Instead of manually configuring each minion and each application one-by-one, you can use the Automation Config state management system to create state files that you can apply to many minions simultaneously. These state files can include a set of instructions that tell Automation Config which operations to run on the minion and in which order. The state management system also ensures each minion is configured properly and as efficiently as possible.
However, if you have hundreds of state files and thousands of minions, running each state file takes time. A faster, more intuitive way to apply state files to your minions is to run highstate. When you run highstate, the state.highstate function is called. The state.highstate function applies all Salt states outlined in a top.sls file that you create on your Salt master.
- You are setting up your infrastructure for the first time and need to standardize configurations across your environment.
- You have critical systems that must remain in compliance. The Schedules workspace in Automation Config can be used to schedule a highstate job that runs once a week to ensure that your machines are configured properly.
The top.sls file maps Salt states to a group of targets and consists of the Salt environment, target criteria, and state files to run. The minions download the top.sls file and attempt to match the expressions in it. If a minion matches the expression, it runs the specified state files. For example, you might organize your top.sls file to apply certain state files to all minions, some to CentOS minions, and others to minions whose IDs contain the string web.
base:
'*':
- vim
- scripts
'os:CentOS':
- match: grain
- centos-config
'*web*':
- apache
- django
While running highstate, if a configuration has drifted, Automation Config puts minions and applications back into its compliant configuration state. If a configuration must change, Automation Config quickly deploys those changes to the affected minions. You can then use the Highstate workspace to see what changes were made during the last highstate run. Charts and reports in the workspace also help visualize the changes across your infrastructure.
Consider testing your changes in a development environment or staging environment. If your states succeed, you can apply them in your production environment.
You can also run highstate in dry run mode. If you run highstate in dry run mode, Automation Config generates a job return that shows what changes would be made on each minion if the states were applied. To learn more about dry run mode, see How do I create jobs.
You must create and configure the top.sls file on your Salt master before you run highstate for the first time. See Organizing states in the Salt user guide for more information about Salt states and the top.sls file.
Run highstate
After you configure the top.sls file, you can run your first highstate.
To run highstate:
- Click the States tab in the Highstate workspace.
- Click Run Highstate.
- In the dialog box, select the target to run highstate on.
- Click Run Highstate.
After the highstate run is finished, you can view the results of your first highstate run at several levels:
- Global level: see highstate returns across your environment
- Target level: see highstate returns by target group
- Minion level: see highstate returns by minion
Global level
You can see highstate compliance across your environment from the States tab and Minions tabs. This global level helps you assess the overall state of your system's configuration since the latest highstate run.
Across all levels, the latest highstate data is pulled from each minion, whether the data is from a recent highstate run or from many highstate runs. You can use the Limit by drop-down menu in the States tab to ensure that the Highstate workspace only displays the most recent data.
The global level includes the following information for all states:
| Term | Description |
|---|---|
| State | The ID declaration for the state, for example, common-centos-packages. |
| SLS file | The file that contains the state declaration, for example, centos-config.sls. |
| Minions | The numbers of minions that ran the state. |
| Errors | Indicates if the state run failed. The state fail for many reasons. For example:
The Comments column provides more details about the cause of the state failure. |
| Succeeded | Indicates the overall percentage of states that succeeded. |
| Changed | Indicates the overall percentage of minions that reported changes for each state. |
| Changes | Indicates if the most recent highstate run resulted in changes. The Comments column provides more details about the changes that were made. |
| Avg Duration | Average duration for the state run in milliseconds (ms). |
| Environments | The Salt environment that contains the state file, for example, base or dev. |
| JID | The job identification number for the state.highstate run. |
If you click a state from the States tab, you can view the most recent highstate run on each of your Salt minions.
The following information is provided for each state:
| Term | Description |
|---|---|
| Minion ID | The ID for the minion that ran the state. |
| Status | Indicates if the state run succeeded or failed. |
| Changes | Indicates if the most recent highstate run resulted in changes. The Comments column provides more details about the changes that were made. |
| Environment | The Salt environment that contains the state file, for example, base or dev. |
| Cluster/Master | The Salt master associated with the minion. |
| Duration | Duration for the state run in milliseconds (ms). |
| Comment | A message that provides further details about the success or failure of the state run. |
| JID | The job identification number for the state.highstate run. |
You can also run an ad-hoc highstate job at the global level:
- In Automation Config, open the Highstate workspace.
- Click Run Highstate.
- In the dialog box, select the target to run highstate on.
- Click Run Highstate.
You can monitor the job's progress in the Activity tab of the Highstate workspace. After the job is complete, you can view the updated reports and details of the highstate run at the global, target, or minion levels.
Target level
To view highstate returns at the target level, you can open the Targets workspace. If you click a target, you can see all previous state runs for all minions in the target group.
The target level provides the following information about each minion in the target group:
| Term | Description |
|---|---|
| Minion ID | The ID of the minion. |
| State Runs | The number of state runs for a minion. |
| Errors | Indicates if the state run failed. The state might fail for many reasons. For example:
The Comments column provides more details about the cause of the state failure. |
| Succeeded | Indicates the overall percentage of states that succeeded. |
| Changed | Indicates the overall percentage of states that resulted in changes. |
| Changes | Indicates if the most recent highstate run resulted in changes. The Comments column provides more details about the changes that were made. |
| Duration | Duration for the state run in milliseconds (ms). |
| Environments | The Salt environment that contains the state file, for example, base or dev. |
| JID | The job identification number for the state.highstate run. |
To run an ad-hoc highstate job at the target level:
- In Automation Config, open the Targets workspace.
- Select the target group you want to run highstate on.
- Click the Highstate tab.
- Click Run Highstate.
- In the dialog box, confirm the target group you want to run highstate on.
- Click Run Highstate.
Minion level
To view highstate returns at the minion level, you can open the Minions tab in the Highstate workspace.
The minion level provides the following information for all minions:
| Term | Description |
|---|---|
| Minion ID | The ID of the minion. |
| State Runs | The number of state runs for a minion. |
| Errors | Indicates if the state run failed. The state might fail for several reasons:
The Comments column provides more details about the cause of the state failure. |
| Succeeded | Indicates the overall percentage of states that succeeded. |
| Changed | Indicates the overall percentage of states that resulted in changes for the minion. |
| Changes | Indicates if the most recent highstate run resulted in changes. |
If you click a minion ID, you can view all previous state runs for a minion.
The following information is provided:
| Term | Description |
|---|---|
| Succeeded | Indicates if the state run succeeded or failed. |
| Changes | Indicates if the most recent highstate run resulted in changes. |
| Environment | The Salt environment that contains the state file, for example, base or dev. |
| SLS file | The file that contains the state declaration, for example, centos-config.sls. |
| State | The ID declaration for the state, for example, common-centos-packages. |
| Duration | Duration for the state run in milliseconds (ms). |
| Comment | A message that provides further details about the success or failure of the state run. |
| JID | The job identification number for the state.highstate run. |
To run an ad-hoc highstate job at the minion level:
- In Automation Config, open the Highstate workspace.
- Select the minion or minions you want to run highstate on.
- Click Run Highstate.
- In the dialog box, click Run Highstate.
View highstate activity
The Activity tab in the Highstate workspace is used to monitor the status of highstate runs. The Activity tab gives visibility to many types of events and activities, such as scheduled jobs or ad-hoc jobs.
The interactive chart in the Activity tab includes several reports to provide an overview of previous highstate runs in your Automation Config environment. You can view results from the past 24 hours up to the past 30 days for each report.
| Report | Description |
|---|---|
| Minion: average duration | Displays the average duration for state runs across all minions, over time. |
| Minions: changes vs no changes | Displays the number of minions that reported changes versus minions that did not report changes, over time. |
| Minions: success vs failure | Displays the numbers of minions that reported successful state runs versus minions that reported failed state runs, over time. |
| States: changes vs no changes | Displays the number of states that resulted in changes versus states that did not result in changes, over time. |
| States: success vs failure | Displays the number of states that succeeded versus states that failed. |
The Activity tab has three sections that sort jobs by status:
- Completed: used to monitor the status of completed highstate jobs.
- In Progress: used to monitor the status of highstate jobs that are currently running.
- Upcoming: used to monitor the status of upcoming highstate jobs.
The table in the Activity tab shows information about the job's status, origin, schedule (if available), job, target group, and associated job ID (JID). You can filter the columns as needed.
