Migrate to RHEL 8/9

After you've created and prepared your Salt infrastructure for the new RHEL 8/9 systems, you can perform these migration steps to complete your upgrade to RHEL 8/9.

Prepare and perform migration

Stop the RaaS service on both the RHEL 7 and RHEL 8/9 systems.
Copy the gz backup file from the old server to the new server. The gz file must be stored in the /var/lib/pgsql directory with ownership=postgres:postgres.

From the postgres user, run these commands to remove the database directory:

su - postgres
psql -U postgres
drop database raas_43cab1f4de604ab185b51d883c5c5d09

Create an empty database and verify user:

create database raas_43cab1f4de604ab185b51d883c5c5d09
\du – should display users for postgres and salteapi

Copy the /etc/raas/pki/.raas.key and /etc/raas/pki/.instance_id files from the old RaaS server to the new RaaS Server.
Run the upgrade commands for the new Postgresql database:
```
su – raas
raas -l debug upgrade
```

You can now start up the raas service on the new rhel9-raas server. You can also access the SaltStack Config UI in your browser. Next, you must configure the master plugin on the new RHEL 8/9 Salt Master.

Configure the Master Plugin on the new Salt Master

Peform these steps on your new rhel9-master node.

Log in to your Salt master and verify the /etc/salt/master.d directory exists, or create it.
Generate the master configuration settings.
Caution: If you want to preserve your settings when upgrading your installation, make a backup of your existing Master Plugin configuration file before running this step. Then copy relevant settings from your existing configuration to the newly generated file.
```
sudo sseapi-config --all > /etc/salt/master.d/raas.conf
```
If running this command causes an error, it might be related to the method you used when initially installing Salt. If you installed Salt through the SaltStack Config installer, your installation likely includes an offline package, called the Salt Crystal, that requires special upgrade instructions. For more information, see the Troubleshooting page.

Edit the generated raas.conf file and update the values as follows to validate the certificate the API (RaaS) uses and set its IP address.


Value	Description
`sseapi_ssl_validate_cert`	Validates the certificate the API (RaaS) uses. The default is `True`. If you are using your own CA-issued certificates, set this value to `True` and configure the `sseapi_ssl_ca`, `sseapi_ssl_cert`, and `sseapi_ssl_cert:` settings. Otherwise, set this to `False` to not validate the certificate. sseapi_ssl_validate_cert:False
`sseapi_server`	HTTP IP address of your RaaS node, for example, `http://example.com`, or `https://example.com` if SSL is enabled.
`sseapi_command_age_limit`	Sets the age (in seconds) after which old, potentially stale jobs are skipped. For example, to skip jobs older than a day, set it to: sseapi_command_age_limit:86400 Skipped jobs continue to exist in the database and display with a status of `Completed` in the SaltStack Config user interface. Some environments might need the Salt master to be offline for long periods of time and will need the Salt master to run any jobs that were queued after it comes back online. If this applies to your environment, set the age limit to `0`.
`sseapi_windows_minion_deploy_delay`	Sets a delay to allow all requisite Windows services to become active. The default value is 180 seconds.
`sseapi_linux_minion_deploy_delay`	Sets a delay to allow all requisite Linux services to become activate. The default value is 90 seconds.
`sseapi_local_cache`	Sets the length of time that certain data is cached locally on each salt master. The default value is 300 seconds (5 mins).

OPTIONAL: This step is necessary for manual installations only. To verify you can connect to SSL before connecting the Master Plugin, edit the generated raas.conf file to update the following values. If you do not update these values, the Master Plugin uses the default generated certificate.


Value	Description
`sseapi_ssl_ca`	The path to a CA file.
`sseapi_ssl_cert`	The path to the certificate. The default value is `/etc/pki/raas/certs/localhost.crt`.
`sseapi_ssl_key`	The path to the certificate’s private key. The default value is `/etc/pki/raas/certs/localhost.key`.
`id`	Comment this line out by adding a `#` at the beginning. It is not required.

OPTIONAL: Update performance-related settings. For large or busy environments, you can improve the performance of the communications between the Salt master and SaltStack Config by adjusting the following settings.
- Enable event queuing (available in salt 2019.2.2 and later). Events can be queued on the Salt master and sent to the event returner in a batched fashion using a single transaction for multiple events. By default, this queuing mechanism is disabled. To enable event queuing, set the following in your Salt Master Plugin configuration file:
```
event_return_queue:2500
event_return_queue_max_seconds:5
```
  The suggested maximum event queue size is 2500 as shown. The queue will be flushed, forwarding events to the event returner, when the queue is full. A lower value may be a better fit for smaller or less busy environments.
  In some cases, the Salt event bus may not busy enough to cause the queue to regularly reach its maximum size. Setting event_return_queue_max_seconds will cause the queue to be flushed when the oldest event in the queue is older than the configured value regardless of how many events are in the queue.
- Enable and configure the eventqueue and rpcqueue engines:
  These engines offload some communications with SaltStack Config from performance-critical code paths to dedicated processes. While the engines are waiting to communicate withSaltStack Config, payloads are stored in the Salt master’s local filesystem so the data can persist across restarts of the Salt master.
  To enable the engines, uncomment the following settings in the Salt Master Plugin configuration file (raas.conf):
```
engines:
  - sseapi: {}
  - eventqueue: {}
  - rpcqueue: {}
  - jobcompletion: {}
  - keyauth: {}
```
  To configure the eventqueue engine, uncomment and update the following settings:
```
sseapi_event_queue:
  name: sseapi-events
  strategy: always
  push_interval: 5
  batch_limit: 2000
  age_limit: 86400
  size_limit: 35000000
  vacuum_interval: 86400
  vacuum_limit: 350000
  forward: []
```
  The queue parameters can be adjusted with consideration to how they work together. For example, assuming an average of 400 events per second on the Salt event bus, the settings shown above allow for about 24 hours of queued event traffic to collect on the Salt master before the oldest events are discarded due to size or age limits.
  To configure the rpcqueue engine, uncomment and update the following settings:
```
sseapi_rpc_queue:
    name: sseapi-rpc
    strategy: always
    push_interval: 5
    batch_limit: 500
    age_limit: 3600
    size_limit: 360000
    vacuum_interval: 86400
    vacuum_limit: 100000
```
- Enable load caching:
```
sseapi_local_cache:
    load:3600
```
  Note: If the rpcqueue engine is enabled, load caching must also be enabled in order for the Salt master to handle jobs correctly.
- Limit minion grains payload sizes:
```
sseapi_max_minion_grains_payload:2000
```
- Enable skipping jobs that are older than a defined time (in seconds). For example, use 86400 to set it to skip jobs older than a day. When set to 0, this feature is disabled:
```
sseapi_command_age_limit:0
```
  Note:
  This is useful during upgrade, to prevent old commands stored in the database from running unexpectedly.
Together, event queuing in Salt and the queuing engines, load caching, grains payload size limit, and command age limit in the Salt Master Plugin increase the throughput and reduce the latency of communications between the Salt master and SaltStack Config in the most performance-sensitive code paths.
Restart the master service.
```
sudo systemctl restart salt-master
```
OPTIONAL: You might want to run a test job to ensure the Master Plugin is now enabling communication between the master and the RaaS node.
```
salt -v '*' test.ping
```

The RHEL 8/9 Master now appears on the Master Keys page.

Caution: Do not accept the master key at this point.

Configure the Minion Agent

Follow these steps to configure the minion agent on the rhel9-master node to point to itself.

SSH into the rhel9-master node and browse to the /etc/salt/minion.d directory.
Edit the minion.conf file and change the master setting to master:localhost.
Browse to the /etc/salt/pki/minion directory and delete the minion_master.pub file.
Restart the salt-minion service using
```
systemctl restart salt-minion
```
View and accept the minion key on the rhel9-master by running:
```
salt-key
salt-key -A
```
In SaltStack Config, navigate to Administration > Master Keys and accept the Master Key.
The RHEL8/9 Master should now appear on the Targets page.
SSH into the RHEL7 Master and delete the key for the rhel9-master minion.

Migrate Salt-Minion systems

There are many ways to migrate your managed systems. If you already have a process set up, follow that process. If not, use these instructions to migrate your salt-minions from an old Salt Master to a new Salt Master.

Note: These steps do not apply to multi-master systems.

Create an orchestration file. For example,

# Orchestration to move Minions from one master to another
# file: /srv/salt/switch_masters/init.sls
{% import_yaml 'switch_masters/map.yaml' as mm %}
{% set minions = mm['minionids'] %}

{% if minions %}
{% for minion in minions %}
move_minions_{{ minion }}:
  salt.state:
    - tgt: {{ minion }}
    - sls:
      - switch_masters.move_minions_map

{% endfor %}    
{% else %}
no_minions:
  test.configurable_test_state:
    - name: No minions to move
    - result: True 
    - changes: False 
    - comment: No minions to move
{% endif %}

remove_minions:
  salt.runner:
    - name: manage.down
    - removekeys: True 

# map file for moving minions
# file: /srv/salt/switch_masters/map.yaml
newsaltmaster: <new_ip_address>
oldsaltmaster: <old_ip_address>
minionids:
  - minion01
  - minion02
  - minion03
state to switch minions from one master to another
# file: /srv/salt/swith_masters/move_minions_map.sls
{% set minion = salt['grains.get']('os') %}
# name old master and set new master ip address
{% import_yaml 'switch_masters/map.yaml' as mm %}
{% set oldmaster = mm['oldsaltmaster'] %}
{% set newmaster = mm['newsaltmaster'] %}

# remove minion_master.pub key
{% if minion == 'Windows' %}
remove_master_key:
  file.absent:
    - name: c:\ProgramData\Salt Project\Salt\conf\pki\minion\minion_master.pub

change_master_assignment:
  file.replace:
    - name: c:\ProgramData\Salt Project\Salt\conf\minion.d\minion.conf 
    - pattern: 'master: {{oldmaster}}'
    - repl: 'master: {{newmaster}}'
    - require:
      - remove_master_key
{% else %}
remove_master_key:
  file.absent:
    - name: /etc/salt/pki/minion/minion_master.pub

# modify minion config file
change_master_assignment:
  file.replace:
    - name: /etc/salt/minion.d/minion.conf 
    - pattern: 'master: {{oldmaster}}'
    - repl: 'master: {{newmaster}}'
    - require:
      - remove_master_key
{% endif %}
# restart salt-minion
restart_salt_minion:
  service.running:
    - name: salt-minion 
    - require:
      - change_master_assignment
    - watch:
      - change_master_assignment

Create a map.yaml file that includes (see the above code example):
1. <old salt master> IP address/FQDN
2. <new salt master> IP address/FQDN
3. List of saltminionIDs to be moved.
Create a state file (see the above code example) to process the migration. For example, move_minions_map.sls.
Add these files to a directory (For example, /srv/salt/switch_masterson the RHEL7 Salt Master.
Run the orchestration file on the RHEL7 Salt Master. This results in some errors as the Salt minion service restarts, and does not come back online for the RHEL7 Salt Master.
Monitor the progress in SaltStack Config. Accept the migrate Salt minion IDs as they populate in the UI.
After all the systems have migrated, run a test.ping job against them to verify that everything is communicating properly.

Migrate Existing Files

This process is completely dependent on how your organization creates, stores, and manages your state and configuration file. The most common use cases are outlined below as reference.

Use Case 1: SaltStack Config file server

In this use case, your SaltStack Config files are stored in the Postgres database and appear in the SaltStack Config UI.

During the Postgres database restore, these files are recovered and migrated. There are no additional steps you need to take to migrate these files to your RHEL8/9 environment.

Use Case 2: Github/Gitlab file server

In this use case, your SaltStack Config state and configuration files are stored in Github/Gitlab/Bitbucket or some other code version-control system.

Since these files are stored in a third party tool, you need to configure your new RHEL8/9 Master to connect to your repository system. This configuration will mirror your RHEL7 repository configuration.

Use Case 3: Local file roots of Salt Master

In this use case, you SaltStack Config are stored on a local file server directory on the Salt Master.

To migrate these files to your RHEL8/9 Master, copy the appropriate directories from your RHEL7 Master to your RHEL8/9 Master.

Files are stored in /srv/salt and /srv/pillar for state files and pillar files, respectively.
Perform a secure copy of these directories from your RHEL7 Master to your RHEL8/9 Master using a secure copy tool such as winscp or the command line.
Refresh pillar date using Salt \* saltutil.refresh_pillar