Perform an In-Place-Based and Photon OS Upgrade on vSphere

In-place-based upgrade updates the blockchain version and the photon operating system.

Depending on your upgrade requirements, you can upgrade all the components simultaneously or separately.

With in-place-based upgrade, you cannot revert to the pre-upgraded version of the product. If there is an upgrade failure, the default automatic rollback parameter value, which is set to True reverts to the latest snapshot of your blockchain nodes. The snapshot is created during the upgrade process.

Prerequisites

Familiarize yourself with the upgrade workflow. See Considerations for Upgrading VMware Blockchain Nodes on vSphere.
Verify that you have a VMware Blockchain version deployed. See Deploy VMware Blockchain Nodes Using VMware Blockchain Orchestrator on vSphere.
Verify that you have adequate storage space to accommodate the existing and new versions of VMware Blockchain Orchestrator.
Verify that you have the private operator key for the upgrade process. See Generate Operator Keys.

Procedure

SSH into the VMware Blockchain Orchestrator appliance.
Enter the login credentials for the blockchain user account.
Navigate to the /home/blockchain directory.
Verify that the provisioning and configuration service containers are up and running using the docker ps -a command.
If the provisioning and configuration service containers are not running, run the following command:
```
CONFIG_SERVICE_IP=<orchestrator-ip-address> 
docker-compose -f docker-compose-orchestrator-prereqs.yml up
```
The <orchestrator-ip-address> is the VMware Blockchain Orchestrator appliance IP address running the configuration service container.

Create a directory for the descriptor files.

See the sample directory structure.

Option	Description
/home/blockchain/descriptors	Directory stores the infrastructure and deployment descriptor JSON files. VMware Blockchain Orchestrator uses the parameter values set in the infrastructure and deployment descriptor files during provisioning.
/home/blockchain/output	Directory stores the deployment details, such as IP address, Replica and Client node names, blockchain ID, and consortium ID.
INFRA_DESC_FILENAME=infrastructure_descriptor.json	File provides infrastructure details for deployment that VMware Blockchain Orchestrator uses to prescribe the parameters and connect to vCenter Server.
DEPLOY_DESC_FILENAME=deployment_descriptor.json	File provides deployment details for Replica and Client nodes. If you deploy Full Copy Client nodes, the deployment details are included in this file. For example, the file also consists of zone names, Client group names, and IP addresses.

Copy the old VMware Blockchain Orchestrator output JSON file to the new VMware Blockchain Orchestrator deployment.
Set read and write permissions for the new VMware Blockchain Orchestrator deployment.

Set the environment variables for the blockchain upgrade.

Open the upgrade folder.
```
cd /home/blockchain/upgrade/
```

List the parameter values and associated description for the upgrade script.

'-j' or '--deploymentInfo' --> The absolute json file path that contains the blockchain node IP/username/password and other details
'-i' or '--infraDescFile' --> The absolute json file Infrastructure descriptor path, to execute reconfiguration workflow
'-d' or '--deployDescFile' --> The absolute json file Deployment descriptor path, to execute reconfiguration workflow
'-o' or '--outputFolder' --> The path of output folder. Reconfiguration output json file is generated in this path
'-s' or '--snapshotName' --> Snapshot name to create vm snapshot.
'-r' or '--autoRollback' --> Auto rollback flag, used to rollback to latest snapshot in case of update failure. Default value for this flag is 'True'
'-a' or '--action' --> Action/operation to be performed. For blockchain upgrade, action should be 'update'
'-t' or '--updateType' --> Type of upgrade process. For blockchain with os upgrade updateType should be 'blockchain_with_os'.There is no need to pass this variable explicitly. This is the default upgrade type
'-f' or '--force' --> To force the upgrade without os, users needs to pass this variable explicitly with value as 'forceWithOutOS'
'-c' or '--dockerComposeYaml' --> The absolute path of docker-compose-orchestartor.yml file. Default value for this option is /home/blockchain/orchestrator-runtime/docker-compose-orchestrator.yml
'-p' or '--osPatchImage' --> Use this flag only if you want to override the default os patch image associated with the blockchain version. Else please ignore this for blockchain_with_os upgrade type

Update the component version information in the UpgradeVersionDetails.json file.

The following component versions must be updates.

Blockchain
Fluentd
Jager-agent
Wavefront
Telegraf

Sample component version updates.

{
 "blockchain":{
    "oldVersion":"1.7.0.0.55",
    "newVersion":"1.8.0.0.53"
  },
  "fluentd":{
    "oldVersion":"fluentd:1.1",
    "newVersion":"fluentd:1.2"
  },
  "jaeger-agent":{
    "oldVersion":"jaeger-agent:1.22",
    "newVersion":"jaeger-agent:1.22"
  },
  "wavefront":{
    "oldVersion":"wavefront-proxy:10.12",
    "newVersion":"wavefront-proxy:10.12"
  },
  "telegraf":{
    "oldVersion":"telegraf:1.18.3",
    "newVersion":"telegraf:1.18.3"
  }
}

Maintain SSH connection throughout the upgrade script execution.
The current ideal session timeout duration is 15 minutes or 900 seconds. You can increase the session timeout to avoid SSH timeout. The updated session timeout duration is reflected in a new SSH connection.
```
sed -i "s@ClientAliveInterval 900@ClientAliveInterval 5400@" /etc/ssh/sshd_config systemctl reload sshd
```
If any interruptions occur, revert and restart the upgrade process.

Run the UpgradeController.py script to upgrade the blockchain and photon OS versions.

#upgrade blockchain with default os patch image associated with the release
python UpdateController.py -j /home/blockchain/output/provisioning_output.json -i /home/blockchain/descriptors/reconfig_infra.json -a update -d /home/blockchain/descriptors/reconfig_deploy.json -o /home/blockchain/output -s snapshot_before_upgrade_sep29
 
#Upgrade blockhain with customized os version (command with -p)
python UpdateController.py -j /home/blockchain/output/provisioning_output.json -i /home/blockchain/descriptors/reconfig_infra.json -a update -p capupdaterepo:updaterepo1.5.15  -d /home/blockchain/descriptors/reconfig_deploy.json -o /home/blockchain/output -s snapshot_before_upgrade_sep29

Upgrade script prompts for the current and new operator EdDSA private key.
Paste the private key and press Ctrl+d in a new line for the upgrade process to continue.

Verify that the health status of the latest blockchain nodes.

python UpdateController.py -j /home/blockchain/output/provisioning_output.json -i /home/blockchain/descriptors/reconfig_infra.json -d /home/blockchain/descriptors/reconfig_deploy.json -a health_check

Upgrade the USB HSM manager and OS RPMS.

Update the nodePassword, zoneName, and vmId parameter values in the deployment descriptor file USB HSM Manager section.

hsmManagers": [
        {
            "zoneName": "zone-4",
            "providedIp": "10.10.18.10",
            "esxiHostName": "esxihost",
            "hsmPassword": "password",
            "nodePassword": "<passowrd>",
            "vmId": "usb-hsm-manager-appliance"
        }
    ],

Run the UpdateController.py script to initiate the upgrade process with USB HSM Manager.

#upgrade blockchain with default os patch image associated with the release
python UpdateController.py -j /home/blockchain/output/provisioning_output.json -i /home/blockchain/descriptors/reconfig_infra.json -a update -d /home/blockchain/descriptors/reconfig_deploy.json -o /home/blockchain/output -s snapshot_before_upgrade_sep29
 
#Upgrade blockhain with customized os version (command with -p)
python UpdateController.py -j /home/blockchain/output/provisioning_output.json -i /home/blockchain/descriptors/reconfig_infra.json -a update -p capupdaterepo:updaterepo1.5.15  -d /home/blockchain/descriptors/reconfig_deploy.json -o /home/blockchain/output -s snapshot_before_upgrade_sep29

(Optional) Set up the existing blockchain node as an LDAP client.

The parameter values must be replaced as per the Windows ADDS configuration details.

Run the LDAP script.

#!/bin/bash

usageFunction()
{
   echo ""
   echo "Usage: $0 uri_val=<URI> base_dn=<BASE_DN> bind_dn=<BIND_DN> bind_pw=<BIND_PW> ldap_groupName=<LDAP_GROUP_NAME> login_shell=<LOGIN_SHELL> home_directory=<HOME_DIRECTORY>"

   echo ""  
  
   echo -e "\t<URI> : List of URIs of Domain Controllers"
   echo -e "\t<BASE_DN> : Domain Name of Domain Controller"
   echo -e "\t<BIND_DN> : Bind Domain Name of User which has admin access in Domain Controller"
   echo -e "\t<BIND_PW> : Bind password of the bind user"
   echo -e "\t<LDAP_GROUP_NAME> : Group Name which has to be provided sudo access"
   echo -e "\t<LOGIN_SHELL> : Login shell in blockchain node of AD users"
   echo -e "\t<HOME_DIRECTORY> : Home Directory in blockchain node for AD users"
   exit 1
}

for ARGUMENT in "$@"
do
   KEY=$(echo $ARGUMENT | cut -f1 -d=)

   KEY_LENGTH=${#KEY}
   VALUE="${ARGUMENT:$KEY_LENGTH+1}"

   export "$KEY"="$VALUE"
done

if [ -z "$uri_val" ] || [ -z "$base_dn" ] || [ -z "$ldap_groupName" ] || [ -z "$login_shell" ] || [ -z "$home_directory" ]
then
   echo "";
   echo "Error while executing script: Mandatory parameters are missing";
   usageFunction
fi

#######################################
####### Set up open ldap client #######
#######################################
# Add nslcd group
echo 'Adding nslcd group...'
groupadd nslcd

#Edit ldap config
echo 'Editing ldap config...'
mv /etc/openldap/ldap.conf /etc/openldap/ldap.conf.backup
echo "#
# LDAP Defaults
#

BASE $base_dn
URI $uri_val

#SIZELIMIT 12
#TIMELIMIT 15
#DEREF never" > /etc/openldap/ldap.conf

chmod 644 /etc/openldap/ldap.conf

# Edit nslcd config
echo 'Editing nslcd config...'
mv /etc/nslcd.conf /etc/nslcd.conf.backup
echo "# This is the configuration file for the LDAP nameservice
# switch librarys nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.

# The user and group nslcd should run as.
uid nslcd
gid ldap

# The uri pointing to the LDAP server to use for name lookups.
# Multiple entries may be specified. The address that is used
# here should be resolvable without using LDAP (obviously).
# Note: %2f encodes the '/' used as directory separator
uri $uri_val

# The distinguished name of the search base.
base $base_dn

# The distinguished name to bind to the server with.
# Optional: default is to bind anonymously.
binddn $bind_dn

# The credentials to bind with.
# Optional: default is no credentials.
# Note that if you set a bindpw you should check the permissions of this file.
bindpw $bind_pw

# The default search scope.
scope sub

# Alternative mappings for Active Directory
# (replace the SIDs in the objectSid mappings with the value for your domain)
filter passwd (&(objectClass=user)(objectClass=person)(!(objectClass=computer)))
map passwd uid cn
map passwd homeDirectory $home_directory
map passwd gecos displayName
map passwd loginShell $login_shell
filter group (|(objectClass=group)(objectClass=person))" > /etc/nslcd.conf
chmod 600 /etc/nslcd.conf

# Edit nsswitch config
echo 'Editing nsswitch config...'
mv /etc/nsswitch.conf /etc/nsswitch.conf.backup
echo "# Begin /etc/nsswitch.conf

passwd: files ldap
group: files ldap
shadow: files ldap

hosts: files resolve dns
networks: files

protocols: files
services: files
ethers: files
rpc: files
# End /etc/nsswitch.conf" > /etc/nsswitch.conf

chmod 644 /etc/nsswitch.conf

# Edit pam.d files
echo 'Editing pamd files...'
mv /etc/pam.d/systemd-user /etc/pam.d/systemd-user.backup
echo "# This file is part of systemd.
#
# Used by systemd --user instances.
account sufficient pam_unix.so
account sufficient pam_ldap.so
session required pam_loginuid.so
session optional pam_keyinit.so force revoke
session optional pam_systemd.so" > /etc/pam.d/systemd-user

chmod 644 /etc/pam.d/systemd-user

mv /etc/pam.d/system-account /etc/pam.d/system-account.backup
echo "# Begin /etc/pam.d/system-account
account required pam_unix.so broken_shadow
#account sufficient pam_succeed_if.so uid < 1000 quiet
account sufficient pam_ldap.so
account required pam_permit.so
# End /etc/pam.d/system-account" > /etc/pam.d/system-account

chmod 644 /etc/pam.d/system-account

mv /etc/pam.d/system-auth /etc/pam.d/system-auth.backup
echo "# Begin /etc/pam.d/system-auth
auth sufficient pam_unix.so
auth sufficient pam_ldap.so
auth required pam_deny.so
# End /etc/pam.d/system-auth" > /etc/pam.d/system-auth

chmod 644 /etc/pam.d/system-auth

mv /etc/pam.d/system-password /etc/pam.d/system-password.backup
echo "# Begin /etc/pam.d/system-password
# use sha512 hash for encryption, use shadow, and try to use any previously
# defined authentication token (chosen password) set by any prior module
password requisite pam_cracklib.so
password sufficient pam_unix.so sha512 shadow try_first_pass
password sufficient pam_ldap.so
password required pam_deny.so
# End /etc/pam.d/system-password" > /etc/pam.d/system-password

chmod 644 /etc/pam.d/system-password

mv /etc/pam.d/system-session /etc/pam.d/system-session.backup
echo "# Begin /etc/pam.d/system-session
session required pam_mkhomedir.so skel=/etc/skel/ umask=0022
session sufficient pam_ldap.so
session sufficient pam_unix.so
session required pam_limits.so
session optional pam_systemd.so
session optional pam_loginuid.so
# End /etc/pam.d/system-session" > /etc/pam.d/system-session

chmod 644 /etc/pam.d/system-session

# Restart services
echo 'restarting nslcd...'
systemctl restart nslcd
echo 'restarting sshd...'
systemctl restart sshd

#Add groups to Sudoers file config
echo 'Editing sudoers config...'
cp /etc/sudoers /etc/sudoers.backup
echo "%$ldap_groupName ALL=(ALL:ALL) ALL" >> /etc/sudoers

#Verify AD config
echo 'Verifying ldap config...'

mkdir -p /config/system

echo "#!/bin/bash
ldapsearch -H $uri_val -D $bind_dn -b $base_dn -w $bind_pw
if [ $? -eq 0 ]; then
echo ldapsearch was successful >> /config/system/DirectoryServiceSetupLogs.log
else
echo ldapsearch was failed >> /config/system/DirectoryServiceSetupLogs.log
fi" >> /config/system/ldapSearch.sh
chmod +x /config/system/ldapSearch.sh
bash /config/system/ldapSearch.sh

echo "#!/bin/bash
getent group $ldap_groupName
if [ $? -eq 0 ]; then
echo getent group $ldap_groupName was successful >> /config/system/DirectoryServiceSetupLogs.log
getent group $ldap_groupName >> /config/system/DirectoryServiceSetupLogs.log
else
echo getent group $ldap_groupName was failed >> /config/system/DirectoryServiceSetupLogs.log
fi" >> /config/system/getentGroup.sh
chmod +x /config/system/getentGroup.sh
bash /config/system/getentGroup.sh

echo 'LDAP setup complete.'

Run the LDAPs script.

#!/bin/bash

usageFunction()
{
   echo ""
   echo "Usage: $0 uri_val=<URI> base_dn=<BASE_DN> bind_dn=<BIND_DN> bind_pw=<BIND_PW> ldap_groupName=<LDAP_GROUP_NAME> login_shell=<LOGIN_SHELL> home_directory=<HOME_DIRECTORY> tls_dir=<TLS_DIR>"

   echo ""  
  
   echo -e "\t<URI> : List of URIs of Domain Controllers"
   echo -e "\t<BASE_DN> : Domain Name of Domain Controller"
   echo -e "\t<BIND_DN> : Bind Domain Name of User which has admin access in Domain Controller"
   echo -e "\t<BIND_PW> : Bind password of the bind user"
   echo -e "\t<LDAP_GROUP_NAME> : Group Name which has to be provided sudo access"
   echo -e "\t<LOGIN_SHELL> : Login shell in blockchain node of AD users"
   echo -e "\t<HOME_DIRECTORY> : Home Directory in blockchain node for AD users"
   echo -e "\t<TLS_DIR> : Directory containing certificate data"
   exit 1
}

for ARGUMENT in "$@"
do
   KEY=$(echo $ARGUMENT | cut -f1 -d=)

   KEY_LENGTH=${#KEY}
   VALUE="${ARGUMENT:$KEY_LENGTH+1}"

   export "$KEY"="$VALUE"
done


if [ -z "$uri_val" ] || [ -z "$base_dn" ] || [ -z "$ldap_groupName" ] || [ -z "$login_shell" ] || [ -z "$home_directory" ] || [ -z "$tls_dir" ]
then
   echo "";
   echo "Error while executing script: Mandatory parameters are missing";
   usageFunction
fi


#######################################
####### Set up open ldap client #######
#######################################
# Add nslcd group

echo 'Adding nslcd group...'
groupadd nslcd

#Edit ldap config
echo 'Editing ldap config...'
mv /etc/openldap/ldap.conf /etc/openldap/ldap.conf.backup
echo "#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

BASE $base_dn
URI $uri_val
BINDDN $bind_dn

#cacert File located for ADDS Server connection via ldaps
#TLS_CACERT /etc/openldap/cacerts/ldapTlsCacert.pem
#TLS_CACERT $tls_ca_cert

TLS_CACERTDIR  $tls_dir

TLS_REQCERT allow

#SIZELIMIT 12
#TIMELIMIT 15
#DEREF never" > /etc/openldap/ldap.conf

chmod 644 /etc/openldap/ldap.conf

# Edit nslcd config
echo 'Editing nslcd config...'
mv /etc/nslcd.conf /etc/nslcd.conf.backup
echo "# This is the configuration file for the LDAP nameservice
# switch librarys nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.

# The user and group nslcd should run as.
uid nslcd
gid ldap

# The uri pointing to the LDAP server to use for name lookups.
# Multiple entries may be specified. The address that is used
# here should be resolvable without using LDAP (obviously).
# Note: %2f encodes the '/' used as directory separator
uri $uri_val

# The distinguished name of the search base.
base $base_dn

# The distinguished name to bind to the server with.
# Optional: default is to bind anonymously.
binddn $bind_dn

# The credentials to bind with.
# Optional: default is no credentials.
# Note that if you set a bindpw you should check the permissions of this file.
bindpw $bind_pw

# The default search scope.
scope sub

#cacert directory where Certificate is located for ADDS Server connection via ldaps
tls_cacertdir $tls_dir
tls_reqcert allow
# Alternative mappings for Active Directory
# (replace the SIDs in the objectSid mappings with the value for your domain)
#pagesize 1000
#referrals off
#idle_timelimit 800
filter passwd (&(objectClass=user)(objectClass=person)(!(objectClass=computer)))
map passwd uid cn
#map passwd homeDirectory <homeDirectory>
map passwd homeDirectory $home_directory
map passwd gecos displayName
#map passwd loginShell <loginShell>
map passwd loginShell $login_shell
filter group (|(objectClass=group)(objectClass=person))" > /etc/nslcd.conf

chmod 600 /etc/nslcd.conf

# Edit nsswitch config
echo 'Editing nsswitch config...'
mv /etc/nsswitch.conf /etc/nsswitch.conf.backup
echo "# Begin /etc/nsswitch.conf

passwd: files ldap
group: files ldap
shadow: files ldap

hosts: files resolve dns
networks: files

protocols: files
services: files
ethers: files
rpc: files
# End /etc/nsswitch.conf" > /etc/nsswitch.conf

chmod 644 /etc/nsswitch.conf

# Edit pam.d files
echo 'Editing pamd files...'
mv /etc/pam.d/systemd-user /etc/pam.d/systemd-user.backup
echo "# This file is part of systemd.
#
# Used by systemd --user instances.
account sufficient pam_unix.so
account sufficient pam_ldap.so
session required pam_loginuid.so
session optional pam_keyinit.so force revoke
session optional pam_systemd.so" > /etc/pam.d/systemd-user

chmod 644 /etc/pam.d/systemd-user

mv /etc/pam.d/system-account /etc/pam.d/system-account.backup
echo "# Begin /etc/pam.d/system-account
account required pam_unix.so broken_shadow
#account sufficient pam_succeed_if.so uid < 1000 quiet
account sufficient pam_ldap.so
account required pam_permit.so
# End /etc/pam.d/system-account" > /etc/pam.d/system-account

chmod 644 /etc/pam.d/system-account

mv /etc/pam.d/system-auth /etc/pam.d/system-auth.backup
echo "# Begin /etc/pam.d/system-auth
auth sufficient pam_unix.so
auth sufficient pam_ldap.so
auth required pam_deny.so
# End /etc/pam.d/system-auth" > /etc/pam.d/system-auth

chmod 644 /etc/pam.d/system-auth

mv /etc/pam.d/system-password /etc/pam.d/system-password.backup
echo "# Begin /etc/pam.d/system-password
# use sha512 hash for encryption, use shadow, and try to use any previously
# defined authentication token (chosen password) set by any prior module
password requisite pam_cracklib.so
password sufficient pam_unix.so sha512 shadow try_first_pass
password sufficient pam_ldap.so
password required pam_deny.so
# End /etc/pam.d/system-password" > /etc/pam.d/system-password

chmod 644 /etc/pam.d/system-password

mv /etc/pam.d/system-session /etc/pam.d/system-session.backup
echo "# Begin /etc/pam.d/system-session
session required pam_mkhomedir.so skel=/etc/skel/ umask=0022
session sufficient pam_ldap.so
session sufficient pam_unix.so
session required pam_limits.so
session optional pam_systemd.so
session optional pam_loginuid.so
# End /etc/pam.d/system-session" > /etc/pam.d/system-session

chmod 644 /etc/pam.d/system-session

# Restart services
echo 'restarting nslcd...'
systemctl restart nslcd
echo 'restarting sshd...'
systemctl restart sshd

#Add groups to Sudoers file config
echo 'Editing sudoers config...'
cp /etc/sudoers /etc/sudoers.backup
echo "%$ldap_groupName ALL=(ALL:ALL) ALL" >> /etc/sudoers

#Verify AD config
echo 'Verifying ldap config...'

mkdir -p /config/system

echo "#!/bin/bash
ldapsearch -H $uri_val -D $bind_dn -b $base_dn -w $bind_pw
if [ $? -eq 0 ]; then
echo ldapsearch was successful >> /config/system/DirectoryServiceSetupLogs.log
else
echo ldapsearch was failed >> /config/system/DirectoryServiceSetupLogs.log
fi" >> /config/system/ldapSearch.sh
chmod +x /config/system/ldapSearch.sh
bash /config/system/ldapSearch.sh

echo "#!/bin/bash
getent group $ldap_groupName
if [ $? -eq 0 ]; then
echo getent group $ldap_groupName was successful >> /config/system/DirectoryServiceSetupLogs.log
getent group $ldap_groupName >> /config/system/DirectoryServiceSetupLogs.log
else
echo getent group $ldap_groupName was failed >> /config/system/DirectoryServiceSetupLogs.log
fi" >> /config/system/getentGroup.sh
chmod +x /config/system/getentGroup.sh
bash /config/system/getentGroup.sh

echo 'LDAP setup complete.'

LDAP sample command.

sudo sh ./adds_using_ldap.sh 
uri_val='ldap://35.173.230.128/' 
base_dn='dc=corp,dc=blockchain,dc=local' 
bind_dn='cn=administrator,CN=Users,dc=corp,dc=blockchain,dc=local' 
bind_pw='<password>' 
ldap_groupName='dev' 
login_shell='"/bin/bash"' 
home_directory='"/home/$cn"'

LDAPS sample command.

sudo sh ./adds_using_ldaps.sh 
uri_val='ldaps://35.173.230.128/' 
base_dn='dc=corp,dc=blockchain,dc=local' 
bind_dn='cn=administrator,CN=Users,dc=corp,dc=blockchain,dc=local' 
bind_pw='<password>' 
ldap_groupName='dev' 
login_shell='"/bin/bash"' 
home_directory='"/home/$cn"' 
tls_dir='/tmp'

(Optional) Rollback a failed upgrade process.

The rollback process reverts to the latest snapshot of your blockchain nodes. The snapshot is created during the upgrade process.
1. Run the rollback script to revert to the latest snapshot.
```
python UpdateController.py -j /home/blockchain/output/provisioning_output.json -i /home/blockchain/descriptors/reconfig_infra.json -d /home/blockchain/descriptors/reconfig_deploy.json -a revert_snapshot -s snapshot_before_bc_upgrade_sep29
```
2. Start the blockchain nodes.
```
python UpdateController.py -j /home/blockchain/output/provisioning_output.json -i /home/blockchain/descriptors/reconfig_infra.json -d /home/blockchain/descriptors/reconfig_deploy.json -a start_blockchain
```
3. Verify that the blockchain nodes are running.
  
  See Validate a Client Node and Backup in VMware Blockchain Orchestrator for vSphere, Validate a Replica Node in VMware Blockchain Orchestrator on vSphere, and Validate a Full Copy Client Node in VMware Blockchain Orchestrator for vSphere.