This document addresses common issues in using the airgap appliance and related solutions.

Issues and solutions

  1. Issue: Resize appliance disks

    Cause: While using the airgap server, there might be chances that disk can go low on available space.

    Solution: Follow the steps below to increase the disk size:
    1. Open VM Edit settings wizard.
    2. Specify a new size for the selected disk and click OK.
    3. Open VM Edit settings wizard again.
    4. Take note of total provisioned disk size in line Hard disks.
    5. Expand Hard disks list, resize the Hard disk 7 with value of the provisioned disk size in step 4 multiplied by 15%.

      For example, if the provisioned disk size is 1000GB, then the size for Hard disk 7 is 150GB (1000 * 15%).

    6. SSH login airgap appliance console.
    7. Check the list of disks by running the command fdisk -l.
    8. Rescan the extended disk: echo 1 > /sys/class/block/<disk-name>/device/rescan.

      For example: echo 1 > /sys/class/block/sdc/device/rescan.

    9. Perform pvresize to resize physical volume: pvresize /dev/<disk-name>.

      For example: pvresize /dev/sdc.

    10. Check mount points and logic volume mapping by df -h |grep ^/dev, find out the logic volume to extend size.
    11. Extend logical volume: lvextend -l +100%FREE <LV-name>.

      For example: lvextend -l +100%FREE /dev/VGOS/LV_OS.

    12. Perform a filesystem resize of logical volume: resize2fs <LV-name>. For example: resize2fs /dev/VGOS/LV_OS.
    13. Execute df -h |grep ^/dev to validate the disk resize is completed.
    14. Scan the snapshot disk echo 1 > /sys/class/block/sdg/device/rescan.
    15. Perform pvresize to the snapshot disk: pvresize /dev/sdg.
    16. Extend snapshot logical volume: lvextend -l +100%FREE /dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot.
    17. Perform a filesystem resize of snapshot volume: resize2fs /dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot.
    18. Execute df -h |grep ^/dev to validate the disk resize is completed.
  2. Issue: Failed to execute rsync operation when airgap appliance configured with proxy

    Cause: This is a known issue that the source host and target host should be within the same network which can connect to each other directly. Access to the remote airgap server via proxy is not supported.

    Solution: Follow the steps below to resolve this issue:
    1. SSH to target airgap host.
    2. Clear proxy info: bash /usr/local/airgap/scripts/bin/clear-proxy.sh.
    3. Logout from ssh connection.
    4. Login to target host again via SSH.
    5. Start rsync agctl rsync.
  3. Issue: Base64 encoding via command line. In some environments, the online Base64 encode/decode tool, ​​https://www.base64encode.org/, may not be accessible.
    Solution: When user wants to encode or decode certificates for deploying airgap OVA or for proxy server, following steps can be performed from command line:
    • Encode certificate:
      #base64 -w0 <certificate>
    • Decode certificate:
      #base64 -d <encoded-certificate>
  4. Issue: Using chained certificates in airgap server.
    Solution: When user has chained certificate which includes server cert, intermediate CA and root CA, user can follow the below steps to correctly apply the certificates on airgap server:
    1. Create a folder, certificate, on airgap server.
    2. Copy the 3 certificates in the above folder.
    3. Merge the certificates into chained certificate: cat <server-certificate> <intermediate-CA> <global-root-CA> > <new-chaine-certificate>.
    4. After the new chained certificate is created, convert it into Base64 format and apply when deploying airgap appliance OVA. Apply chained cert in server cert field and intermediate CA cert in CA cert field.
  5. Issue: Configure proxy in Day 1.

    Solution: To configure proxy on airgap appliance, it requires three parameters:

    • http proxy server URL– The proxy server that handles proxy HTTP traffic.
    • https proxy server URL– The proxy server that handles proxy HTTPS traffic.
    • no proxy list – Comma-separated exclusion list of networks for bypassing proxy.
    Then run the script: 
    /usr/local/airgap/scripts/bin/setup-proxy.sh <http-proxy-sever-url> <https-proxy-server-url> <no-proxy-list>
    Example: 
    /usr/local/airgap/scripts/bin/setup-proxy.sh http://proxy.example.com:3128 https://proxy.example.com:3443 tca-ag-tmp.example.com,192.168.0.0/24
    Note: To avoid Harbor image publishing failure, ensure that the airgap server FQDN and the local network are added to the no_proxy list.

    If the proxy server uses HTTPS port and has configured a private CA-signed certificate or self-signed certificate, the user must upload and add the private root CA or self-signed certificate to airgap server root CA certificate bundle.

    Ensure that the proxy CA certificate is named with surfix .pem, and avoid naming it with cacert.pem, which is usually reserved for auto-generated CA certificates.

    If the environment uses other approaches to access the internet, configure your network infrastructure to ensure that the internet is accessible.

    Upload private root CA or self-signed certificate to the airgap server and run the following steps to activate it: 
    cp <proxy-ca>.pem /etc/ssl/certs/
cat <proxy-ca>.pem >> /etc/pki/tls/certs/ca-bundle.crt
tdnf update
tdnf install openssl-c_rehash -y
rehash_ca_certificates.sh
    To verify that the airgap virtual machine can access the required internet resources, run the following commands: 
    curl https://projects.registry.vmware.com --head
curl https://vmwaresaas.jfrog.io --head
curl https://packages.vmware.com/photon/ --head
curl https://github.com --head
tdnf update

    These commands must return 200 OK, which ensures that your network is ready.

  6. Issue: Cannot login airgap VM via SSH due to PAM authentication issue.

    Cause: Sometimes users cannot login to the airgap server through SSH which might be caused by PAM authentication failure.

    Solution: Below steps can be used to disable PAM authentication in SSH service to enable user login:
    1. Login to vCenter server.
    2. Open the airgap appliance VM console.
    3. Login to the VM with root user in the VM console.
    4. Check SSHD service status systemctl status sshd.
    5. Edit sshd_config file: vi /etc/ssh/sshd_config.
    6. Change line UsePAM yes to # UsePAM yes.
    7. Save the config file.
    8. Restart SSHD service: systemctl restart sshd.
    9. Try to login to the airgap appliance via SSH.
  7. Issue: Airgap Server upgrade and troubleshooting.

    If the upgrade log ends with the error message Airgap appliance upgrade failed! Please check log /var/log/vmware/capengine/cap-update/workflow.log for more details, refer to the specified log to investigate the causes of the failure.

    Here are a few common errors encountered in the workflow.log along with their solutions:
    1. Issue: Build version mismatch

      Cause: There are source and target build numbers in the log. This issue might occur if the target build number is smaller than the source build number.

      Solution: Cancel upgrade as the existing build is newer than the target build.

    2. Issue: Failed to execute command: umount -l /storage/alt_root/boot/efi

      Cause: This is a temporary file system issue.

      Solution: Redo agctl upgrade.

    3. Issue: One of the mandatory disks (lvm_snapshot) required for update is not present

      Cause: This might occur if the snapshot volume does not automatically mount after reboot/power on.

      Solution:

      1. Check /etc/fstab if lvm_snapshot is available: grep lvm_snapshot /etc/fstab.

      2. If the mount point is available, run command mount -a.

      3. Check if the volume is mounted: mount |grep lvm_snapshot.

      4. If the mount point is not available in /etc/fstab and the volume is not mounted, run command mount /dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot /storage/lvm_snapshot/ to mount the volume manually.

      5. Rerun upgrade: agctl upgrade.

    4. Issue: agctl upgrade fails at times with error iso does not exist even if correct ISO path is provided especially when actual ISO name is used as is which contains version numbering.

      Cause: This is a known issue caused by combining source and target user-inputs.yml files during upgrade.

      Solution:

      1. Modify the upgraded ISO file name to a simple text like update.iso.

      2. Edit /usr/local/airgap/scripts/vars/user-inputs.yml, go to the field local_iso_path: and revise the value of it.

        For example, if the ISO is located under the /tmp folder, then the value should be /tmp/update.iso.

      3. Save the user-inputs.yml and rerun agctl upgrade.

Best Practices

  1. Encoding certificate using command line interface

    In some environments, the online Base64 encode/decode tool ​​https://www.base64encode.org/ may not be accessible. When user wants to encode or decode certificates for deploying airgap ova or for proxy server following steps can be performed from command line:

    Encode certificate: 
    #base64 -w0 <certificate>

    Decode certificate:

    #base64 -d <encoded-certificate>

  2. Correct way of using chained certificates in airgap server

    When user has chained certificate which includes server cert, intermediate CA, and root CA, user can follow the below steps to correctly apply the certificates on airgap server:

    1. Create a folder certificate on airgap server.
    2. Copy the 3 certificates into above folder.
    3. Merge the server certificate and intermediate CA certificate into one file:
      cat <server-certificate> <intermediate-CA> > <new-chaine-certificate>
    4. After the new chained certificate was created, convert it into Base64 format and apply in the following way:
      1. While deploying the OVA, provide the merged certificate as server certificate and Root CA certificate as CA certificate.
      2. While registering the airgap server in the TCA partner system, provide the root CA certificate in the certificate field.