NSX Advanced Load Balancer Controller cluster instances are frequently required to be deployed from a common initial single Controller configuration that differs from the factory-default settings. This can be accomplished by creating a JSON file with the required object definitions and then using it to deploy subsequent controllers, which, as leaders, can then add followers to become clusters.  You will have a set of identically initialized Controller clusters, ready to be individualized as needed.

Create setup.json

In most cases, these objects can be created by referring to the NSX Advanced Load Balancer REST API section.

The following example updates the system configuration by adding 8.8.8.8 to the DNS configuration:

{
    "SystemConfiguration": [
        {
            "dns_configuration": {
                "search_domain": "",
                "server_list": [
                    {
                        "type": "V4",
                        "addr": "8.8.8.8"
                    }
                ]
            }
        }
    ]
}

In the case of complex objects such as the SSLKeyAndCertificate object, the JSON file can be created by running a diff command against two configuration files. In a typical deployment, generating setup.json on a test Controller environment is recommended. This generated file can then be used as a template for actual deployments. An NSX Advanced Load Balancer Controller configuration snapshot can be taken using the export CLI command.

> export configuration file before.cfg
Please enter the passphrase to encrypt configuration:
Downloaded the attachment to before.cfg
Completed writing the export configuration to before.cfg

Configure objects using the UI or CLI, as required.

> export configuration file after.cfg
Please enter the passphrase to encrypt configuration:
Downloaded the attachment to after.cfg
Completed writing the export configuration to after.cfg

Beyond this, configuration diff can be taken using a Python script.NSX Advanced Load Balancer has written expressly to customize initial configuration of another Controller.

/opt/avi/scripts/diff_config.py -f before.cfg -t after.cfg > setup.json

User passwords can be encrypted using the following code when creating setup.json with the User object.

/opt/avi/scripts/avi_passwd_tool.py --password admin --salt fF6ngAb3pvPgpbkdf2_sha256$100000$fF6ngAb3pvPg$ijkEue1M9fR/qsLVgzvPe7N0VvOxIjDiJVmK9NIx+0Q=$6$fF6ngAb3pvPg$CqAKtNRZtgXtJchrPmoxUgdLFM7rFGmta1tWb7sobQI4iSZAY2QuAOBNtboVGrmDYPMCvqXXH6lARr9RedCJT.

Deployment using setup.json

It is recommended to take a configuration backup before deploying the Controller using the setup.json file created by the Python script. Use the following command to create an encrypted backup of the existing configuration.

/opt/avi/python/bin/portal/manage.py export_configuration --file ~/setup-old.json --passphrase secret

For a Mesos/Bare-Metal Deployment

Copy setup.json to the persistent directory in the host mounted as /vol in the Controller container. If you are using the avi_baremetal script, the default location is /opt/avi/controller/data on the host. When deploying the Controller as a container, setup.json can be passed as an additional argument to the avi_baremetal_setup.py script. For example:

./avi_baremetal_setup.py -c -cc 4 -cm 12 -i 10.10.22.108 -m 10.10.22.108 --setup-json /root/configs/avi-setup.json

For a Controller Deployment as a Virtual Machine

Wait until the Controller comes up. Place the config file on the Controller as /var/lib/avi/etc/setup.json (note the filename). Upon reboot or fresh-start, the NSX Advanced Load Balancer Controller will self-configure using the provided setup.json file.

reboot

For an OpenStack Deployment

UserData config size is limited to 48 Kb if the size of setup.json is within allowable limits.

># cfgdrv userdata
>nova boot --config-drive true --image avicontroller --key-name mykey --flavor 4 --user-data /root/my-avi-config.json --nic net-id=7402bf4f-240f-4172-99c1-90000ea45f86 avicontrollers

># metasvc userdata
>nova boot --config-drive false --image avicontroller --key-name mykey --flavor 4 --user-data /root/my-avi-config.json --nic net-id=7402bf4f-240f-4172-99c1-90000ea45f86 avicontrollers

If the setup.json size is bigger then than the allowable limit, setup.json can be uploaded and referred to in the deployment phase.

User data can refer to the file using the “url” or “file” tag. Following is an example of my-avi-config-url.json with URL.

{
 "META": {
     "init_config": {
         "url": "https://s3-us-west-2.amazonaws.com/avi-controller-configs/linuxserver-awsipam-setup.json"
     }
   } 
}

Following is an example of my-avi-config-url.json with filepath.

{
  "META": {
    "init_config": {
      "file": "/vol/linuxserver-awsipam-setup.json"     
    }
  } 
}

Following is an example of deployment.

># cfgdrv userdata indirection
>nova boot --config-drive true --image avicontroller --key-name mykey --flavor 4 --user-data /root/my-avi-config-url.json --nic net-id=7402bf4f-240f-4172-99c1-90000ea45f86 --min-count=3 --max-count=3 avicontrollers

># metasvc userdata indirection
nova boot --config-drive false --image avicontroller --key-name mykey --flavor 4 --user-data /root/my-avi-config-url.json --nic net-id=7402bf4f-240f-4172-99c1-90000ea45f86 --min-count=3 --max-count=3 avicontrollers

For an AWS Deployment

UserData config size is limited to 16Kb. If the size of setup.json is within allowable limits, cut-paste the my-avi-config.json into the user-data section during launch from AWS Web Console.

# metasvc userdata  ec2-run-instances ami-b7ea27d7 -f /root/my-avi-config.json -t c4.2xlarge -s subnet-62f1b707 -g sg-642d8d02

If the setup.json size is bigger than the allowable limit, cut-paste the my-avi-config-url.json into the user-data section during launch from AWS Web Console.

# metasvc userdata indirection ec2-run-instances ami-b7ea27d7 -f /root/my-avi-config-url.json -t c4.2xlarge -s subnet-62f1b707 -g sg-642d8d02

my-avi-config-url.json follows similar formats as discussed in the OpenStack section. Following is a sample my-avi-config-url.json file for the S3 bucket.

{
  "META": {
    "init_config": {
      "s3": "avi-controller-configs/linuxserver-awsipam-setup.json"
    }
  } 
}

For uploading setup.json on the S3 bucket:

  • Public : use the url style or s3 style

  • Private through RBAC on VM: use the s3 style. The VM role must have s3:GetObject action allowed to be able to s3-get the object using IAM.

  • Private through RBAC on S3-bucket: use the s3 style. The VM role must have AWS access. The S3 bucket must have permissions for the account or user or VM role to download the object.

Example Bucket Policy:

{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "AddPerm",
       "Effect": "Allow",
       "Principal": {
         "AWS": [
           "arn:aws:iam::139284885014:role/BM-AviController-Role",
           "arn:aws:iam::139284885014:root"
          ]
        },
       "Action": "s3:*",
       "Resource": "arn:aws:s3:::avi-controller-configs/*"
     }
   ]
 }

For an Azure Deployment

There are two ways to provide initial configuration in an Azure environment:

  • Using Azure CLI

  • Using ARM (Azure Resource Manager) template

Using Azure CLI

If the NSX Advanced Load Balancer Controller is deployed using Azure CLI, the JSON file can be provided during the deployment.

az vm create --resource-group rahulr-jenkins-resource-group --location centralus --image avi-networks:avi-vantage-adc:avi-vantage-adc-byol:17.2.7 --name Avi-Test-Controller --size Standard_F8s --subnet /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/virtualNetworks/<vnet_name>/subnets/rahulr-subnet --public-ip-address "" --nsg "" --custom-data ./initial_config.json

Using Azure ARM Template

If the NSX Advanced Load Balancer Controller is deployed using the ARM template, the JSON data is provided as the Custom Data on the Custom deployment page of the Azure portal. Navigate to Home > Templates > avi-cluster-managed-disks-market place > Custom deployment.



Use the required JSON template in the Custom Data field. For reference, the below JSON template is for adding 8.8.8.8 to DNS configuration. Copy the JSON configuration mentioned below, and add it to the Custom Data field in the ARM template.

{
    "SystemConfiguration": [
        {
            "dns_configuration": {
                "search_domain": "",
                "server_list": [
                    {
                        "type": "V4",
                        "addr": "8.8.8.8"
                    }
                ]
            }
        }
    ]
}