Cinder Backup Fails Under High Concurrency

The default VMware Integrated OpenStack configuration may be insufficient for Cinder backup operations with high concurrency or large volumes.

Problem

When you increase the concurrency of Cinder backup operations or the size of Cinder volumes, operations may fail and GetResourceFailure errors may be displayed in the logs.

Solution

1. Scale out the control plane and number of Cinder backup pods.
   Each controller node can contain only one Cinder backup pod.
   1. Increase the number of controller nodes in your deployment.
      See Add Controller Nodes to Your Deployment.
   2. Increase the number of Cinder backup pods in your deployment.
      See Scale OpenStack Services.
2. Log in to the Integrated OpenStack Manager as the root user.

   ssh root@mgmt-server-ip

3. Update the RPC response timeout and executor thread pool size for Cinder.
   1. Modify the Cinder configuration.

      viocli update cinder

   2. In the DEFAULT section, add the rpc_response_timeout parameter and set its value to 6000.
   3. Add the executor_thread_pool_size parameter and set its value to 640.
      The configuration file now looks similar to the following.

      conf:
        backends:
          [...]
        cinder:
          DEFAULT:
            [...]
            rpc_response_timeout: 6000
            executor_thread_pool_size: 640

4. Update the database timeout and maximum connection parameters.
   1. Modify the MariaDB configuration.

      viocli update mariadb

   2. In the conf section, add the connect_timeout parameter and set its value to 5.
   3. Add the max_connections parameter and set its value to 5000.
   4. Add the net_read_timeout parameter and set its value to 1200.
   5. Add the net_write_timeout parameter and set its value to 1200.
   6. In the conf section, add the ingress section.
   7. In the ingress section, add the proxy-read-timeout parameter and set its value to 1200.
   8. Add the proxy-send-timeout parameter and set its value to 1200.
   9. Add the proxy-stream-timeout parameter and set its value to 3600s.
      The configuration file now looks similar to the following.

      conf:
        connect_timeout: 5
        max_connections: 5000
        net_read_timeout: 1200
        net_write_timeout: 1200
        ingress:
          proxy-read-timeout: "1200"
          proxy-send-timeout: "1200"
          proxy-stream-timeout: 3600s

5. Update the pool sizes and allocation ratios for Nova.
   1. Modify the Nova configuration.

      viocli update nova

   2. In the nova section, add the DEFAULT section.
   3. In the DEFAULT section, add the cpu_allocation_ratio parameter and set its value to 30.
   4. Add the executor_thread_pool_size parameter and set its value to 640.
   5. Add the ram_allocation_ratio parameter and set its value to 6.
   6. In the nova section, add the database section.
   7. In the database section, add the max_pool_size parameter and set its value to 50.
   The configuration file now looks similar to the following.

   conf:
     nova:
       DEFAULT:
         cpu_allocation_ratio: 30
         executor_thread_pool_size: 640
         ram_allocation_ratio: 6
       database:
         max_pool_size: 50

6. Update the token expiration and Web Server Gateway Interface (WSGI) parameters for Keystone.
   1. Modify the Keystone configuration.

      viocli update keystone

   2. In the conf section, add the keystone section.
   3. In the keystone section, add the wsgi_processes parameter and set its value to 8.
   4. Add the wsgi_threads parameter and set its value to 15.
   5. In the keystone section, add the token section.
   6. In the token section, add the expiration parameter and set its value to 28800.
      The configuration file now looks similar to the following.

      conf:
        keystone:
          wsgi_processes: 8
          wsgi_threads: 15
            token:
              expiration: 28800