Resolving service interruptions

This topic tells you about the lifecycle of a VMware Tanzu SQL with MySQL for VMs service instance that might cause temporary service interruptions.

Stemcell or service update

You can update a stemcell version or their version of Tanzu SQL for VMs.

• Impact: Apps lose access to the MySQL service while Tanzu Operations Manager updates the service instance they are bound to. The service resumes within 10-15 minutes.
• Required Actions: None. If the update deploys successfully, the apps automatically reconnect.

Plan change

A developer can change their service instance to provide a different service plan, using cf update-service or Apps Manager.

• Impact: Apps lose access to the MySQL service while Tanzu Operations Manager updates the service instance they are bound to. The service resumes within 10-15 minutes.
• Required Actions: None. If the plan change deploys successfully, apps reconnect automatically.

VM process failure

A process, like the MySQL server, fails on the service instance VM.

• Impact:
  • BOSH (monit) brings the process back automatically.
  • Depending on the process and what it was doing, the service can experience 60-120 seconds of downtime.
  • Until the process resumes, apps might be unable to use MySQL, metrics or logging can stop, and other features might be interrupted.
• Required Actions: None. If the process resumes cleanly and without manual intervention, apps reconnect automatically.

VM failure

A Tanzu SQL for VMs VM fails and goes offline due to either a virtualization problem or a host hardware problem.

• Impact:
  • If the BOSH Resurrector is enabled (recommended), BOSH can detect the failure, recreate the VM, and reattach the same persistent disk and IP address.
  • Downtime largely depends on how quickly the Resurrector notices, usually 1-2 minutes, and how long it takes the IaaS to create a replacement VM.
  • If the Resurrector is not enabled, some IaaSes, for example, vSphere, have similar resurrection or HA features.
  • Apps cannot connect to MySQL until the VM is recreated and the My SQL server process is resumed.
  • Based on prior experience with BOSH Resurrector, typical downtime is 8-10 minutes.
• Required Actions:
  • If the VM is part of a leader-follower pair, when the VM comes back, it is read-only. Therefore, run the configure-leader-follower errand to ensure the leader VM is writable. For more information, see configure-leader-follower in Running Errands.
  • If the VM is not part of a leader-follower pair, when the VM comes back, no further action is required for the app developer to continue operations.

AZ failure

An availability zone (AZ) goes offline entirely or loses connectivity to other AZs (net split). This causes service interruption in multi-AZ deployments where Diego has placed multiple instances of a MySQL-using app in different AZs.

• Impact:
  • Some app instances can still be able to connect and continue operating.
  • App instances in the other AZs are not able to connect.
  • Downtime: Unknown
• Required Actions: Recovery of the app - database connection must be automatic. Depending on the app, manual intervention might be required to check data consistency.

Region failure

• Example: An entire region fails, bringing VMware Tanzu Application Service for VMs components offline.

• Impact:

  • The entire installation needs to be brought back up manually.
  • Downtime: Unknown
• Required Actions: Each service instance might need to be restored individually depending on the restored state of the platform.