This topic explains recommended testing practices for Ops Manager tile developers.
Good testing assures tile developers that their product installs and runs properly on diverse platforms and assures platform operators that the tile they install can provide its service successfully on their platform.
VMware recommends a pyramid structure for testing, starting with unit tests and stepping up to successively broader and more automated levels of integration. VMware uses and recommends Concourse for creating build pipelines that follow this test structure. Other continuous integration tools should also support a pyramid testing approach.
For Ops Manager tiles, a typical test pyramid progresses as follows:
Unit tests for each tile component (e.g. service components, broker, adapter, and metrics emitter), manual by developer and in automated pipeline.
System tests of the tile’s BOSH release, including:
System tests of tile operation within Ops Manager.
Note: System tests might incur costs from using third party services, IaaS resources, etc.
Smoke tests are end-to-end lifecycle tests for service instances that you can include as post-deploy errands within a tile and also automate in Concourse or other integration platforms.
A typical smoke test runs as follows:
Create an org and space for the test to run in.
Register the tile’s service broker.
Enable service access for the created org.
Iterate through all service plans (or a subset of them) to do the following:
Delete the service broker.
Delete the test org and space.
The following are general recommendations for designing and running tests on Ops Manager tiles:
Clean up after yourself. Leave the environment exactly as it was before the test was run.
Generate verbose logging with lots of contextual data to make troubleshooting easier.
Design test suites for re-usability by making them highly parameterizable. Important parameters include:
Re-use tests that exist already, for example in Concourse.
Use an example CF app that uses your service. This app can serve for testing, demoing your tile capabilities, and as a code example. See the MySQL Test App an example.
When testing manually, using the UI is better than calling the underlying API directly. Use UIs and APIs the way a customer would.