...
- Database Ids may overflow over time. It may have developed unnecessary gaps in Ids.
- The system leaks memory slowly and has to be restarted
- The system becomes slower over time and has to be restarted
- The system becomes slower with increase in database size
- Individual services runs out of threads
- Individual services runs out of database connection allocated to them
- The system is not reporting the errors
- The system becomes slower with increasing dataThe system is using more disk space than it is should
- On machine restart the services do not come back up
- On service restart it doesn't become available for service
- Active to Passive data replication is not happening
- Automatic switch back from active to passive doesn't work
- Automatic switch back from passive to active doesn't work
- Database connection becomes stale if not used for some time
- Failed events do not resolve on their own (including scenarios of restart of services)
- A very large file is uploaded causing server to become less responsive
- Redirect loop causes denial of service (low priority)
Automation
The above scenarios need to be tested manually or in an automated fashion.
Environment
Functional Tests
Setup an environment with enough disk space, cpu and memory. Ensure that all the basic scenarios are covered by the functional tests. Run these tests continuously for days.
Environment Configuration
The system should be configured such that after it gives warning it continues working for sometime. For example, if OpenMRS runs out of number of threads
...
| Server | Database | Max Connections |
|---|---|---|
| MySQL | OpenMRS | |
| PostgreSQL | OpenELIS (clinlims) | |
| PostgreSQL | OpenERP | |
| MySQL | Jasper |
Tomcat
...
...
Automation
Environment
Functional Tests
...
Monitoring
- Icinga
- How to notify when something goes wrong in production environment
- Test whether monitoring is working or not
...