Quality Assurance in Bahmni

We rely on the contributors, reviewers, core-dev-team/maintainers and community in general for doing quality checks on Bahmni. We heavily focus on test automation to ensure existing features continue to work while we build new ones. 

Typically we focus on functional testing over non-functional testing (see Appendix: Types of Testing in software projects below)  in every release. We mostly rely on the community to provide feedback on usability. Sometimes, when a completely new feature is built, we would also do some non-functional testing especially usability testing. 

How QA fits into our development process

Development process 

We follow the Agile Software development process. We typically do two major releases in a year. This picture below shows the different phases of the Bahmni development process and how we ensure quality before we release a new version to the community. 

During the Planning Phase

• We prioritise the work items for each release using the roadmap process defined here. 

• See here for an eg of what goes into a release 

• Once release deliverables are identified at a fairly detailed level, user stories are created in Jira (which is our development tracking tool). For example, check the JIRA board for 0.92 release

QA activities that happen in this phase

1. Acceptance criteria are added to each story. See example below

Open

During Development Phase

• As part of writing code, we encourage developers to also write automated unit tests

• Wherever applicable and feasible, automated Integration Tests and Functional tests are also written by the developers.  See the following examples

  • Unit tests: example 1, example 2

  • Integration tests: example 1, example 2

  • Functional tests: example 1, example 2

Note, check the individual modules source codes or repositories for existing test suites. Tests may also be written using different technologies (for example, js or specs). 

• Most often we do not have dedicated QA capacity so we rely on the automated unit and other tests to ensure that the quality of the code written is good.

• As a contributor, we expect you to follow the community process. Check here for PR submission. The reviewer needs to be sufficiently convinced of your code and test coverage for merging a PR. Please respect his/her opinion. For some features, the reviewer might request you for a dev-box-testing

• Once the contributor, raises a PR, the associated Travis-CI builds (for the code repository) will kick in, please verify and fix any error reported. 

• As a maintainer/reviewer, when you are reviewing the pull requests and merging it, please also ensure that there are no obvious issues. For more information on the review process, please check here. 

• Once the PR is merged, Bahmni CI/CD process will trigger the builds, execute associated unit/functional tests, and may deploy on one of the QA environments. If not, you may ask the maintainers to pull in the latest build on a QA environment. You should verify your features on QA boxes. 

During Testing Phase

This internally has two sub-phases

Step One: Regression Testing

Once all stories for the release have completed the development we announce a dev freeze. After this only bug fixes are allowed. Members of the community who have been actively involved/have developed the features in that release would get together and do a round of Regression Testing. The objective of this is to ensure the version is stable before opening up to the community for testing. This typically takes 2 weeks. 

Step Two: Community Testing 

Once we feel reasonably confident that the version is stable, we open up for community testing. This version is called a release candidate The release manager posts a request for contribution in the community channel on talk with instructions on how to contribute. 

• Testing is done for the designated time (usually 2 weeks + ) during which the community tests on the public QA instances. 

• regular updates are given to the community. 

• decisions on fixes are often taken on the standup. Contentious issues and/or significant issues are also discussed on the talk channel. 

• community takes a call if any of the issues can not be resolved - this goes onto the release notes. 

During the Release Phase

• Documentation for features are created, videos, wiki updated 

• Release notes are prepared and reviewed. 

• Once the final builds are uploaded, final testing (primarily installation on target environment) are done by the maintainers.  

• The release is announced on community channels - talk, tweeter etc. 

Appendix - Types of testing in software development

Typical software goes through various types of testing as shown in this diagram below

This article gives a good overview of what each of them means.