Setting up alerts based on Prometheus metrics

Grafana supports configuring of alerts based on the metrics aggregated by the connected data-sources. Read more about it in the below page.

Bahmni out of the box comes with few common alert rules configured. The following are the out-of-the box alerts and configuration.

1. Node Level Alerts (defined in node_alerts_group.yaml)

   1. High CPU Usage Alert if utilisation > 75%

   2. Low Memory Availability Alert if available memory < 30%

   3. Low Disk Space Alert if available space < 30%

2. Container Level Alerts (defined in container_alert_group.yaml)

   1. Container High CPU Usage Alert if CPU usage of container > 50%

   2. Container High Memory Usage Alert if memory usage > 1024 MB (except OpenMRS container)

   3. Container High Memory Usage Alert for OpenMRS if memory usage > 3000 MB

There is also a default SMTP alerting contact point and notification policy which comes configured out of the box. There are numerous notification contact points that can be integrated with Grafana for alerting. Read about it here.

Configuring SMTP to receive alerts on email

1. Update the values of SMTP server details in the .env file in the bahmni-observability directory

   SMTP_HOST= SMTP_USERNAME= SMTP_PASSWORD= SMTP_FROM_NAME= SMTP_FROM_ADDRESS=

    

2. Configure the receiver email address. There are two environment variables which are available to send the email alerts to. The DEFAULT_ALERT_MAIL_RECEIVER variable is used to send all the emails. The CRITICAL_ALERT_MAIL_RECEIVER variable is used to configure the list of address for whom only critical and high severity alerts will be sent. Bothof these variable support comma separated input for multiple email adress.
   Example:

   DEFAULT_ALERT_MAIL_RECEIVER=example@example.com CRITICAL_ALERT_MAIL_RECEIVER=example1@example.com, example2@example.com

3. Once completing the configuration, update the Grafana service by running docker compose --profile logging up -d grafana

4. The alert email content and subject lines are templates that can be configured. Refer to the files mail_content_template.yaml and mail_subject_template.yaml under the bahmni-observability/config/alerting/ directory.

Updating the threshold values for alert rules

All the above mentioned default alerts are created using the configuration files in the bahmni-observability/config/alerting/ directory. You can update the alert threshold values in the node_alerts_group.yaml or container_alert_group.yaml files at the following snippet for each rule