[Monitoring] Monitoring & Centralised Logging Installation & Usage

Introduction

The current Bahmni setup is using below tools for Monitoring & Observability:

1. Monitoring & Alerting: https://prometheus.io/
Prometheus: Prometheus is an open-source systems monitoring and alerting toolkit. Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

2. Centralised Logging: https://grafana.com/oss/loki/ consisting of Promtail, Loki and https://grafana.com/
Grafana: Grafana is a multi-platform open source analytics and interactive visualisation web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

Loki: Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Bahmni setup

Bahmni is managing its monitoring & observability using the latest official helm-charts published & maintained by providers (Prometheus/Grafana). Bahmni comes with out of the box dashboards and metrics configured.

• We are using kube-prometheus-stack for monitoring which is a combined stack of Prometheus, Grafana & Loki. This stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
  
  

• The centralised logging setup has been done with Loki stack which consists of Promtail as log collector, Loki as log storage and indexing and Grafana for visualisation. More about Loki here.

Currently all the kubernetes components & application logs for all namespaces are available in Loki.

Monitoring & Logging Installation:

Using Github Actions

You can fork the helm-umbrella-charts repository and run the Github actions workflow to provision infrastructure.

Using Docker-compose

Please refer the Bahmni-docker wiki page.

Steps for login Grafana

1. Navigate to Bahmni-Lite Monitoring dashboard.

2. Click on Sign in with Github.

3. Authorise Bahmni Github to allow Github authorisation. You are IN

Viewing Grafana Dashboards

1. To View dashboards, click on the below sign.

   Open

   

    

2. This will open a search Dashboards.

   Open

   

    

3. Clicking on any of the dashboard that you would want to view.

Steps to view logs

1. Once logged in, navigate to Explore in Grafana.

2. Once you are in the Explore page. Set the datasource as Loki.

3. Now use the Log Browser to select apps and namespace for which you want to view the logs. You can also write your own LogQL query as well.

4. You can filter labels to see the required logs.
Search can be done for a specific namespace, instance, container, application etc.

5. Post selection of filtered label, You can select the relevant component for which you want to see the logs.

Example 1: Let's suppose we have selected app as the label, now if we click on select value it will list down all the app available with logs.

Example 2: Let's suppose we have selected namespace as the label, now if we click on select value it will list down all the namespace available with logs.

Similarly can be done for other label as well.

Demo Video

To understand more details around how to use Monitoring & centralised Logging we have recorded the following DEMO on Youtube. This explains the login process & how to view logs. This is a demo that was done during the weekly PAT call some time ago.

(Duration: 20 mins)

Examples