Monitoring & Centralised Logging Installation for Kubernetes environments

The below steps and explanations are specific to Bahmni AWS Helm Charts Implementation of Grafana monitoring and Loki centralised logging.

Introduction

 

The current Bahmni setup is using below tools for Monitoring & Observability:


1. Monitoring & Alerting: https://prometheus.io/
Prometheus: Prometheus is an open-source systems monitoring and alerting toolkit. Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

2. Centralised Logging: https://grafana.com/oss/loki/ consisting of Promtail, Loki and https://grafana.com/
Grafana: Grafana is a multi-platform open source analytics and interactive visualisation web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

Loki: Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Bahmni setup

Bahmni is managing its monitoring & observability using the latest official helm-charts published & maintained by providers (Prometheus/Grafana). Bahmni comes with out of the box dashboards and metrics configured.

  • We are using kube-prometheus-stack for monitoring which is a combined stack of Prometheus, Grafana & Loki. This stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

  • The centralised logging setup has been done with Loki stack which consists of Promtail as log collector, Loki as log storage and indexing and Grafana for visualisation. More about Loki here.

Currently all the kubernetes components & application logs for all namespaces are available in Loki.

Monitoring & Logging Installation:


Using Github Actions

You can fork the helm-umbrella-charts repository and run the Github actions workflow to provision infrastructure.

1️. Add AWS Secrets to Github Actions Secrets

You need to add the AWS Secrets to Github actions secrets for the workflow to authenticate to AWS.

BAHMNI_AWS_ID → Access Key ID of the user provisioned

BAHMNI_AWS_SECRET → Secret Access Key of the user

BAHMNI_INFRA_ADMIN_ROLE → Role ARN of the BahmniInfraAdminRoleForIAMUsers

BAHMNI_AWS_REGION → AWS Region

HOSTED_ZONE_ID → hosted zone is a container for DNS records.

2️. Run the pipeline

There is a dedicated pipeline for deploying Monitoring

a) Deploy Monitoring and Logging: This pipeline would provision monitoring & logging resources using helm-charts.

Steps for login Grafana

  1. Navigate to Bahmni-Lite Monitoring dashboard.

  2. Click on Sign in with Github.

  3. Authorise Bahmni Github to allow Github authorisation. You are IN

Viewing Grafana Dashboards

  1. To View dashboards, click on the below sign.

     

  2. This will open a search Dashboards.

     

3. Clicking on any of the dashboard that you would want to view.

For more deeper understanding about Grafana Dashboards & usage refer their official documentation.

Steps to view logs


1. Once logged in, navigate to Explore in Grafana.

2. Once you are in the Explore page. Set the datasource as Loki.

3. Now use the Log Browser to select apps and namespace for which you want to view the logs. You can also write your own LogQL query as well.

Currently we are storing two types of logs:
1. Kubernetes components (nodes, pods, services, deployments)
2. Application logs (OpenMRS, appointments, bahmni-lab, bahmni-web, clinic-config, crater, ABDM components, nginx, etc)


4. You can filter labels to see the required logs.
Search can be done for a specific namespace, instance, container, application etc.


5. Post selection of filtered label, You can select the relevant component for which you want to see the logs.

Example 1: Let's suppose we have selected app as the label, now if we click on select value it will list down all the app available with logs.

Example 2: Let's suppose we have selected namespace as the label, now if we click on select value it will list down all the namespace available with logs.


Similarly can be done for other label as well.


Demo Video

To understand more details around how to use Monitoring & centralised Logging we have recorded the following DEMO on Youtube. This explains the login process & how to view logs. This is a demo that was done during the weekly PAT call some time ago.



(Duration: 20 mins)


Examples

Name

Query

Name

Query

1

QA server monitoring link

https://monitoring-lite.mybahmni.in/goto/OYnNoCKVz?orgId=1

2

QA server bahmni-web logs

(app: bahmni-web, namespace: qa)

3

QA server openmrs logs

(app: openmrs, namespace: qa)

4

All POST requests logs to QA server

{namespace="ingress-nginx"} |= `POST` |= `qa.lite.mybahmni` != `monitoring.lite`
5

All OpenMRS Admin UI login requests on QA server

{namespace="ingress-nginx"} |= `POST` |= `qa.lite.mybahmni` != `monitoring.lite` |= `/openmrs/ms/legacyui/loginServlet`

The Bahmni documentation is licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)