[Monitoring] Grafana Loki - Persistent logs

1 Initial Context
2 Grafana Overview
3 BAHMNI Configurations

Initial Context

This document help us in enabling persistent logs in Grafana.
This setup is required when you want to store logs for longer duration for analysis & reference purpose.

Grafana Overview

The current Bahmni setup is using below tools for Monitoring & Observability:

Monitoring & Alerting: - Promethus, Grafana
Centralised Logging:- Loki Stack consisting of Promtail, Loki and Grafana

Grafana Loki needs to store two different types of data: chunks and indexes.
Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are compressed as “chunks” and saved in the chunks store.
The index stores each stream’s label set and links them to the individual chunks.

https://grafana.com/docs/loki/latest/configuration/ is the detail example available on Grafana page for Loki configurations i.e. loki.yaml

Supported Stores:- Grafana supports wide variety of storage to setup log retention.
A documented list is available on Grafana docs page as well.

Grafana Loki Storage Retention:- Retention in Grafana Loki is achieved either through the Table Manager or the Compactor.

BAHMNI Configurations

Log retention is needed whenever we want to do some detail analysis or debug an issue.
Depends on the requirements log retention can be setup. We can setup for 1 day, 7 days, 15 days, 1 month or for a year depends how much budget is available for storage.

With respect to Bahmni, we have implemented persistent logs configurations inside our helm-charts.
All the implementation is automated & is executed using github workflows.
Bahmni is using Compactor storage setup.

In Bahmni, we are using below configurations to setup our log retention:

Pre defined helm charts. https://artifacthub.io/packages/helm/grafana/loki-stack
Storage: boltdb-shipper
Storage size: 10 Gi
Retention duration: 48 hrs

For detailed configuration refer this Grafana documentation. This explains all the supported contents and default values.

Currently we are storing two types of logs:
1. Kubernetes components (nodes, pods, services, deployments)
2. Application logs (OpenMRS, appointments, bahmni-lab, bahmni-web, clinic-config, crater, ABDM components, nginx, etc )

You can explore more on Bahmni MONITORING PORTAL

Below are the snippets of Bahmni configuration which is used for K8s & Docker-compose setup.

1. KUBERNETES SETUP

You can refer details of code on Helm Umbrella Chart repo.

  config:
    auth_enabled: false

    ingester:
      chunk_idle_period: 3m
      chunk_block_size: 262144
      chunk_retain_period: 1m
      max_transfer_retries: 0
      wal:
        dir: /data/loki/wal
      lifecycler:
        ring:
          replication_factor: 1

    limits_config:
      retention_period: 48h    # CURRENT RETENTION TIME IMPLEMENTED
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_entries_limit_per_query: 5000

    schema_config:
      configs:
        - from: 2020-10-24
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index:
            prefix: index_
            period: 24h

    server:
      http_listen_port: 3100
      grpc_listen_port: 9095

    storage_config:
      boltdb_shipper:
        active_index_directory: /data/loki/boltdb-shipper-active
        cache_location: /data/loki/boltdb-shipper-cache
        cache_ttl: 24h
        shared_store: filesystem
      filesystem:
        directory: /data/loki/chunks

    chunk_store_config:
      max_look_back_period: 0s

    table_manager:
      retention_deletes_enabled: false
      retention_period: 0s

    compactor:
      working_directory: /data/loki/boltdb-shipper-compactor
      shared_store: filesystem
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150

2. DOCKER-COMPOSE SETUP

auth_enabled: false

server:
    http_listen_port: 3100

common:
    path_prefix: /loki
    storage:
        filesystem:
            chunks_directory: /loki/chunks
            rules_directory: /loki/rules
    replication_factor: 1
    ring:
        kvstore:
            store: inmemory

schema_config:
    configs:
        - from: 2020-10-24
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index:
              prefix: index_
              period: 24h

ruler:
    alertmanager_url: http://localhost:9093

limits_config:
    retention_period: 48h

compactor:
    working_directory: /loki/boltdb-shipper-compactor
    shared_store: filesystem
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150

ingester:
    chunk_idle_period: 3m
    chunk_block_size: 262144
    chunk_retain_period: 1m
    max_transfer_retries: 0
    wal:
        dir: /loki/wal
    lifecycler:
        address: 127.0.0.1
        ring:
            replication_factor: 1

STEPS TO FOLLOW WHILE MODIFYING RETENTION TIME IN CONFIGURATION FILE

FOR KUBERNETES & DOCKER-COMPOSE

1. Change retention_period value under limits_config section

limits_config:
      retention_period: 24h   #CHANGE THIS VALUE AS PER REQUIREMENT