Grafana Loki Log Persistance Configuration

This document help us in enabling persistent logs in Grafana.
This setup is required when you want to store logs for longer duration for analysis & reference purpose.

Grafana Overview

The current Bahmni setup is using below tools for Monitoring & Observability:

  1. Monitoring & Alerting: - Promethus, Grafana

  2. Centralised Logging:- Loki Stack consisting of Promtail, Loki and Grafana

Grafana Loki needs to store two different types of data: chunks and indexes.
Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are compressed as “chunks” and saved in the chunks store.
The index stores each stream’s label set and links them to the individual chunks.

https://grafana.com/docs/loki/latest/configuration/ is the detail example available on Grafana page for Loki configurations i.e. loki.yaml

Supported Stores:- Grafana supports wide variety of storage to setup log retention.
A documented list is available on Grafana docs page as well.

Grafana Loki Storage Retention:- Retention in Grafana Loki is achieved either through the Table Manager or the Compactor.

Bahmni Configurations

Log retention is needed whenever we want to do some detail analysis or debug an issue.
Depends on the requirements log retention can be setup. We can setup for 1 day, 7 days, 15 days, 1 month or for a year depends how much budget is available for storage.

With respect to Bahmni, we have implemented persistent logs configurations inside our helm-charts.
All the implementation is automated & is executed using github workflows.
Bahmni is using Compactor storage setup.

In Bahmni, we are using below configurations to setup our log retention:

For detailed configuration refer this Grafana documentation. This explains all the supported contents and default values.

Currently we are storing two types of logs:
1. Kubernetes components (nodes, pods, services, deployments)
2. Application logs (OpenMRS, appointments, bahmni-lab, bahmni-web, clinic-config, crater, ABDM components, nginx, etc )

You can explore more on Bahmni MONITORING PORTAL

 


Below are the snippets of Bahmni configuration which is used for K8s & Docker-compose setup.

1. Kubernetes Environments

You can refer details of code on Helm Umbrella Chart repo.

config: auth_enabled: false ingester: chunk_idle_period: 3m chunk_block_size: 262144 chunk_retain_period: 1m max_transfer_retries: 0 wal: dir: /data/loki/wal lifecycler: ring: replication_factor: 1 limits_config: retention_period: 48h # CURRENT RETENTION TIME IMPLEMENTED enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h max_entries_limit_per_query: 5000 schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h server: http_listen_port: 3100 grpc_listen_port: 9095 storage_config: boltdb_shipper: active_index_directory: /data/loki/boltdb-shipper-active cache_location: /data/loki/boltdb-shipper-cache cache_ttl: 24h shared_store: filesystem filesystem: directory: /data/loki/chunks chunk_store_config: max_look_back_period: 0s table_manager: retention_deletes_enabled: false retention_period: 0s compactor: working_directory: /data/loki/boltdb-shipper-compactor shared_store: filesystem compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150

2. Docker Compose Environments

auth_enabled: false server: http_listen_port: 3100 common: path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h ruler: alertmanager_url: http://localhost:9093 limits_config: retention_period: 48h compactor: working_directory: /loki/boltdb-shipper-compactor shared_store: filesystem compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 ingester: chunk_idle_period: 3m chunk_block_size: 262144 chunk_retain_period: 1m max_transfer_retries: 0 wal: dir: /loki/wal lifecycler: address: 127.0.0.1 ring: replication_factor: 1

Steps to update the retention configuration for logs

1. Change retention_period value under limits_config section

limits_config: retention_period: 24h ---->CHANGE THIS VALUE AS PER REQUIREMENT

 

Then restart loki service by running docker compose restart loki


The Bahmni documentation is licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)