Grafana Loki Log Persistance Configuration
This document help us in enabling persistent logs in Grafana.
This setup is required when you want to store logs for longer duration for analysis & reference purpose.
Grafana Overview
The current Bahmni setup is using below tools for Monitoring & Observability:
Centralised Logging:- Loki Stack consisting of Promtail, Loki and Grafana
Grafana Loki needs to store two different types of data: chunks and indexes.
Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are compressed as โchunksโ and saved in the chunks store.
The index stores each streamโs label set and links them to the individual chunks.
Grafana Loki configuration parameters | Grafana Loki documentation is the detail example available on Grafana page for Loki configurations i.e. loki.yaml
Supported Stores:- Grafana supports wide variety of storage to setup log retention.
A documented list is available on Grafana docs page as well.
Grafana Loki Storage Retention:- Retention in Grafana Loki is achieved either through the Table Manager or the Compactor.
Bahmni Configurations
Log retention is needed whenever we want to do some detail analysis or debug an issue.
Depends on the requirements log retention can be setup. We can setup for 1 day, 7 days, 15 days, 1 month or for a year depends how much budget is available for storage.
With respect to Bahmni, we have implemented persistent logs configurations inside our helm-charts.
All the implementation is automated & is executed using github workflows.
Bahmni is using Compactor storage setup.
In Bahmni, we are using below configurations to setup our log retention:
Pre defined helm charts. loki-stack 2.10.2 ยท grafana/grafana
Storage:
boltdb-shipper
Storage size: 10 Gi
Retention duration: 48 hrs
For detailed configuration refer this Grafana documentation. This explains all the supported contents and default values.
Currently we are storing two types of logs:
1. Kubernetes components (nodes, pods, services, deployments)
2. Application logs (OpenMRS, appointments, bahmni-lab, bahmni-web, clinic-config, crater, ABDM components, nginx, etc )
You can explore more on Bahmni MONITORING PORTAL
ย
Below are the snippets
of Bahmni configuration which is used for K8s & Docker-compose setup.
1. Kubernetes Environments
You can refer details of code on Helm Umbrella Chart repo.
config:
auth_enabled: false
ingester:
chunk_idle_period: 3m
chunk_block_size: 262144
chunk_retain_period: 1m
max_transfer_retries: 0
wal:
dir: /data/loki/wal
lifecycler:
ring:
replication_factor: 1
limits_config:
retention_period: 48h # CURRENT RETENTION TIME IMPLEMENTED
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_entries_limit_per_query: 5000
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
server:
http_listen_port: 3100
grpc_listen_port: 9095
storage_config:
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
cache_ttl: 24h
shared_store: filesystem
filesystem:
directory: /data/loki/chunks
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
compactor:
working_directory: /data/loki/boltdb-shipper-compactor
shared_store: filesystem
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
2. Docker Compose Environments
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
limits_config:
retention_period: 48h
compactor:
working_directory: /loki/boltdb-shipper-compactor
shared_store: filesystem
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
ingester:
chunk_idle_period: 3m
chunk_block_size: 262144
chunk_retain_period: 1m
max_transfer_retries: 0
wal:
dir: /loki/wal
lifecycler:
address: 127.0.0.1
ring:
replication_factor: 1
Steps to update the retention configuration for logs
1. Change retention_period
value under limits_config
section
limits_config:
retention_period: 24h ---->CHANGE THIS VALUE AS PER REQUIREMENT
ย
Then restart loki service by running docker compose restart loki
The Bahmni documentation is licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)