Security Scanning with Trivy

Scanning Vulnerabilities, Secrets, Container Images, Kubernetes Cluster and Secrets with Trivy

Vulnerability Scanning with Trivy

Trivy scans for OS packages and software dependencies in use (SBOM), Known vulnerabilities (CVEs), IaC issues and misconfigurations, Sensitive information and secrets and Software licenses while doing a scan. Trivy supports various types of targets like Container Images, Filesystem, Git Repository (remote), Virtual Machine Image, Kubernetes and AWS.

Bahmni uses Trivy in the CI/CD Pipelines at various stages - from writing the code -> pushing it to GitHub -> building Images to when the code is running the Kubernetes Cluster

Filesystem Scan

To perform a Filesystem scan, Trivy must be installed on the system. Follow the installation steps here. Once installed, a filesystem scan can be performed by trivy fs <path-to-scan>. A Filesystem scans by default scans the files in your system for Vulnerabilities (language-specefic files) and Secrets.

Bahmni utilises Trivy’s Filesystem Scan feature for doing a Static Code Analysis and Secrets Scanning on some of the repositories so that whenever the Developers push their code, before the build happens, it goes through a scan to make sure that the code does not have any out-dated and vulnerable libraries or secrets. To see this in detail, check out the Secrets Scanning Section.

Container Image Scanning

To scan a container image with Trivy, simply specify an image name (and a tag).:

This pulls the image from its remote repository and performs the scan. The scan results will report Critical, High, Medium and Unknown level vulnerabilities and these results will be grouped together as per their source.

To make sure no vulnerabilities are being pushed to Dockerhub from our Images Bahmni has Trivy Scanning enabled at multiple steps in the pipelines. At first, when the code is pushed we perform a Filesystem scan that scans the files for Vulnerabilities and secrets. If Any Critical or High level Vulnerability is found the pipeline stops even before build and notifies the Developers about it. Another benefit is that the artifacts built from that code are also generated free from the Vulnerabilities. If the Filesystem scan succeeds, then after performing the build again a Trivy Image scan is performed. This time Trivy scans the layers of the container Images including the Base Image used. Checkout the example of such a pipeline here.

Kubernetes Cluster Scanning

Once the workloads are deployed to Kubernetes, Bahmni has set up automated, continual resource scanning of the running workloads. Scanning the running workloads becomes important because even if the Image scan was successful at the time of build and publish, after some time if the image is not updated some of the libraries or the base image can become vulnerable.

Our setup includes a workflow that runs daily. Trivy as of now does not support generation of reports in html format or some other visual format therefore, we use custom scripts to generate such files and visualise the reports. At first the whole cluster is scanned and a cluster summary is generated and stored in a txt file. After that detailed reports are generated for each namespaces. These files are separated as per their severity. After generating all these reports, a customised webpage visualises them and makes them easier to read. This customised setup is hosted as GitHub Pages. Its code can be found here and the customisation is done as part of this job.

At the end of the scan, a slack notification is automatically sent to Bahmni’s #cicd-monitor channel. This is also customisable and can be replaced with any other notification service.

Secrets Scanning

Trivy, by default checks for Secrets like passwords, api keys, and tokens while performing a scan. The secret scanning works by using its built-in rules which allows it only report very sensitive secrets like AWS access keys, GCP service accounts, GitHub personal access tokens, GitLab personal access tokens, Slack access tokens etc. This is done to minimize false positives. Here is the full list of Trivy’s built in rules.

Bahmni uses Trivy as a Secrets Scanner in the pipelines. Here is an example workflow for Trivy Secrets Scanning in piplelines. The recommended way of dealing with secrets is to have Talisman setup in Dev-machines as a global hook. Here are the instructions to set it up.

Trivy uses trivy-secret.yaml file, which can be put in the project root location. Custom rules are defined in this file. This file can be used to add secrets or files that we want to skip/allow during Trivy's scan for secrets. To skip/allow a particular secret add that secret in the regex field. We can give an id or description of that secret as well. To skip/allow a file, use the path field.