/
Agility monitoring

Agility monitoring

Owned by Juan Frias
Feb 05, 2025
7 min read

Composition

The monitoring stack is composed by four charts and has the following functionalities.

  • Agility-logging for logging collection by using the Loki and Opentelemetry collector for logs collection.

  • Agility-opentelemetry exposes the opentelemetry-collector service that metrics, traces and logs, then distributes them to its correct endpoint.

  • Agility-observability for trace analysis by using Grafana Tempo.

  • Agility-metrics which uses the full prometheus stack (Prometheus, Grafana and Alert Manager) including some other configurations like dashboards, custom prometheus rules and a Pager Duty service (Agility-pagerduty) for ingesting data from Pager Duty into Prometheus.

Install AGILITY Monitoring on Agility VM

Prerequisites

Before installing it, make sure you have the following prerequisites:

  • Admin Permissions. You should have Kubernetes administrative permissions to perform the installation.

  • B-Yond AGILITY. To install the monitoring stack, you will need to have previously installed AGILITY and have it activated with its proper license provided by B-Yond customer support.

 

Please ensure that you have fulfilled all these prerequisites before moving on to the next step.

Execute the following Agility CLI command to install the monitoring stack.

# agility monitoring install Namespace 'monitoring' created successfully. Secret agility-metrics created successfully. Performing 'agility-opentelemetry' upgrade to version 1.0.9 Namespace 'monitoring' already exists. Skipping creation. Docker registry secret 'byond-container-registry-credential' created in namespace 'monitoring' Pulling Helm chart 'agility-opentelemetry' version 1.0.9... Helm chart 'agility-opentelemetry' pulled successfully. Running Helm upgrade for chart 'agility-opentelemetry'... Helm upgrade executed successfully for chart 'agility-opentelemetry' Performing 'agility-metrics' upgrade to version 1.0.9 Namespace 'monitoring' already exists. Skipping creation. Docker registry secret 'byond-container-registry-credential' updated in namespace 'monitoring' Pulling Helm chart 'agility-metrics' version 1.0.9... Helm chart 'agility-metrics' pulled successfully. Running Helm upgrade for chart 'agility-metrics'... Helm upgrade executed successfully for chart 'agility-metrics' Performing 'agility-logging' upgrade to version 1.0.9 Namespace 'monitoring' already exists. Skipping creation. Docker registry secret 'byond-container-registry-credential' updated in namespace 'monitoring' Pulling Helm chart 'agility-logging' version 1.0.9... Helm chart 'agility-logging' pulled successfully. Running Helm upgrade for chart 'agility-logging'... Helm upgrade executed successfully for chart 'agility-logging' Performing 'agility-observability' upgrade to version 1.0.9 Namespace 'monitoring' already exists. Skipping creation. Docker registry secret 'byond-container-registry-credential' updated in namespace 'monitoring' Pulling Helm chart 'agility-observability' version 1.0.9... Helm chart 'agility-observability' pulled successfully. Running Helm upgrade for chart 'agility-observability'... Helm upgrade executed successfully for chart 'agility-observability' AGILITY Monitoring Stack was succesfully installed/updated. Grafana UI is available at: https://<vm-ip|vm-hostname|fqdn>/metrics Initial Admin Credentials: admin/******* Please run agility telemetry local enable to collect data into this monitoring platform.

The version installed is always the latest, but this can be changed by adding the --version option

Enable telemetry

Once the installation is completed, it’s important to enable telemetry with the following command.

# agility telemetry local enable Do you want to enable local data collection? [y/N]: y # agility telemetry local status Local usage data collection is enabled

As displayed on the message, this commands allows Agility to send its data to the newly installed monitoring application.

Test & Usage

As it says on the output, Grafana is exposed at <VM's IP>/metrics/. In this example https://10.1.255.125/metrics/

image-20250204-200701.png

You should see the following login page. By default, the user is admin and the password is provided in the installation output for Initial Admin Credential.

Exploring Grafana

To explore the data in Grafana, you can go to the Home button → Explore

image-20250204-201531.png

You should see the following interface. For this page, you’ll be able to see metrics, traces and logs.

Metrics

To view metrics, select the Prometheus datasource.

Under metrics, select the fast-api_app_info as example. Then, hit Run query and you should see something like this.

Traces

To view traces, select the Tempo datasource.

You might need to have performed, at least, one analysis on Agility to generate traces.

Then select the Search tab to view all the traces, the time you'd like to see and hit Run Query.

You can click on a Trace to view more details about the services involved on the analysis.

Logs

To view logs coming on Loki. Select the Loki datasource from the top left corner.

Then select a label to view the logs, the time you'd like to see and hit Run Query. The cv namespace was used for this example.

You'll be able to see the logs.

Dashboards

This stack also contains some dashboards curated for Agility. They show important information about the monitoring stack.

Agility Overview

This dashboard will show information about the general status of Agility like user sessions, DB size and MinIO capacity.

Agility Data Pipeline

PostgresSQL DB Summary

This dashboards shows deeper information related to the status of the postgres database running.

Alerting configuration

The monitoring stack two types of alerting systems, each one for a component.

  • Alert manager: Contains prometheus rules included by the kube-prometheus-stack that keeps an eye at cluster level components.

Alert manager

The cli offers the possibility of configuring and email or a pagerduty URL to send the alerts to.

# agility monitoring set --help Usage: agility monitoring set [OPTIONS] COMMAND [ARGS]... Configure Alert manager notifications Options: --help Show this message and exit. Commands: both Configure email and pagerduty notifications for Alert manager email Configure email notifications for Alert manager pagerduty Configure pager duty notifications for Alert manager

Email

These are the options available for the emailcommand.

These are the options for the pager duty configuration type.

As an example, you’ll configure an endpoint for alert manager using an email. Please run the following command.

Reset

In case you’ll like to remove the configuration, please run the following command.

Then run the showcommand. You should see the following message.

Uninstall

In order to remove the stack, please run the following command.