Agility monitoring

By Juan Frias
7 min

Composition

The monitoring stack is composed by four charts and has the following functionalities.

  • Agility-logging for logging collection by using the Loki and Opentelemetry collector for logs collection.

  • Agility-opentelemetry exposes the opentelemetry-collector service that metrics, traces and logs, then distributes them to its correct endpoint.

  • Agility-observability for trace analysis by using Grafana Tempo.

  • Agility-metrics which uses the full prometheus stack (Prometheus, Grafana and Alert Manager) including some other configurations like dashboards, custom prometheus rules and a Pager Duty service (Agility-pagerduty) for ingesting data from Pager Duty into Prometheus.

Install AGILITY Monitoring on Agility VM

Prerequisites

Before installing it, make sure you have the following prerequisites:

  • Admin Permissions. You should have Kubernetes administrative permissions to perform the installation.

  • B-Yond AGILITY. To install the monitoring stack, you will need to have previously installed AGILITY and have it activated with its proper license provided by B-Yond customer support.

 

Please ensure that you have fulfilled all these prerequisites before moving on to the next step.

Execute the following Agility CLI command to install the monitoring stack.

# agility monitoring install Namespace 'monitoring' created successfully. Secret agility-metrics created successfully. Performing 'agility-opentelemetry' upgrade to version 1.0.9 Namespace 'monitoring' already exists. Skipping creation. Docker registry secret 'byond-container-registry-credential' created in namespace 'monitoring' Pulling Helm chart 'agility-opentelemetry' version 1.0.9... Helm chart 'agility-opentelemetry' pulled successfully. Running Helm upgrade for chart 'agility-opentelemetry'... Helm upgrade executed successfully for chart 'agility-opentelemetry' Performing 'agility-metrics' upgrade to version 1.0.9 Namespace 'monitoring' already exists. Skipping creation. Docker registry secret 'byond-container-registry-credential' updated in namespace 'monitoring' Pulling Helm chart 'agility-metrics' version 1.0.9... Helm chart 'agility-metrics' pulled successfully. Running Helm upgrade for chart 'agility-metrics'... Helm upgrade executed successfully for chart 'agility-metrics' Performing 'agility-logging' upgrade to version 1.0.9 Namespace 'monitoring' already exists. Skipping creation. Docker registry secret 'byond-container-registry-credential' updated in namespace 'monitoring' Pulling Helm chart 'agility-logging' version 1.0.9... Helm chart 'agility-logging' pulled successfully. Running Helm upgrade for chart 'agility-logging'... Helm upgrade executed successfully for chart 'agility-logging' Performing 'agility-observability' upgrade to version 1.0.9 Namespace 'monitoring' already exists. Skipping creation. Docker registry secret 'byond-container-registry-credential' updated in namespace 'monitoring' Pulling Helm chart 'agility-observability' version 1.0.9... Helm chart 'agility-observability' pulled successfully. Running Helm upgrade for chart 'agility-observability'... Helm upgrade executed successfully for chart 'agility-observability' AGILITY Monitoring Stack was succesfully installed/updated. Grafana UI is available at: https://<vm-ip|vm-hostname|fqdn>/metrics Initial Admin Credentials: admin/******* Please run agility telemetry local enable to collect data into this monitoring platform.

The version installed is always the latest, but this can be changed by adding the --version option

Enable telemetry

Once the installation is completed, it’s important to enable telemetry with the following command.

# agility telemetry local enable Do you want to enable local data collection? [y/N]: y # agility telemetry local status Local usage data collection is enabled

As displayed on the message, this commands allows Agility to send its data to the newly installed monitoring application.

Test & Usage

As it says on the output, Grafana is exposed at <VM's IP>/metrics/. In this example https://10.1.255.125/metrics/

image-20250204-200701.png

You should see the following login page. By default, the user is admin and the password is provided in the installation output for Initial Admin Credential.

Exploring Grafana

To explore the data in Grafana, you can go to the Home button → Explore

image-20250204-201531.png

You should see the following interface. For this page, you’ll be able to see metrics, traces and logs.

image-20250205-172223.png

Metrics

To view metrics, select the Prometheus datasource.

image-20250204-202512.png

Under metrics, select the fast-api_app_info as example. Then, hit Run query and you should see something like this.

image-20250204-201925.png

Traces

To view traces, select the Tempo datasource.

image-20250205-163324.png

You might need to have performed, at least, one analysis on Agility to generate traces.

Then select the Search tab to view all the traces, the time you'd like to see and hit Run Query.

image-20250205-163506.png

You can click on a Trace to view more details about the services involved on the analysis.

image-20250205-163643.png

Logs

To view logs coming on Loki. Select the Loki datasource from the top left corner.

image-20250204-202011.png

Then select a label to view the logs, the time you'd like to see and hit Run Query. The cv namespace was used for this example.

image-20250204-202304.png

You'll be able to see the logs.

image-20250204-202619.png

Dashboards

This stack also contains some dashboards curated for Agility. They show important information about the monitoring stack.

image-20250204-203408.png

Agility Overview

This dashboard will show information about the general status of Agility like user sessions, DB size and MinIO capacity.

image-20250205-162206.png

Agility Data Pipeline

image-20250205-162541.png

PostgresSQL DB Summary

This dashboards shows deeper information related to the status of the postgres database running.

image-20250205-163146.png

Alerting configuration

The monitoring stack two types of alerting systems, each one for a component.

  • Alert manager: Contains prometheus rules included by the kube-prometheus-stack that keeps an eye at cluster level components.

Alert manager

The cli offers the possibility of configuring and email or a pagerduty URL to send the alerts to.

# agility monitoring set --help Usage: agility monitoring set [OPTIONS] COMMAND [ARGS]... Configure Alert manager notifications Options: --help Show this message and exit. Commands: both Configure email and pagerduty notifications for Alert manager email Configure email notifications for Alert manager pagerduty Configure pager duty notifications for Alert manager

Email

These are the options available for the emailcommand.

# agility monitoring set email --help Usage: agility monitoring set email [OPTIONS] Configure email notifications for Alert manager Options: --email-from TEXT Specify the adddress from email will come. --email-to TEXT Specify the adddress where email will be sent to. --smarthost TEXT Specify the SMTP for email service. <host>:<port> --auth-password TEXT Specify the auth password for SMTP service. --auth-username TEXT Specify the user for SMTP service. --require-tls Enable TLS if required. --send-resolved Send resolved alerts notifications. --namespace TEXT Namespace in where monitoring is installed. Default 'monitoring'. --help Show this message and exit.

These are the options for the pager duty configuration type.

# agility monitoring set pagerduty --help Usage: agility monitoring set pagerduty [OPTIONS] Configure pager duty notifications for Alert manager Options: --pager-duty-service-key TEXT Specify Pager duty service key --pager-duty-url TEXT Specify the pager duty url. --namespace TEXT Namespace in where monitoring is installed. Default 'monitoring'. --help Show this message and exit.

As an example, you’ll configure an endpoint for alert manager using an email. Please run the following command.

# agility monitoring set email --email-from no-reply@b-yond.com --email-to=my-email@domain.com --send-resolved Secret agility-alertmanager-config created successfully. # agility monitoring show ###### EMAIL CONFIGURED ###### Destination: my-email@domain.com Origin: no-reply@b-yond.com SMTP : None TLS required: false Send resolved: true

Reset

In case you’ll like to remove the configuration, please run the following command.

# agility monitoring reset Secret agility-alertmanager-config updated successfully.

Then run the showcommand. You should see the following message.

# agility monitoring show No receviers configured for alert manager. Please try running agility monitoring set --help to see the options available to configure it.

Uninstall

In order to remove the stack, please run the following command.

# agility monitoring uninstall Do you really want to uninstall the AGILTY Monitoring Stack? [y/N]: y Removing Helm release 'agility-observability'... Helm delete executed successfully for release 'agility-observability' Removing Helm release 'agility-logging'... Helm delete executed successfully for release 'agility-logging' Removing Helm release 'agility-metrics'... Helm delete executed successfully for release 'agility-metrics' Removing Helm release 'agility-opentelemetry'... Helm delete executed successfully for release 'agility-opentelemetry' Namespace 'monitoring' deleted successfully. AGILITY Monitoring Stack was succesfully uninstalled. Please run agility telemetry local disable to stop collecting data into this monitoring platform..