...
The AGILITY Active Probe is designed to automate triggering analysis, collecting results, and tracking performance metrics.
Prerequisites:
Basic understanding of network diagnostics, SFTP, and S3 operations.
Installed dependencies for the Active Probe (see Dependencies section)Grafana access for monitoring and visualization.
The Active Probe’s dependencies are automatically installed during deployment.
...
Running the Active Probe
...
Automatic Triggering: The probe automatically triggers diagnostic analysis , either on demand or automatically according to a defined schedule.predefined schedule set during deployment. This schedule is expressed in minutes and is configured via the
EXECUTION_SCHEDULEenvironment variable:Code Block name: EXECUTION_SCHEDULE value: {{ (index .Values "active-probe" "interval") | default "15" | quote }}Analysis Collection: The Active Probe fetches collects analysis results using by accessing AGILITY API endpoints (e.g.,
/v1/analysisand/v1/analysis/{analysis_id}/summary).Result Validation: The probe compares the collected results with expected outcomes, logging failures the success or successes failure based on the analysis status.
Monitoring and Viewing Results
...
Handling Failures and Retries
In case of failure, the Active Probe employs automatic retry logic to ensure consistent results:addition, probe is designed to gracefully handle errors in all stages of operation.
Retry Logic: The probe automatically retries the analysis collection process if initial attempts fail.
Failure Logging: If the analysis cannot be completed successfully within the retry window, the failure is logged, and counters are updated accordingly.
...
The Active Probe integrates with OpenTelemetry to provide detailed metrics on network performance:
active_probe_attempt_count: Total number of analysis attempts.
active_probe_analysis_count: Total number of successfully completed analyses.
These metrics provide insights into probe performance and diagnostics, with additional labels for classificationAdditionally, the probe sends the status of each analysis (success, failure, and other statuses) as labels within these metrics, allowing for more granular tracking and analysis of probe activity.
...
Troubleshooting
For troubleshooting, follow these steps:
SFTP/Minio S3 Configuration: Ensure that file locations are accessible by the Active Probe.
API Connectivity: Verify that the AGILITY API is responding and providing expected data.
Telemetry Monitoring: Confirm that OpenTelemetry metrics are correctly forwarded to monitoring systems.
Log Review: Examine logs for errors related to analysis collection or comparison failures.
Dependencies
Ensure the following dependencies are installed for proper operation:
APScheduler==3.10.4
requests==2.31.0
PyYAML==6.0
opentelemetry-api==1.21.0
opentelemetry-sdk==1.21.0
opentelemetry-exporter-otlp==1.21.0
boto3==1.26.96
...