Determine disk space requirements

Disk space will depend on the number of PCAP files you want to process daily.

Initial specifications depending on the number of PCAP files to be processed daily:

The Agility Monitoring Stack, which includes a comprehensive metrics, logging, and traces platform requires allocating additional resources:

• CPU: 2

• Memory: 4GB

• Disk: 40GB

The disk requirement applies either to the boot disk, when it is the sole storage option, or to the external disk when it is used as an alternative.

Choose your installation

A simple and straightforward way to utilize AGILITY is by provisioning an existing Cloud image. This option is available in both public and private cloud environments.

The access to the VM image will be provided by B-Yond.

On-Premises Virtualization Platforms

• OpenStack

• VMware

Public Clouds

• AWS: The AMI (Amazon Machine Image) ID will be shared with target account.

• Azure: The Azure VM image will be shared with target subscription/tenant.

• Google Cloud: The Google Cloud VM image will be shared with target organization.

If you are using other Cloud providers or virtualization solutions, you may need to convert the qcow2 or VMware disk images to the format required by your platform. Consult the documentation of your specific provider or platform for instructions on image conversion.

Begin Installation

OpenStack

From the email sent from B-Yond, download the qcow2 image specifically configured for OpenStack.

1. (As an administrator) Create an image:

   Code Block
   glance image-create --disk-format qcow2 --container-format bare --file ./Agility-X.YY.Z-AlmaLinux-X-GenericCloud-X.Y-YYYYMMDD.x86_64.qcow2 --min-disk 25 --min-ram 2048 --name Agility-X.Y.Z

2. (As an administrator) Create a member for the glance image:

   Code Block
   glance member-create <image-id> <member-id>

3. (As an administrator) Accept the membership for the glance image:

   Code Block
   glance member-update <image-id> <member-id> accepted

4. (As a user) Create a VM using the image (minimum use m1.medium which is 2 CPU / 4096 RAM / 40G disk):

openstack server create --flavor <your-flavor> --image <image-id> agility  --nic net-id=<network-id> --security-group <your-security-group> --key-name <your-key>

VMware ESXi

From the email sent from B-Yond, download the provided VMware disk image specifically configured for VMware virtualization environments.

1. To import a virtual machine stored on a VMware Hosted product to an ESX/ESXi host, run:

   Code Block
   vmkfstools -i virtual_machine.vmdk /vmfs/volumes/datastore/my_virtual_machine_folder/virtual_machine.vmdk

2. Create the VM using the imported disk. Option Guest OS: Other Linux (64-bit).

3. Using the console login as root , password almalinux.

4. Set up static network configuration, e.g. using nmtui.

5. Increase VM disk size:
   -Increase disk size from ESXi
   -Rescan using echo 1>/sys/class/block/sda/device/rescan
   -Recreate the partition 2 with fdisk

   Code Block
   printf "d\n\nn\n\n\n\np\nw\n" | fdisk /dev/sda

   -Increase the filesystem size using xfs_growfs /dev/sda2

6. Configure ssh options, e.g. set authorized keys for default cloud-user almalinux or another user.

...

...

A simple and straightforward way to utilize AGILITY is by provisioning an existing Cloud image. This option is available in both public and private cloud environments.

The access to the VM image will be provided by B-Yond.

Public Clouds:

AWS: The AMI (Amazon Machine Image) ID will be shared with target account.

Azure: The Azure VM image will be shared with target subscription/tenant.

Google Cloud: The Google Cloud VM image will be shared with target organization.

On-Premises Virtualization Platforms:

OpenStack: Download the provided qcow2 image specifically configured for OpenStack.

VMware: Download the provided VMware disk image specifically configured for VMware virtualization environments.

If you are using other Cloud providers or virtualization solutions, you may need to convert the qcow2 or VMware disk images to the format required by your platform. Consult the documentation of your specific provider or platform for instructions on image conversion.

Using the B-Yond provided images is recommended as they are pre-configured and optimized for running AGILITY.

Provision the VM

Initial specifications depending on the number of PCAP files to be processed daily:

ℹ️
Assumptions: processing 100 files/day, retention period is 3 days.

Please note that the disk requirement applies either to the boot disk, when it is the sole storage option, or to the external disk when it is used as an alternative.

ℹ️
The values associated with AGILITY application or its monitoring stack can be customized according to specific requirements and file sizes.

OpenStack

1. (As an administrator) Create an image:

   Code Block
   language bash
   glance image-create --disk-format qcow2 --container-format bare --file ./Agility-X.YY.Z-AlmaLinux-X-GenericCloud-X.Y-YYYYMMDD.x86_64.qcow2 --min-disk 25 --min-ram 2048 --name Agility-X.Y.Z

2. (As an administrator) Create a member for the glance image:

   Code Block
   language bash
   glance member-create <image-id> <member-id>

3. (As an administrator) Accept the membership for the glance image:

   Code Block
   language bash
   glance member-update <image-id> <member-id> accepted

4. (As a user) Create a VM using the image (minimum use m1.medium which is 2 CPU / 4096 RAM / 40G disk):

   Code Block
   language bash
   openstack server create --flavor <your-flavor> --image <image-id> agility --nic net-id=<network-id> --security-group <your-security-group> --key-name <your-key>

Public Clouds (AWS, Azure, GCP, etc.)

Follow the procedures specified by your Cloud provider. These procedures typically include the following steps:

• Image selection: Choose the AGILITY VM image obtained from B-Yond or the converted image.

• Shape specification: Specify the number of virtual CPUs (vCPUs) and RAM for the instance.

• Boot disk specification: Define the size and type of the boot disk.

• Networking configuration: Configure the network settings for the VM.

• Public SSH key(s): Provide the SSH key(s) that will be used to access the VM.

• Provide an init-cloud script to run (this is in general an optional step).

Access the VM

1. SSH in using the cloud-user and the associated private key:
   –           Generic Cloud: almalinux
   – AWS - AMI: ec2-user

   Code Block
   ssh -i <private_key> <cloud-user>@<vm_ip>

2. Verify that all components are up and running:

   Code Block
   sudo su -

   Code Block
   kubectl get pods -A

   All Kubernetes pods should be in Runningand Ready status.

3. Access the user interface (UI):
   Open your browser and put the AGILITY VM IP, .e.g https://10.0.0.1/cv/
   Use the following credentials:
   username: agility-admin@b-yond.com
   password: agility-admin@b-yond.com

...

• optional step).

ℹ️
The VM boot time might take between 5 and 10 minutes in total.

VMware ESXi

1. To import a virtual machine stored on a VMware Hosted product to an ESX/ESXi host, run:

   Code Block
   language bash
   vmkfstools -i virtual_machine.vmdk /vmfs/volumes/datastore/my_virtual_machine_folder/virtual_machine.vmdk

2. Create the VM using the imported disk. Option Guest OS: Other Linux (64-bit)

3. Using the console login as root , password almalinux

4. Set up static network configuration, e.g. using nmtui.

5. Increase VM disk size:

   • Increase disk size from ESXi

   • Rescan using echo 1>/sys/class/block/sda/device/rescan

   • Recreate the partition 2 with fdisk

   Code Block
   language bash
   printf "d\\n\\nn\\n\\n\\n\\np\\nw\\n" | fdisk /dev/sda

   • Increase the filesystem size using xfs_growfs /dev/sda2

Accessing the VM

1. SSH into it using the cloud-user and the associated private key:

   • AMI: ec2-user

   • Generic Cloud: almalinux

   Code Block
   language bash
   ssh -i <private_key> <cloud-user>@<vm_ip>

[Optional] Configure DNS Servers

ℹ️
The DNS server is by default provided via DHCP. This section is relevant if you need to specify an additional DNS server or

...

if the DHCP option is unavailable.

To configure nameservers, domain search suffixes, etc., use the NetworkManager tool:

1. Check the current DNS configuration:

   Code Block
   language bash
   cat /etc/resolv.conf

   Example An example output:

   Code Block
   language bash
   cat /etc/resolv.conf # Generated by NetworkManager nameserver 169.254.169.254

2. Identify the network connection to configure:

   Code Block
   language bash
   sudo nmcli con show

   Example This is an example of the output:

   NAME         UUID                                  TYPE     

   Code Block
   language bash	bash
   NAME UUID TYPE DEVICE System eth0 eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 d6edd65f3e03 ethernet ethernet  eth0 cni0        cni0 4e4c9ecf-cc82-49eb-bda6-99317c953691  bridge   99317c953691 bridge cni0 flannel.1   1 021d7133-dad5-4a02-a035-b11009ac943a  vxlan    b11009ac943a vxlan flannel.1

3. Add a new DNS server:

   Code Block
   language bash
   sudo nmcli con mod <connection-name> +ipv4.dns <dns-server-ip> sudo nmcli con up <connection-name>For name>

   For example,

   to

   add Google’s DNS server to the device eth0, the commands will

   add Google's DNS server to the device eth0, the commands will be:

   Code Block
   language bash
   sudo nmcli con mod "System eth0" +ipv4.dns 8.8.8.8 sudo nmcli con up "System eth0"

4. To check the change run againcheck the change run again:

   Code Block
   language bash
   $ cat /etc/resolv.conf # Generated by NetworkManager nameserver 8.8.8.8 nameserver 169.254.169.254

Removing DHCP DNS

For reference purposes, to remove the DNS server specified by DHCP run the following commands:

...

sudo nmcli 

...

Removing DHCP DNS

...

con mod <connection-name> ipv4.ignore-auto-dns yes
sudo nmcli con up <connection-name>

This will leave only the DNS servers configured manually.

Specify domain search

Use the ipv4.dns-search option to change the domain name if necessary. Ensure that the correct fully qualified domain name (FQDN) is set before by using the hostnamectl set-hostname command.

These commands have to be executed, adjust accordingly:

sudo nmcli con mod <connection-name> +ipv4.ignore-auto-dns yesdns-search <domain>
sudo nmcli con up <connection-name>

For example, to add a domain name in the search list (here http://example.com ), run:

$ sudo nmcli con up <connection-name>

This will leave only the DNS servers configured manually.

Changing the domain name

If you need to change to domain name, use the ipv4.dns-search option. Ensure that the correct fully qualified domain name (FQDN) is set before by using the hostnamectl set-hostname command.

Executed the following commands:

 mod "System eth0"  +ipv4.dns-search example.com
$ sudo nmcli con mod <connection-name> +ipv4.dns-search <domain>
sudo nmcli con up <connection-name>

For example, to add a domain name in the search list (here http://example.com ), run:

$ sudo nmcli con mod "System eth0"  +ipv4.dns-search example.com 
$ sudo nmcli con up "System eth0"
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)
$ cat /etc/resolv.conf
# Generated by NetworkManager
search example.com
nameserver 8.8.8.8
nameserver 169.254.169.254

Configure System Clock (Optional)

 up "System eth0"
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)
$ cat /etc/resolv.conf
# Generated by NetworkManager
search example.com
nameserver 8.8.8.8
nameserver 169.254.169.254

[!WARNING]
There should be at least 1 (one) nameserver defined for AGILITY in the VM.

[Optional] Configuring Proxy

1. Set proxy settings to the System wide

   Code Block
   vi /etc/profile.d/proxy.sh MY_PROXY_URL="example.com:3128" HTTP_PROXY=$MY_PROXY_URL HTTPS_PROXY=$MY_PROXY_URL FTP_PROXY=$MY_PROXY_URL http_proxy=$MY_PROXY_URL https_proxy=$MY_PROXY_URL ftp_proxy=$MY_PROXY_URL export HTTP_PROXY HTTPS_PROXY FTP_PROXY NO_PROXY http_proxy https_proxy ftp_proxy no_proxy

   There is no NO_PROXY standard and depends on the application.

2. Load proxy configuration

   Code Block
   source /etc/profile.d/proxy.sh

[Optional] Configuring System Clock

ℹ️
This section is crucial for situations where non-default NTP servers are required or when there are limitations in accessing external public ones.

AGILITY VM facilitates clock synchronization using the Chrony service, which is enabled by default and synchronizes with a pool of public NTP servers.

Using a Custom NTP Server

To synchronize the VM clock with a specific NTP server, follow these instructions:

1. Check the current configured servers:

   Code Block
   language bash
   chronyc sources

   Example output:

   Code Block
   language bash
   $ chronyc sources MS Name/IP address        IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^- 68.64.173.196                 2  10   377   132 196 2 10 377 132 +6288us[+6288us] +/-  119ms ^- tick.srs1.ntfo.org            3   9   377    57  org 3 9 377 57 -693us[ -693us] +/-  148ms ^* http://ntp1.wiktel.com        1  10   377   738   1 10 377 738 +362us[ +214us] +/-   22ms ^+ 23.150.40.242                 2  10   377   103 242 2 10 377 103 -1361us[-1361us] +/-   32ms

2. Add your server definition in the file /etc/chrony.conf:

   Code Block
   language bash
   server <my-server-ip>

   For example, using a public cloud NTP server:

   Code Block
   language bash
   echo "server 169.254.169.254" | sudo tee -a /etc/chrony.conf

3. Comment out the entry pool 2.almalinux.pool.ntp.org iburst to enforce using only the specified NTP server:

   Code Block
   language bash
   sudo sed -i '/^pool 2\\.almalinux\\.pool\\.ntp\\.org iburst/s/^/#/' /etc/chrony.conf

4. Restart the Chrony service:

   Code Block
   language bash
   sudo systemctl restart chronyd

5. Check the changes were applied (Wait until the status changes from ^? to ^*, it might take several minutes):

   Code Block
   language bash
   chronyc sources

   Example output:

   Code Block
   language bash
   $ chronyc sources MS Name/IP address        address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 169.254.169.254               2   6     3    51  254 2 6 3 51 -491us[ -491us] +/-   23ms

6. Enable NTP and trigger a synchronization:

   Code Block
   language bash
   sudo timedatectl set-ntp true sudo chronyc -a makestep

7. Verify the clock is synchronized:

   Code Block
   language bash
   timedatectl

   Example The output should resemble:

   Code Block
   language bash
   $ timedatectl                Local time: Mon 2024-03-11 22:05:42 UTC            Universal time: Mon 2024-03-11 22:05:42 UTC                  RTC time: Mon 2024-03-11 22:05:43                -11 22:05:43 Time zone: UTC (UTC, +0000) System clock synchronized: yes               NTP service: active           RTC in local TZ: no

8. To confirm Chrony chrony tracking, run the command:

   Code Block
   language bash
   chronyc tracking The output also shows the configured NTP server:

   The output also shows the configured NTP server:

   Code Block
   language bash
   Reference ID ID    : A9FEA9FE (169.254.169.254) Stratum        254) Stratum : 3 Ref time (UTC)  : Mon Mar 11 22:06:03 2024 System time time     : 0.000000751 seconds slow of NTP time Last offset offset     : -0.000023889 seconds RMS offset offset      : 0.000017941 seconds Frequency Frequency       : 18.960 ppm slow Residual freq  freq : -0.001 ppm Skew           Skew : 0.011 ppm Root delay delay      : 0.000524478 seconds Root dispersion : 0.010530258 seconds Update interval : 1026.3 seconds Leap status status     : Normal

9. Ensure the Chrony service is available after reboot:

   Code Block
   language bash
   sudo systemctl enable chronyd

...

Configuring the

...

Timezone

Your system’s time zone system's timezone settings are stored in the /usr/share/zoneinfo directory. To ensure your system is set to the appropriate time zonetimezone, such as Europe/Paris, execute the following command:

...

Additionally, you can confirm your current time zone timezone by inspecting the /etc/localtime file:

ls -l /etc/localtime

[Optional] Attach an External Disk

...

In cases where external disk attachment is necessary, follow these steps. This will depend on the type of external disk used.

Prepare the VM

1. Access the VM using ssh

2. Stop the processes:

   Code Block
   language bash
   sudo su - systemctl stop k3s

3. Place the persisted data into a different location:

   Code Block
   language bash
   mv /var/lib/rancher/k3s/storage /var/lib/rancher/k3s/storage-bkp

NFS example

1. Create a directory on your Ubuntu system to serve as the mount point for the NFS share:

   Code Block
   language bash
   sudo mkdir -p /var/lib/rancher/k3s/storage

2. Edit the /etc/fstab file as root using a text editor, such as nano or vim:

   code

   Code Block
   language
   bash
   sudo nano /etc/fstab

3. Add an entry at the end of the /etc/fstab file to specify the NFS share and the mount point. The entry should follow this format:

   Code Block
   language bash
   <NFS_server_IP_or_hostname>:<remote_directory> <local_mount_point> nfs defaults 0 0

   Replace  Replace <NFS_server_IP_or_hostname> with the IP address or hostname of the NFS server,   <remote_directory> with the path of the directory you want to mount, and  and <local_mount_point> with the path of the local mount point you created in Step 1.

   For example, if the NFS server IP address is 192.168.1.100 and the remote directory you want to mount is /data, the entry would look like this:

   Code Block
   language bash
   192.168.1.100:/data /var/lib/rancher/k3s/storage nfs defaults 0 0

   Save the changes and exit the text editor.

4. To mount all entries listed in /etc/fstab, you can use the mount -a command.

   Ensure that your VM has network connectivity to the NFS server and that you have the necessary permissions to access the NFS share.

Block volume example

Your AGILITYloud AGILITY cloud provider gives you the ability to provision block storage and attach the disk to your VM. Follow Please follow the recommended procedures. E.g., it involves several iscsi commands executions.

1. Once attachedAttached, format the disk (e.g., sdb):

   Code Block
   language bash
   export DEV_PATH=sdb export MOUNT_PATH=/var/lib/rancher/k3s/storage sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/${DEV_PATH} sudo mkdir -p ${MOUNT_PATH} sudo mount -o discard,defaults /dev/${DEV_PATH} ${MOUNT_PATH} sudo chmod 775 ${MOUNT_PATH}

2. Persist the changes:

   Code Block
   language bash
   sudo cp /etc/fstab /etc/fstab.backup UUID=$(sudo blkid -s UUID -o value /dev/${DEV_PATH}) echo $UUID echo UUID=$(sudo blkid -s UUID -o value /dev/${DEV_PATH}) ${MOUNT_PATH} ext4 _netdev,nofail 0 2 | sudo tee -a /etc/fstab

Restore data

...

1. /etc/fstab

Restore data

1. Copy the data to the newly mounted external location:

   Code Block
   language bash
   sudo su - cp -R /var/lib/rancher/k3s/storage-bkp/* /var/lib/rancher/k3s/storage/

2. Start the processes:

   Code Block
   language bash
   systemctl start k3s

3. Wait a few seconds and ensure that all services are in the Running state:

   Code Block
   language bash
   kubectl get pods -n agility

4. Verify that the system is functioning correctly by performing tasks in the UI.

5. Once you have confirmed everything is working as expected, you can delete the old data:

   Code Block
   language bash
   rm -fr /var/lib/rancher/k3s/storage-bkp

   Please note that these steps assume you have the necessary permissions and understand the implications of deleting the old data. Exercise caution while performing these operations.