When AGILITY is installed in customer premises (either private or public clouds controlled by customers), two optional components are included with the AGILITY image and are advisable to deploy.

AGILITY Toolbox

AGILITY-Toolbox is a subject matter expert (SME) execution environment where B-Yond SMEs can perform model improvement-related work at customer-premises. AGILITY-Toolbox allows making incremental changes to deployed models without taking any customer data out of the premises.

• AGILITY-Toolbox has Wireshark and Docker enabled.

• AGILITY-Toolbox must have HTTPS access to AGILITY via the AGILITY API as well as HTTPS access to B-Yond’s artifact registry (hosted on Oracle US-East cloud) for upgrades/updates of toolkit (a Docker container) used by the SME team for processing customer PCAP files on premises.

• AGILITY-Toolbox is deployed as a single VM of 2 vCPU/16GB RAM with 100GB disk.

AGILITY-Toolbox Deployment

AGILITY-Toolbox is available to be deployed in both public and private cloud environments.

Download the VM image sent to you by email. The URL will have a validity period.

The instructions on how to use pre-authenticated URLs are at https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/usingpreauthenticatedrequests.htm.

Public Clouds:

AWS: Use the specified AMI (Amazon Machine Image) provided by B-Yond.

Azure: Use the specified Azure VM image provided by B-Yond.

Google Cloud: Use the specified Google Cloud VM image provided by B-Yond.

On-Premises Virtualization Platforms:

OpenStack: Download the provided qcow2 image specifically configured for OpenStack.

VMware: Download the provided VMware disk image specifically configured for VMware virtualization environments.

If you are using other cloud providers or virtualization solutions, you may need to convert the qcow2 or VMware disk images to the format required by your platform. Consult the documentation of your specific provider or platform for instructions on image conversion.

Using the B-Yond provided images is recommended as they are pre-configured and optimized for running AGILITY-Toolbox.

Provision the VM

Note: These values can be adjusted based on the number of simultaneous SMEs accessing the AGILITY-Toolbox.

VMware ESXi

1. To import a virtual machine stored on a VMware Hosted product to an ESX/ESXi host, run:

   vmkfstools -i virtual_machine.vmdk /vmfs/volumes/datastore/my_virtual_machine_folder/virtual_machine.vmdk

2. Create the VM using the imported disk. Option Guest OS: Other Linux (64-bit)

3. Using the console login as root , password almalinux

4. Set up static network configuration, e.g. using nmtui.

5. Increase VM disk size:

   • Increase disk size from ESXi

   • Rescan using echo 1>/sys/class/block/sda/device/rescan

   • Recreate the partition 2 with fdisk

     printf "d\n\nn\n\n\n\np\nw\n" | fdisk /dev/sda

   • Increase the filesystem size using xfs_growfs /dev/sda2

6. Configure ssh options, e.g. set authorized kes for default cloud-user almalinux or another user.

For ESXi 8.0, use Guest OS: Other Linux (64-bit) enable the LSI Logic parallel SCSI controller option.

OpenStack

1. (As an administrator) Create an image:

   glance image-create --disk-format qcow2 --container-format bare --file ./Agility-Toolbox-X.Y.Z-AlmaLinux-X-YYYYMMDD.x86_64.qcow2 --min-disk 25 --min-ram 2048 --name Agility-Toolbox-X.Y.Z

2. (As an administrator) Create a member for the glance image:

   glance member-create <image-id> <member-id>

3. (As an administrator) Accept the membership for the glance image:

   glance member-update <image-id> <member-id> accepted

4. (As a user) Create a VM using the image:

   openstack server create --flavor <your-flavor> --image <image-id> agility-toolbox --nic net-id=<network-id> --security-group <your-security-group> --key-name <your-key>

Public Clouds (AWS, Azure, GCP, etc.)

Follow the procedures specified by your cloud provider. These procedures typically include the following steps:

• Image selection: Choose the AGILITY-Toolbox VM image obtained from B-Yond or the converted image.

• Shape specification: Specify the number of virtual CPUs (vCPUs) and RAM for the instance.

• Boot disk specification: Define the size and type of the boot disk.

• Networking configuration: Configure the network settings for the VM.

• Public SSH key(s): Provide the SSH key(s) that will be used to access the VM.

[!NOTE] The VM boot time might take between 5 and 10 minutes in total.

Next Steps

1. SSH into it using the cloud-user and the associated private key:

   • AMI: ec2-user

   • OCI: opc

   • Generic Cloud: almalinux

2. Verify that all components are up and running:

3. Add ssh public keys for B-Yond SME team members by updating /home/agilityuser/.ssh/authorized_keys

AGILITY Backup

AGILITY Backup is a back-up service used to back-up AGILITY data and configuration in a location outside the fault domain of AGILITY. In the instance of a catastrophic problem, AGILITY can be restored from its backup.

Options to implement the back-up service:

• Using your own S3-compliant object storage service

• Using the nearest suitable public cloud S3-compliant object storage (if permitted)

• Using a dedicated VM, called “AGILITY-Backup”, where B-Yond’s own S3-compliant object storage service runs. It is deployed in a separate fault domain by the customer.

AGILITY must have https access to AGILITY-Backup to read/write backup.

AGILITY-Backup must have HTTPS access to B-Yond’s artifact registry (hosted on Oracle US-East cloud) for upgrades/updates of object storage server (a Docker container). AGILITY-Backup VM will require 2 vCPU/16GB RAM with 300GB disk.

AGILITY-Backup Deployment

AGILITY-Backup is available to be deployed in private cloud environments. Since all public clouds offer an S3-compliant object storage service, using such a VM on a public cloud is not a viable option.

Use the pre-authenticated URL to download the VM image from B-Yond object storage (in Oracle Cloud): https://objectstorage.us-ashburn-1.oraclecloud.com/p/XXXX

Each URL will have a validity period, so make sure the download the image as instructed by B-Yond.

The instructions on how to use pre-authenticated URLs are at Object Storage Pre-Authenticated Requests

On-Premises Virtualization Platforms:

OpenStack: Download the provided qcow2 image specifically configured for OpenStack.

VMware: Download the provided VMware disk image specifically configured for VMware virtualization environments.

If you are using other virtualization solutions, you may need to convert the qcow2 or VMware disk images to the format required by your platform. Consult the documentation of your specific provider or platform for instructions on image conversion.

Using the B-Yond provided images is recommended as they are pre-configured and optimized for running AGILITY-Backup.

Provision the VM

Note: Disk size for AGILITY-Backup can be adjusted based on the amount of analysis and retention period for AGILITY. B-Yond will provide the guidance for the proper disk size.

VMware ESXi

1. To import a virtual machine stored on a VMware Hosted product to an ESX/ESXi host, run:

   vmkfstools -i virtual_machine.vmdk /vmfs/volumes/datastore/my_virtual_machine_folder/virtual_machine.vmdk

2. Create the VM using the imported disk. Option Guest OS: Other Linux (64-bit)

3. Using the console login as root , password almalinux

4. Set up static network configuration, e.g. using nmtui.

5. Increase VM disk size:

   • Increase disk size from ESXi

   • Rescan using echo 1>/sys/class/block/sda/device/rescan

   • Recreate the partition 2 with fdisk

     printf "d\n\nn\n\n\n\np\nw\n" | fdisk /dev/sda

   • Increase the filesystem size using xfs_growfs /dev/sda2

6. Configure ssh options, e.g. set authorized kes for default cloud-user almalinux or another user.

For ESXi 8.0, use Guest OS: Other Linux (64-bit) enable the LSI Logic parallel SCSI controller option.

OpenStack

1. (As an administrator) Create an image:

   glance image-create --disk-format qcow2 --container-format bare --file ./Agility-Backup-X.Y.Z-AlmaLinux-X-YYYYMMDD.x86_64.qcow2 --min-disk 25 --min-ram 2048 --name Agility-Backup-X.Y.Z

2. (As an administrator) Create a member for the glance image:

   glance member-create <image-id> <member-id>

3. (As an administrator) Accept the membership for the glance image:

   glance member-update <image-id> <member-id> accepted

4. (As a user) Create a VM using the image:

   openstack server create --flavor <your-flavor> --image <image-id> agility-backup --nic net-id=<network-id> --security-group <your-security-group> --key-name <your-key>

Next Steps

1. SSH into it using the cloud-user and the associated private key:

   • AMI: ec2-user

   • OCI: opc

   • Generic Cloud: almalinux

2. Verify that all components are up and running:

3. Collect the minio credentials to be configured in AGILITY back-up configuration as follows:

   agility-backup

4. Follow the instructions listed in https://b-yond-infinite-network.github.io/agility-docs/ to configure back-up destination and schedule for AGILITY.

[Optional] Attach an External Disk

In cases where external disk attachment is necessary, follow these steps. This will depend on the type of external disk used.

All example commands are executed as root user.

Prepare the VM

1. Access the VM using ssh

2. Stop the processes:

   sudo su - /usr/local/bin/k3s-killall.sh

3. Rename the original directory

   export MOUNT_PATH=/var/lib/rancher/k3s/storage mv ${MOUNT_PATH} ${MOUNT_PATH}.old

NFS example

NFS server provides an endpoint to mount locally a remote directory.

1. Create a directory on your VM to serve as the mount point for the NFS share:

   mkdir -p ${MOUNT_PATH} chmod 0700 ${MOUNT_PATH}

2. Back-up the /etc/fstab file

   cp /etc/fstab /etc/fstab.backup

3. Add an entry at the end of the /etc/fstab file to specify the NFS share and the mount point. The entry should follow this format:

   echo "<NFS_server_IP_or_hostname>:<remote_directory> <local_mount_point> nfs _netdev,defaults 0 0" | tee -a /etc/fstab

   1. Replace with the IP address or hostname of the NFS server, with the path of the directory you want to mount, and with the path of the local mount point you created in Step 1.

   2. For example, if the NFS server IP address is 192.168.1.100 and the remote directory you want to mount is /data, the entry would look like this:

      echo "192.168.1.100:/data /var/lib/rancher/k3s/storage nfs _netdev,defaults 0 0" | tee -a /etc/fstab

4. Apply a daemon-reload

   systemctl daemon-reload

5. To mount all entries listed in /etc/fstab, you can use the mount -a command.

Ensure that your VM has network connectivity to the NFS server and that you have the necessary permissions to access the NFS share.

Now that you have mounted the NFS component, go to the Restart back-up process section.

Block volume example

If your cloud gives you the ability to provision block storage and attach the disk to your VM, please follow the recommended procedures. For example, it involves several iscsi commands executions.

1. Once Attached, format the disk (e.g., sdb):

   export DEV_PATH=sdb mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/${DEV_PATH} mkdir -p ${MOUNT_PATH} mount -o discard,defaults /dev/${DEV_PATH} ${MOUNT_PATH} chmod 0700 ${MOUNT_PATH}

2. Persist the changes:

   cp /etc/fstab /etc/fstab.backup UUID=$(sudo blkid -s UUID -o value /dev/${DEV_PATH}) echo $UUID echo UUID=$(sudo blkid -s UUID -o value /dev/${DEV_PATH}) ${MOUNT_PATH} ext4 _netdev,nofail 0 2 | tee -a /etc/fstab

Now that you have mounted the Disk component, go to the Restart back-up process section.

Restart back-up process

1. Synchronize the new volume

   rsync -avl ${MOUNT_PATH}.old/ ${MOUNT_PATH}/

2. Start the process:

   systemctl start k3s

3. Wait a few seconds and ensure that all services are in the Running state:

   kubectl -n agility-backup get pod NAME READY STATUS RESTARTS AGE agility-minio-provisioning-cgtdm 0/1 Completed 0 34d agility-minio-76c4c9d4ff-2252q 1/1 Running 2 (61s ago) 34d

4. Verify that the system is functioning correctly by performing following tasks.

   kubectl -n agility-backup exec -it deploy/agility-minio -- mc ls local/agility-backup kubectl -n agility-backup exec -it deploy/agility-minio -- mc du local/agility-backup

5. Remove the old directory

   rm -rf ${MOUNT_PATH}.old

Troubleshooting

When listing pods, agility-minio-xxxxxxx-xxxxx pod is not on Running status

1. Ensure synchronization between ${MOUNT_PATH}.old/ and ${MOUNT_PATH}/ was successfull

2. Ensure PVCs directory have right permissions

   chmod 0777 /var/lib/rancher/k3s/storage/pvc-* kubectl -n agility-backup rollout restart deployment agility-minio