AGILITY provides ingestion automation that allows you to configure a data source (an SFTP or S3-compatible server) that will be automatically polled by AGILITY. Any new processible file placed in the configured bucket will trigger an analysis which can be viewed on the AGILITY UI.

AGILITY now supports multiplexing multiple models per pipeline, which allows running different models based on the input service or folder context. This feature provides flexibility in how data is processed, offering a range of analysis modes, from service-specific to a more general brute-force approach.
Pipeline Service Multiplexing Logic:

Service as Input Parameter:

If the user provides a service as an input parameter, the pipeline will run the analysis using that specific service model.

No Service as Input Parameter:

If no service is provided, the pipeline will check the folder name where the input data is stored.

a. Folder Name = Service Key:

If the folder name matches a service_key, the pipeline will run the analysis using the model for that specific service.

b. Folder Name != Service Key:

If the folder name does not match any service_key, the pipeline will fall back to a brute force mode, running a generic model that attempts to analyze the data without any service-specific configuration.

Constraints:

Service Key Uniqueness: Each folder used in this pipeline configuration must have a unique service_key associated with a model.
Brute Force Mode: When no service is identified, the brute-force mode attempts to handle the data, but may produce less optimal results compared to using a service-specific model.
Performance Considerations: Running multiple models across services may increase pipeline runtime. Ensure appropriate resource allocation and model selection to avoid performance bottlenecks.

Create a pipeline

From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select New Pipeline.
Enter a Name for your new pipeline.
(Optional) You may also add a description and select the service you wish to analyze. If you choose auto-detection, all services AGILITY is able to detect will be analyzed.
Select your pipeline type (S3-compatible or SFTP) from the Type drop-down list.
Fill in the required fields. Refer to the table below for a description of the fields depending on your choice.
Select Save & Close.

Field	Description

Field	Description
SFTP	Use a Secure File Transfer Protocol (SFTP) pipeline for processing files from a server.
Host	The hostname of the SFTP server.
Port	The port of the SFTP server.
Username	The username of the SFTP server.
Password	The password of the SFTP server. You may enter the password OR the SSH key.
SSH Key	The SSH key for the SFTP server. You may enter the password OR the SSH key.
Path	The path to the files on the SFTP server.
Pattern	The naming pattern of the files on the SFTP server using Ant-style path patterns. You can include: Question mark (?) to match a single character Asterisk () to match zero or more characters Double asterisks () to match zero or more directories Examples: `com/t?st.pcap` — matches `com/test.pcap` but also `com/tast.pcap` or `com/txst.pcap` `com/.zip` — matches all `.zip` files in the `com` directory `com//test.pcap` — matches all `test.pcap` files under the `com` path `org/myteam//.pcapng` — matches all `.pcapng` files under the `org/myteam` path `org/*/test/bla.pcap` — matches `org/anydir/yourdir/bla.pcap` but also `org/mydir/testing/test/bla.pcap` and `org/test/bla.pcap` `com/{filename:\\w+}.pcapng` will match `com/test.pcapng` and assign the value `test` to the file name variable
File Max Added Date	Only consider files newer than this date (format: yyyy-mm-dd).
S3 Compatible	Use S3-compatible pipeline for processing files from a storage bucket.
Bucket Name	The name of the S3-compatible bucket.
Common Prefix	The prefix folder (example: US/East/MD/)
Pattern	The naming pattern of the file names in the S3 bucket using Ant-style path patterns. You can include: Question mark (?) to match a single character Asterisk () to match zero or more characters Double asterisks () to match zero or more directories Examples: `com/t?st.pcap` — matches `com/test.pcap` but also `com/tast.pcap` or `com/txst.pcap` `com/.zip` — matches all `.zip` files in the `com` directory `com//test.pcap` — matches all `test.pcap` files under the `com` path `org/myteam//.pcapng` — matches all `.pcapng` files under the `org/myteam` path `org/*/test/bla.pcap` — matches `org/anydir/yourdir/bla.pcap` but also `org/mydir/testing/test/bla.pcap` and `org/test/bla.pcap` `com/{filename:\\w+}.pcapng` will match `com/test.pcapng` and assign the value `test` to the file name variable
S3 URL	The URL of the S3-compatible server (example: https://s3.amazonaws.com/ )
Access Key	The access key for the S3-compatible bucket.
Secret Key	The secret key for the S3-compatible bucket.
File Max Added Date	Only consider files newer than this date (format: yyyy-mm-dd).

Start a pipeline

Once your pipeline is created, you can start it.

From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select the Start icon for the pipeline you want to start under Actions.
If properly configured, the pipeline starts and the Status will change to started.

Edit a pipeline

From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
If the pipeline you want to edit is running, select the Stop icon for the pipeline under Actions.
Once the pipeline is stopped, select the Edit icon (pencil) under Actions, and edit your pipeline settings.
Select Save & Close.

Stop a pipeline

When you stop a pipeline, it pauses and does not get files for analysis until you start it again.

From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select the Stop icon for the pipeline you want to stop under Actions.
Once the pipeline is stopped, the Status changes to stopped.

Reset the pipeline processing state

When you reset a pipeline, AGILITY will process all available files in the configured storage location regardless of whether they have already been processed.

Use reset if you wish to reprocess files that are in your configured storage location (SFTP server or S3 bucket).

Reset does NOT reset your pipeline parameters.

All files available in your configured storage location will be reprocessed when you start your pipeline again.

From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select the Stop icon for the pipeline you want to stop under Actions.
Once the pipeline is stopped, the Status changes to stopped.
Select the Reset icon under Actions.
A message displays telling you that your reset was successful.
To start your pipeline again, select the Start icon.

Delete a pipeline

Deleting a pipeline removes it completely.

From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select the Stop icon for the pipeline you want to stop under Actions.
Once the pipeline is stopped, the Status changes to stopped.
Select the Delete icon under Actions.
Confirm by selecting Delete under Actions.
Your pipeline is removed from the Pipelines list and a message displays confirming that your pipeline was deleted.

AGILITY Documentation

Manage ingestion pipeline (autoloader)