Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

AGILITY provides ingestion automation that allows you to configure a data source (an SFTP or S3-compatible server) that will be automatically polled by AGILITY. Any new processible file placed in the configured bucket will trigger an analysis which can be viewed on the AGILITY UI.

AGILITY now supports multiplexing multiple models per pipeline, which allows running different models based on the input service or folder context. This feature provides flexibility in how data is processed, offering a range of analysis modes, from service-specific  to a more general brute-force approach.
Pipeline Service Multiplexing Logic:

  1. Service as Input Parameter:

  • If the user provides a service as an input parameter, the pipeline will run the analysis using that specific service model.

  1. No Service as Input Parameter:

  • If no service is provided, the pipeline will check the folder name where the input data is stored.

  1. a. Folder Name = Service Key:

  • If the folder name matches a service_key, the pipeline will run the analysis using the model for that specific service.

  1. b. Folder Name != Service Key:

  • If the folder name does not match any service_key, the pipeline will fall back to a brute force mode, running a generic model that attempts to analyze the data without any service-specific configuration.

Constraints:

  • Service Key Uniqueness: Each folder used in this pipeline configuration must have a unique service_key associated with a model.

  • Brute Force Mode: When no service is identified, the brute-force mode attempts to handle the data, but may produce less optimal results compared to using a service-specific model.

  • Performance Considerations: Running multiple models across services may increase pipeline runtime. Ensure appropriate resource allocation and model selection to avoid performance bottlenecks.

Create a pipeline

  1. From the AGILITY homepage, select Settings > Application Settings.

    application-settings.png
  2. Select Edit on the Autoloader Configuration card.

    autoloader-add.png
  3. Select New Pipeline.

    autoloader-add.png
  4. Enter a Name for your new pipeline.

  5. (Optional) You may also add a description and select the service you wish to analyze. If you choose auto-detection, all services AGILITY is able to detect will be analyzed.

    autoloader-name-type.png

  6. Select your pipeline type (S3-compatible or SFTP) from the Type drop-down list.

  7. Fill in the required fields. Refer to the table below for a description of the fields depending on your choice.

  8. Select Save & Close.

...