AGILITY provides ingestion automation that allows you to configure a data source (an SFTP or S3-compatible server) that will be automatically polled by AGILITY. Any new processible file placed in the configured bucket will trigger an analysis which can be viewed on the AGILITY UI.
Create a pipeline
From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select New Pipeline.
Enter a Name for your new pipeline.
(Optional) You may also add a description and select the service you wish to analyze. If you choose auto-detection, all services AGILITY is able to detect will be analyzed.
Select your pipeline type (S3-compatible or SFTP) from the Type drop-down list.
Fill in the required fields. Refer to the table below for a description of the fields depending on your choice.
Select Save & Close.
Field | Description |
---|---|
SFTP | Use a Secure File Transfer Protocol (SFTP) pipeline for processing files from a server. |
Host | The hostname of the SFTP server. |
Port | The port of the SFTP server. |
Username | The username of the SFTP server. |
Password | The password of the SFTP server. You may enter the password OR the SSH key. |
SSH Key | The SSH key for the SFTP server. You may enter the password OR the SSH key. |
Path | The path to the files on the SFTP server. |
Pattern | The naming pattern of the files on the SFTP server using Ant-style path patterns. You can include:
Examples:
|
File Max Added Date | Only consider files newer than this date (format: yyyy-mm-dd). |
S3 Compatible | Use S3-compatible pipeline for processing files from a storage bucket. |
Bucket Name | The name of the S3-compatible bucket. |
Common Prefix | The prefix folder (example: US/East/MD/) |
Pattern | The naming pattern of the file names in the S3 bucket using Ant-style path patterns. You can include:
Examples:
|
S3 URL | The URL of the S3-compatible server (example: https://s3.amazonaws.com/ ) |
Access Key | The access key for the S3-compatible bucket. |
Secret Key | The secret key for the S3-compatible bucket. |
File Max Added Date | Only consider files newer than this date (format: yyyy-mm-dd). |
Start a pipeline
Once your pipeline is created, you can start it.
From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select the Start icon for the pipeline you want to start under Actions.
If properly configured, the pipeline starts and the Status will change to started.
Edit a pipeline
From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
If the pipeline you want to edit is running, select the Stop icon for the pipeline under Actions.
Once the pipeline is stopped, select the Edit icon (pencil) under Actions, and edit your pipeline settings.
Select Save & Close.
Stop a pipeline
When you stop a pipeline, it pauses and does not get files for analysis until you start it again.
From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select the Stop icon for the pipeline you want to stop under Actions.
Once the pipeline is stopped, the Status changes to stopped.
Reset the pipeline processing state
When you reset a pipeline, AGILITY will process all available files in the configured storage location regardless of whether they have already been processed.
Use reset if you wish to reprocess files that are in your configured storage location (SFTP server or S3 bucket).
Reset does NOT reset your pipeline parameters.
All files available in your configured storage location will be reprocessed when you start your pipeline again.
From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select the Stop icon for the pipeline you want to stop under Actions.
Once the pipeline is stopped, the Status changes to stopped.Select the Reset icon under Actions.
A message displays telling you that your reset was successful.
To start your pipeline again, select the Start icon.
Delete a pipeline
Deleting a pipeline removes it completely.
From the AGILITY homepage, select Settings > Application Settings.
Select Edit on the Autoloader Configuration card.
Select the Stop icon for the pipeline you want to stop under Actions.
Once the pipeline is stopped, the Status changes to stopped.Select the Delete icon under Actions.
Confirm by selecting Delete under Actions.
Your pipeline is removed from the Pipelines list and a message displays confirming that your pipeline was deleted.
Add Comment