Universal Folder Listener

Universal Folder Listener

The Universal Folder Listener allows you to ingest files from a Folder on your system.

The Universal Folder Listener Input connection is available only for on-premise / private cloud instances of Hyperscience.

Sample use cases

Block settings table

In addition to the settings outlined below, you can also configure the settings described in Universal Integration Block Settings.

| Name | Required? | Description | | Folder To Scan For Submissions | No | The folder the connector will scan for image files.
The default path is the path specified in the FS_INPUT_PATH ".env" file variable, which is shown before the text box (e.g., /mnt/hs/input/). If you would like to scan a subfolder of this folder, enter that folder's name or path in the text box.
As a best practice, we recommend having one Listener monitoring the base folder. Setting up multiple Listeners for a single base folder may result in errors and resource collisions. | | File Extensions | Yes | A list of the extensions that image files will need to have to be eligible for processing.
If there are file extensions that you want to support but do not see in the drop-down list, select other, and enter the extensions in Other File Extensions. | | Other File Extensions | No | A comma-separated list of file extensions that do not appear in File Extensions.
This field only appears if other is selected in File Extensions. | | Include Submission Level Parameters | No | Indicates whether the system will ingest JSON files along with document files and submission folders. These JSON files can contain information such as metadata, case data, and external_id values. These JSON file names should match the names of the related files or folders (e.g., XYZ.jpg.json for XYZ.jpg).
If this setting is enabled, any files that do not have metadata in the expected location will not be processed. See Check the location of metadata files for more information. | | Group Subfolder Files | No | Determines how files in a subfolder of the base folder (the folder specified in Folder to Scan for Submissions) are processed:
- If enabled, all files in a subfolder are processed together as a single submission.

- If disabled, each file in a subfolder is processed as a single submission.

Defaults to enabled. | | Poll Interval (In Seconds) | No | The frequency at which the connector will monitor the base folder for submissions.
Defaults to 10. | | Warm-Up Interval (In Seconds) | No | The length of time that a file must remain unmodified before it is eligible for processing.
Defaults to 15. | | Empty Folder Cleanup Delay (In Hours) | No | The frequency at which the connector will delete empty folders inside the base folder.
Defaults to 24. |

Naming of files by the system

After files are ingested, the system adds $ufl_ to the beginning of each file name (e.g., file1.png in the folder1 subfolder becomes $ufl_folder1/file1.png, fileA.png in the base folder becomes $ufl_fileA.png).

These names appear in the submission output as submitted_filename values.

Setting up Universal Folder Listener

To set up the Universal Folder Listener, enter the settings as described in the Block settings table above. You do not need to configure anything in the folder you are monitoring.

Migrating from the Folder Listener

There are numerous benefits for users upgrading from the legacy Folder Listener to the Universal Folder Listener. These include:

Before setting up a Universal Folder Listener connector and retiring your Folder Listener connection, you need to complete some preparatory steps.

Check your folder structure

As mentioned previously, the Universal Folder Listener accepts both single files and folders of files as submissions. Whether a file is processed as its own submission or as part of a larger submission is determined by its location and the configuration of the Listener.

Depending on how you want your files to be processed, you may need to make some changes to your base folder's structure before using the Universal Folder Listener.

Verify the path of your base folder

The system uses the file path in the FS_INPUT_PATH ".env" file variable as the default location of the base folder. While you can enter any subdirectory of that folder in the Folder To Scan For Submissions field, for greater security, you may want to modify the value of FS_INPUT_PATH if you are certain you want to use one of its subfolders as your base folder.

Update your metadata files

The Folder Listener connector supported text files that contained submission metadata. The Universal Folder Listener can accept a JSON file for each submission that contains metadata, case data, and an external_id. The name of this file depends on whether the submission consists of a single file or a folder of files:

The metadata file for a file or folder must be in the base folder. If you put it in one of the base folder's subfolders, a submission won't be created from the file or folder the metadata is for.

An example JSON file for a Universal Folder Listener submission appears below.

{
    "metadata": {
         "test": "Metadata for inner1"
    },
    "cases": [{
         "external_case_id": "900",
         "filenames": ["div_lic_1.jpg", "div_lic_1.jpg"]
    }],
    "external_id": "123"
}

By contrast, the metadata text files supported by the original Folder Listener ( filename_index.txt) consist of key/value pairs that you define:

{
    "key1": "value1",
    "Key2": "value2"
}

Therefore, when migrating to the Universal Folder Listener, you need to change the names, file types, and content of your metadata files, as outlined in the table below.

| | Folder Listener | Universal Folder Listener | | File name | filename_index.txt | filename.ext.json
OR
foldername.json | | File type | Text | JSON | | Contents | JSON-formatted key/value pairs of your choosing | - metadata (key/value pairs of your choosing)

- cases

- external_id |

Check the location of metadata files

If Include Submission Level Parameters is enabled, metadata files are required and must be in the correct location in order for submissions to be created from the files they describe.