Document Processing Subflow Settings

Document Processing Subflow Settings

This article describes the settings available in the Document Processing Subflow included in v42. The settings available in custom flows may differ from those described here, depending on which blocks are included in those flows. To learn more about the settings for individual blocks, see Flow Blocks.

As part of our efforts to give you more precise control over your Hyperscience processes, we’ve made many of our settings configurable on the flow level.

While you can build custom flows, each instance of Hyperscience includes a Document Processing flow. To learn more about the version of this flow that comes with v42, see Document Processing Flow in V42.

The Document Processing flow contains several subflows, including the Document Processing Subflow. This article focuses on the settings available in that subflow.

View the subflow’s settings

To view the settings of the Document Processing Subflow:

  1. Click Flows in the left-hand sidebar, and click on the name of the Document Processing flow that contains the Document Processing Subflow whose settings you would like to view.

  2. Click Edit Flows.

  3. On the Flow Studio canvas, click the Start Document Processing Subflow Block.

  4. Click the Settings Type drop-down list, and click on a setting type.

Edit the subflow’s settings

After you’ve viewed the subflow’s settings, you can make any necessary changes, and then click Save in the upper-right corner of the page. You can save changes to multiple settings types at once.

Available settings

The sections below describe the settings available for each setting type.

File Filter

Setting Description Default Value
All Files or Images Only Determines whether the filters in this block are applied to all files in submissions ( Apply to all files) or only to image files (i.e., files whose MIME type is image) ( Apply to images only). Apply to all files
Minimum Image Width (px) The minimum width in pixels that an image needs to have in order to be allowed by the filter.
This filter applies only to images (i.e., files whose MIME type is image) and has no impact on other files.
(Blank)
Minimum Image Height (px) The minimum height in pixels that an image needs to have in order to be allowed by the filter.
This filter applies only to images (i.e., files whose MIME type is image) and has no impact on other files.
(Blank)
Minimum File Size (KB) The minimum size in kilobytes that a file needs to have in order to be allowed by the filter. (Blank)
File Extension Action Select one of the following options:
- Do not filter files by extension

- Allow only these file extensions

- Deny files with these extensions
Do not filter files by extension
File Extensions A list of file extensions that the filter will allow or deny, based on the option selected in File Extension Action. Select the checkboxes for the file extensions that you would like to filter by.
If zip is selected as a file extension, the filter will not decompress ZIP files included in submissions. Each ZIP file will be treated as an individual file, regardless of the numbers of types of files compressed within it.
If there are file extensions that you want to filter by that do not appear in the drop-down list, select other, and enter the extensions in Other File Extensions.
This field only appears if Allow only these file extensions or Deny files with these extensions is selected in File Extension Action.
(Does not appear)
Other File Extensions A comma-separated list of file extensions that do not appear in File Extensions.
This field only appears if other is selected in File Extensions.
(Does not appear)

Submission Bootstrap

AWS S3

S3 Submission Retrieval Store Configuration

If you are using an S3 bucket as your submission retrieval store and you are not authenticating through IAM roles, provide your AWS access key ID and secret access key in the S3 Submission Retrieval Store Configuration field.

To enter your credentials:

  1. Click Edit value.

  2. Enter your credentials in JSON format:

{
"aws_access_key_id": "<your_access_key_id>",
"aws_secret_access_key": "<your_secret_key>"
}

You can authenticate requests using AWS Signature Version 2 (SigV2). To use AWS Signature Version 2, add the following variable and value to the S3 Submission Retrieval Store field:

"s3_signature_version":"s3"
  1. Click Done.

  2. Click Save in the upper-right corner of the page.

  3. In the dialog box that appears, click Save & Deploy.

For more information about AWS access key IDs and secret access keys, see Amazon's Understanding and getting your AWS credentials.

S3 Submission Retrieval Endpoint URL

If your submission retrieval store is not in the public cloud (i.e., its URL does not point to s3.amazonaws.com), enter its URL in S3 Submission Retrieval Endpoint URL. You do not need to edit your “.env” file to update this URL.

To edit the endpoint URL for your S3 submission retrieval store:

  1. Enter the URL in the S3 Submission Retrieval Endpoint URL field or edit its contents.

  2. Click Save in the upper-right corner of the page.

  3. In the dialog box that appears, click Save & Deploy.

If the bucket you’re using as your submission retrieval store is in a public cloud (as opposed to a government cloud or an S3-compatible internal setup), leave this field blank.

OCS

If you are using an OCS submission retrieval store, enter the configuration details for your file store in these fields.

When you are finished entering or editing these field’s values, click Save in the upper-right corner of the page. Then, in the dialog box that appears, click Save & Deploy.

OCS Configuration

To enter your configuration details:

  1. Click Edit value.

  2. Enter the configuration details in JSON format:

{
"host_url": "<your_host_url>",
"username": "<your_username>",
"password": "<your_password>",
"ssl_cert": "<CA_bundle_filename_OR_SKIP>"
}

The value of ssl_cert should match the CA bundle filename inside the $HS_PATH/certs directory. To disable certificate validation, set this value to SKIP.

  1. Click Done.

  2. Click Save in the upper-right corner of the page.

  3. In the dialog box that appears, click Save & Deploy.

v42.3

Name Required? Description
OCS Host URL Yes, if using an OCS submission retrieval store The OCS host URL for the submission retrieval store.
OCS Username Yes, if using an OCS submission retrieval store The OCS username for authenticating into the submission retrieval store.
OCS Password Yes, if using an OCS submission retrieval store The OCS password for authenticating into the submission retrieval store.
OCS SSL Certificate Yes, if using an OCS submission retrieval store The CA bundle filename inside the $HS_PATH/certs directory. To disable certificate validation, set this value to SKIP.

Generic Web Storage (HTTP/HTTPS)

Generic Web Storage (HTTP/HTTPS) Configuration

If you are using a generic web storage submission file store, enter the configuration details for your file store in this field.

To enter your configuration details:

  1. Click Edit value.

  2. Enter the configuration details in JSON format:

{
"username": "<your_username>",
"password": "<your_password>",
"ssl_cert": "<CA_bundle_filename_OR_SKIP>"
}

The value of ssl_cert should match the CA bundle filename inside the $HS_PATH/certs directory. To disable certificate validation, set this value to SKIP.

  1. Click Done.

  2. Click Save in the upper-right corner of the page.

  3. In the dialog box that appears, click Save & Deploy.

Azure Blob Storage

If you are using Azure Blob Storage as your submission retrieval store, you can use the fields described below to configure the system’s connection to the blob.

Azure Blob Storage Authentication Type

From the Azure Blob Storage Authentication Type drop-down list, select the authentication type the system should use to access the blob:

When you select an authentication type, additional settings appear.

Settings for SAS Token Only authentication

Name Required? Description
Azure Blob Storage Account URL Yes The URL of the storage account (e.g., https://.blob.core.windows.net)

Settings for Service Principal authentication

Name Required? Description
Azure Blob Storage Account URL Yes The URL of the storage account (e.g., https://.blob.core.windows.net)
Azure Blob Storage Tenant ID No The tenant ID of the service principal
Azure Blob Storage Client ID No The client ID of the service principal.
If multiple client IDs exist for the service principle, and Azure Blob Storage Client ID is left blank, the default client ID will be used.
Azure Blob Storage Client Secret No The client secret for the service principal
Azure Blob Storage Authority Host No The host of the Microsoft Entra authority for the storage account.
If omitted, the host of the Azure Public Cloud authority ( login.microsoftonline.com) is used.

Settings for Managed Identity authentication

Name Required? Description
Azure Blob Storage Account URL Yes The URL of the storage account (e.g., https://.blob.core.windows.net)
Azure Blob Storage Client ID No The client ID of the managed identity.
If multiple client IDs exist for the managed identity, and Azure Blob Storage Client ID is left blank, the default client ID will be used.

Settings for Account Key authentication

Name Required? Description
Azure Blob Storage Account URL Yes The URL of the storage account (e.g., https://.blob.core.windows.net)
Azure Blob Storage Account Key No The access key for the storage account
Azure Blob Storage Account Name No The name of the storage account

If incorrect authentication information is entered, the flow runs for the attempted file-ingestion attempts will fail. The flow runs’ output will contain error messages passed to the system by Azure.

For more information about troubleshooting flow runs, see Testing and Debugging Flows.

GCS Storage

If you are using GCS Storage as your submission retrieval store, you can use the fields described below to configure the system’s connection to the blob.

Name Required? Description
Use Workload Identity Must be selected if no value for GCS Service Account JSON is provided Credentials obtained by using Workload Identity Federation, which applies to Hyperscience installations inside GKE clusters.
GCS Service Account JSON Yes, if Use Workload Identity is deselected The service account JSON credential that allows access to the retrieval-store bucket.
To enter the JSON:
1. Click Edit value.

2. Enter your Service Account credentials in valid JSON format.

3. Click Done.

4. Click Save in the upper-right corner of the page.

5. In the dialog box that appears, click Save & Deploy.

Other settings

Setting Description Default Value
Enable File Page-Limit Check Allows you to specify the maximum number of pages that submissions’ files can have.
Enabling this option reveals the Maximum Pages Allowed Per File setting, where you can specify the maximum number of pages each file can have.
Disabled
Maximum Pages Allowed Per File The maximum number of pages each file in a submission can have. If a file has more pages than this maximum, the submission will fail.
This setting is only available if Enable File Page-Limit Check is enabled.
(None)

Classification

Setting Description Default Value
Structured Layout Match Threshold The minimum confidence score a page must have in order for it to be matched to a layout. If the page's confidence score is below this value, the system sends it to Classification Supervision (if enabled) or marks it as "No Layout Found." 0.6
Semi-structured Classification Enables the management of a model that automatically classifies Semi-structured and Additional documents. Enabled
Manual Classification Supervision Enables Classification Supervision. Disabled
Semi-structured Classification Target Accuracy Your desired accuracy for the classification of Semi-structured and Additional documents. If the estimated accuracy of the model's prediction for a document is below this value, the system will send the document to Classification Supervision (if enabled) or mark it as "No Layout Found." 99
Semi-structured Classification Grouping Logic Determines how multiple pages are matched to the same layout variation in a given submission will be handled.
To learn more about this setting, see Document Classification Settings.
Consecutive pages as a document
Semi-structured QA Sample Rate The percentage of documents that the system will randomly select for Classification QA. 5
Validate Classification Using Layout ID Enabling this setting allows Structured documents to be matched using a layout identifier. When this setting is enabled, the system checks for a matching layout identifier in the document. If the identifier matches the expected one in the layout variation, the document is classified accordingly. If it doesn't match, the document is either sent for further review or to Document Drift Management, preventing misclassification. Disabled
Bypass Validation if Layout ID is Missing This setting should be enabled when certain layouts do not contain a layout identifier. It bypasses validation by layout identifier if the matched layout variation doesn’t have an identifier specified. In these cases, the bypass allows the system to continue classifying documents even without layout identifiers, ensuring that documents are still processed but not necessarily tied to a specific layout variation. Disabled

Identification

Setting Description Default Value
Identification Target Accuracy (Entry Level) Allows you to set flow-level Identification Target Accuracy values for fields and table columns (“entries”) included in the flow’s release.
To enter target accuracies, click the pencil icon below Identification Target Accuracy (Entry Level).
Then, click the Target Accuracy cell for a field or table column to enter a target accuracy for it. Click outside of the cell to save the value you entered.
The value set in Field Identification Target Accuracy or Table Identification Target Accuracy
Field Identification Target Accuracy Your desired accuracy for the identification of fields. If the estimated accuracy of the model's prediction for a field is below this value, the system will send the field and all its occurrences (if any) to Field ID Supervision. 95
Table Identification Target Accuracy Your desired accuracy for the identification of tables. If the estimated accuracy of the model's prediction for a table is below this value, the system will send the table to Table ID Supervision. 96
Manual Identification Supervision Enables Field ID Supervision and Table ID Supervision. Enabled
Field Identification Quality Assurance Enables Field ID Quality Assurance. If disabled, the system won't have data to retrain existing Semi-structured models, and Field Identification on new Semi-structured layouts cannot be automated. Enabled
Field Identification QA Sample Rate The percentage of documents that the system will randomly select for Field ID QA.
This setting is only available if Field Identification Quality Assurance is enabled.
5
Table Identification Quality Assurance Enables Table ID Quality Assurance. If disabled, the system won't have data to retrain existing Semi-structured models with tables, and Table Identification on new Semi-structured layouts cannot be automated. Enabled
Table Identification QA Sample Rate The percentage of documents that the system will randomly select for Table ID QA.
This setting is only available if Table Identification Quality Assurance is enabled.
5
Manual Identification Notification Flow Determines which Notification subflow is used to send Manual Identification updates to downstream systems. Submission State Notifications
Default Task Restrictions Determines which users can access Supervision Tasks created by the Manual Identification Block. None

General Transcription

Setting Description Default Value
Customize Field Transcription Allows you to:
- set flow-level Transcription Target Accuracy values for fields included in the flow’s release, and

- view any Transcription Target Accuracy values set in the Field Dictionary for those fields  

If you enter a target accuracy for a field that already has a target accuracy set in the Field Dictionary, the value you enter here will override the one in the Field Dictionary.
The value set in Structured Text Target Accuracy, or the Transcription Target Accuracy value set in the Field Dictionary (if any)
Customize Transcription Target Accuracy For Structured documents, allows you to:
- set flow-level Transcription Target Accuracy values for fields included in the flow’s release, and

- view any Transcription Target Accuracy values set in the Field Dictionary for those fields  

For Semi-structured documents, allows you to:
- set flow-level Transcription Target Accuracy values for specific fields or table columns (a.k.a. “entries”) included in the flow’s release.

If you enter a target accuracy for a field in a Structured layout that already has a target accuracy set in the Field Dictionary, the value you enter here will override the one in the Field Dictionary.
- For fields in Structured documents:

- The value set in Structured Text Target Accuracy, or the Transcription Target Accuracy value set in the Field Dictionary (if any)
- For fields or table columns in Semi-structured documents:

- The value set in Semi-structured Text Target Accuracy
Manual Transcription Supervision Enables Transcription Supervision. Enabled
Transcription Quality Assurance Enables Transcription Quality Assurance. If disabled, the system won't have the data needed to determine the accuracy of transcriptions. Enabled
Automatic QA Sample Rate If enabled, based on the QA records you have, the system automatically calculates QA sample rates for your:
- Structured text,

- Structured checkbox,

- Structured signature, and

- Semi-structured fields.

When enabled, this setting overrides the following flow settings:
- Structured Text Transcription QA Sample Rate

- Structured Checkbox Transcription QA Sample Rate

- Structured Signature Transcription QA Sample Rate

- Semi-structured Transcription QA Sample Rate
Disabled
Structured Text Transcription QA Sample Rate The percentage of text fields in Structured documents that the system randomly samples for Transcription QA. 5
Structured Checkbox Transcription QA Sample Rate The percentage of checkbox fields in Structured documents that the system randomly samples for Transcription QA. 5
Structured Signature Transcription QA Sample Rate The percentage of signature fields in Structured documents that the system randomly samples for Transcription QA. 5
Semi-structured Transcription QA Sample Rate The percentage of fields in Semi-structured documents that the system randomly samples for Transcription QA. 5
Table Transcription Quality Assurance Enables Table Transcription Quality Assurance. If disabled, the system won't have the data needed to determine the accuracy of table cell transcriptions. Disabled
Table Transcription QA Sample Rate The percentage of table cells that the system randomly samples for Transcription QA. 5
Finetuning Only For Trained Layouts If enabled, finetuning (Transcription Automation) only uses the layouts its model was trained on.
​​If you add a new layout, the layout does not use finetuning and defaults to manually entered thresholds until a new finetuning model is trained with that layout.
Enabled
Force Normalization Errors To Supervision This setting allows you to send fields with normalization errors to Supervision. When enabled, the normalization errors are flagged for human review, ensuring data accuracy within the platform. Disabled
Force Missing/Blank Fields To Supervision Fields marked as Required in the Layout Editor can be sent to Supervision when missing or left blank. When this setting is enabled, blank fields are sent to Transcription Supervision, while missing fields are sent to Identification Supervision. Disabled

Structured Document Transcription

Setting Description Default Value
Transcription Automation Training This feature enables the system to use QA data to calculate the optimal mix of data keyer and machine transcriptions to reach a specified target accuracy with the minimum amount of data keyer effort.
To see a graph of current projected automations, click See Projections. You will be redirected to the Transcription section of Administration > System Settings.
Disabled
Period of Records to Use This setting is only available if Transcription Automation Training is enabled. It determines how far in the past to draw training data from. 100 days
Improved Threshold Accuracy This setting helps to ensure that your target accuracy is being met on all processed fields. Enabled
Structured Text Target Accuracy Your desired accuracy for the transcription of fields in Structured documents.
If Transcription Automation Training for Structured documents is enabled, the system uses this value to calculate the Structured Text Automation once the minimum amount of training data is obtained through Transcription QA.
95
Structured Text Automation This setting shows the level of automation you can expect when the system is working to reach the target accuracy set in Structured Text Target Accuracy.
The system automatically calculates this value after the minimum amount of training data is obtained through Transcription QA.
N/A
Structured Text Threshold This setting determines the minimum confidence thresholds needed for a field to be automatically processed.
Fields with confidence scores above this threshold are automatically processed.
.5
Structured Text Minimum Legibility Threshold The minimum confidence score a text field must have in a Structured document in order for the system to automatically process the field. If a field's confidence score is below this value, the system will mark the field as illegible. 0.1
Structured Checkbox Target Accuracy Your desired accuracy for the transcription of checkboxes in Structured documents.
If Transcription Automation Training for Structured documents is enabled, the system uses this value to calculate the Structured Checkbox Automation once the minimum amount of training data is obtained through Transcription QA.
95
Structured Checkbox Automation This setting shows the level of automation you can expect when the system is working to reach the target accuracy set in Structured Checkbox Accuracy.
The system automatically calculates this value after the minimum amount of training data is obtained through Transcription QA.
N/A
Structured Checkbox Threshold This setting determines the minimum confidence thresholds needed for a checkbox to be automatically processed.
Checkboxes with confidence scores above this threshold are automatically processed.
.56
Structured Checkbox Minimum Legibility Threshold The minimum confidence score a checkbox field must have in a Structured document in order for the system to automatically process the field. If a checkbox field's confidence score is below this value, the system will mark it as illegible. 0.25
Structured Signature Target Accuracy Your desired accuracy for the transcription of signatures in Structured documents.
If Transcription Automation Training for Structured documents is enabled, the system uses this value to calculate the Structured Signature Automation once the minimum amount of training data is obtained through Transcription QA.
95
Structured Signature Automation This setting shows the level of automation you can expect when the system is working to reach the target accuracy set in Structured Signature Accuracy.
The system automatically calculates this value after the minimum amount of training data is obtained through Transcription QA.
N/A
Structured Signature Threshold This setting determines the minimum confidence thresholds needed for a signature to be automatically processed.
Signatures with confidence scores above this threshold are automatically processed.
.56
Structured Signature Minimum Legibility Threshold The minimum confidence score a signature field must have in a Structured document in order for the system to automatically process the field. 0.5

Semi-structured Document Transcription