Training an Identification Model

Hyperscience extracts data from documents and converts them into a machine-readable format. We support Structured, Semi-structured, and Additional documents. To learn how to differentiate between the document types, see Understand Document Types.

Requirements for training an ID model

Model training is handled by the Trainer, which operates independently from the main application to prevent performance degradation during document processing. Learn about the trainer in our Trainer article.
For optimal performance, the trainer requires a dedicated machine with at least 64GB of RAM and 16 CPU cores. To learn more, see Infrastructure Requirements.
Learn more about the product’s limits in Product Limits and Guidelines.
The system's default minimum requirement to run a model training is 100 documents.
To train a new Identification model, it is generally recommended to have at least 400 training documents.

Step 1 — Sampling documents

Before uploading documents, make sure your Dataset reflects the documents you expect to process in production. Learn how to prepare your dataset in Preparing training data.

Step 2 — Build a layout, add it to a release, and assign it to a flow

After identifying the fields you want to extract, create a layout to define how those fields are captured.

Your layout determines:

which fields are extracted
how the model learns to identify them

Example

Consider the following insurance claim form:

This is a typical semi-structured document where key information is spread across sections and may appear in different locations depending on the format.

Identification models are designed to extract this type of information consistently, even when layouts vary.

Step 3 — Training Data Management (TDM)

Use the tools in Training Data Management to control, manage, and adjust the ground truth of your training sets for Identification and Classification models. In this section, you will learn how to upload your data using TDM. Before you start:

Ensure you meet the requirements outlined in Requirements for training an ID model section of this article.
Make sure to keep 50-100 documents for testing purposes.

Ground truth

Ground truth is manually annotated data used to train our machine-learning models. We use a subset of this data to assess the performance of your models.

Step 4 — Analyze your data

Training Data Analysis allows you to group your training documents and receive recommendations to improve the quality of your dataset. Learn more in Training Data Analysis

Step 5 — Annotate your documents

Learn how to annotate your documents by following the best practices listed below.

General guidelines

Once you analyze the data, you’ll be able to annotate by group.
After annotating 2-3 documents per group, you’ll be able to use guided data labeling.
Follow the general rule for annotating: left to right, top to bottom.
Make sure to maintain consistent annotations for your fields or columns.

Field Identification

Annotate fields with Multiple Occurrences only when multiple instances of a field are present.
Use multiple bounding boxes when a text is logically connected.
If you don’t see a value for a field (i.e., the field is blank), do NOT annotate it.

Table Identification

When annotating a table, select a row where all data is present as your template row.
Always find your table's first and last rows and ensure they are properly annotated.
Draw one large bounding box capturing all rows of your table.

Next steps

Check if all training documents are eligible for training.
Ensure you have the required number of training documents.

Step 6 - Review your flow and train your model

Once you’ve reviewed your annotations and addressed any potential anomalies, you’re ready to initiate model training.

Review the flow’s configurations:

Set your Target Accuracy to achieve better performance.

Run Training

Initiate a model training by clicking the Run Training button.

Step 7 — Evaluate the training results

Deploy your model

Once the model training is complete, you’ll find the candidate model in the model details page. To deploy it, click on your candidate model, then click Deploy Model.

Evaluate the performance

Use the documents you’ve chosen for testing purposes to evaluate the performance of your model.