Training a Semi-structured Model

• Updated on Aug 18, 2025
• Published on May 8, 2025
• 17 minute(s) read

Hyperscience extracts data from documents and converts them into a machine-readable format. We support Structured, Semi-structured, and Additional documents. To learn how to differentiate between the document types, see Understand Document Types.

Semi-structured use cases

Layouts for Semi-structured documents help identify and extract data from pages that do not have a consistent structure or fixed visual templates. While the information you need to extract (e.g., identification number, address) remains the same, its location may vary and appear under different names or labels. Examples of Semi-structured documents are paystubs and invoices. They contain key pieces of information that are always present, but their placement can vary significantly across different versions of the document.

Use Semi-structured layouts in the following scenarios:

• When field positions vary
• When tables vary in size or structure across documents

In this article, you’ll learn how to build a robust Semi-structured model corresponding to your business needs by using our Training Data Management tools.

Step 1 - Sampling Documents

Review your documents

Having a diverse, representative training set is crucial for a high-quality Identification model. Selecting the appropriate documents for training will optimize your Semi-structured model’s performance.

• Determine the common types of documents you'll be processing, and ensure you have at least:

  • 15 documents for a Field Identification model, or
  • 20 documents for a Table Identification model.

• Become familiar with the edge cases (i.e., documents that are completely different from the main ones) and determine their variety. Exclude them if they are not suitable for your use case.

• Remove documents that would reduce model performance (e.g., documents containing unrelated information, highly distorted pages (pages that are noisy, skewed, pixelated ones, duplicates).

• Choose at least 50-100 documents for testing purposes. Note that these documents should be representative of the data you expect in production.

Review your fields and columns

• Your fields and columns should be representative of the information you want to extract.
• Ensure they are present in your documents to achieve a high-performance model.
• Review any interchangeable fields or columns, as this might result in poor model performance.

Step 2 - Build your layout and add it to a release

Build your layout

Once you’ve determined the information you want to extract, you need to build your Semi-structured layout.

1. Create your layout by following the steps in Creating Semi-structured Layouts.

• Make sure to set the proper data type for each field or column you create to obtain a high-performance model.
• Ensure your configurations are suitable for the fields and columns for extraction:
  • Check Multiple Occurrences if your fields have more than one occurrence.
  • Enable the Multiline setting if required.
  • Set Identification Supervision to Always for each field you want to guarantee a manual review for.
  • Set Transcription Supervision to Always if there are issues in the document that could prevent the machine from reading the field or the column.

Assign to a release

1. Add your layout to a release by following the steps described in Adding a New Release.
2. Follow the steps in Assigning a Release to a Flow to match your release to the flow you are using.

Step 3 - Using Training Data Management (TDM)

Use the tools in Training Data Management to control, manage, and adjust the ground truth of your training sets for Identification and Classification models. Before you start:

• Ground truth is manually annotated data used to train our machine-learning models.
• See the training requirements in Requirements for Training a New Model.
• Make sure to keep 50-100 documents for testing purposes. Note that they should be representative of the data expected in production. You’ll upload them after the model training is completed.

Upload your documents

1. Go to the Model Management page for your layout ( Library > Models).
2. Click Upload Training Documents and upload each document as its own file.
3. Click Upload in the dialog box.

All uploaded documents will appear on the Training Documents card.

Step 4 - Analyze your data

Training Data Analysis allows you to group your training documents and receive recommendations to improve the quality of your dataset.

Running Training Data Analysis

We recommend running training data analysis once you’ve uploaded your documents. The system will create groups based on the similarity of your training documents which improves the efficiency of the annotation process.

Receive insights for improving your training data by clicking the Analyze Data button, located in the Training Data Health card.

Analysis results

The results show you the eligibility and importance of each document.

• Groups - Training data analysis groups your training set by visual similarity. For best data representation, we recommend having at least 10 groups of each document type.
• Importance - The Training Data Curator labels each training document as having high or low importance.
• Eligibility - with Document Eligibility Filtering, you can see which documents are incompatible with training and why, allowing you to address any issues accordingly and achieve better model performance.
• Detect anomalies - Re-analyze your data and find inconsistencies across your annotations with Labeling Anomaly Detection.

Step 5 - Annotate your documents

Consistent annotations are crucial for a high-performance locator model.

Best practices

1. Once you analyze the data, you’ll be able to annotate by group. Doing so provides you with more control over the dataset.
2. After annotating 2-3 documents per group, you’ll be able to use guided data labeling.
3. Follow the general rule for annotating: left to right, top to bottom.
4. Make sure to maintain consistent annotations for your fields or columns.

Field Identification

1. Annotate fields with Multiple Occurrences only when multiple instances of a field are present.
2. Use multiple bounding boxes when a text is logically connected.
3. If you don’t see a value for a field (i.e., the field is blank), do NOT annotate it.

Table Identification

1. When annotating a table, make sure to select a row where all data is present.
2. Always find your table's first and last rows and ensure they are properly annotated.
3. Always press the ESC button before submitting a table to ensure the annotations are correct.

Next steps

• Check if all training documents are eligible for training.
• Ensure you have the required number of training documents.

Step 6 - Review your flow’s settings and train your model

Review the flow’s configurations:

For more precise control over the process, you can configure your flow’s settings.

• Set your Target Accuracy to achieve better performance.

Run Training

Initiate a model training by clicking the Run Training button.

Step 7 - Evaluate the training results

Deploy your model

Once the model training is complete, you’ll find the candidate model in the model details page.

Evaluate the performance Use the documents you’ve chosen for testing purposes to evaluate the performance of your model.