Training Data Curator

• Updated on Oct 15, 2025

• Published on Sep 24, 2025

• 4 minute(s) read

Having a diverse and representative Training Set is essential for building a high-quality identification model. The Hyperscience Platform helps you identify which documents are most valuable for training, allowing you to prioritize annotations more efficiently.

How data is curated

The Training Data Curator labels each training document as having high or low importance. The importance is calculated by determining which data would best contribute to the model’s performance. For each group of documents, the system labels the most impactful ones as having high importance.

Before using the Training Data Curator, make sure that you've:

• uploaded at least the Recommended Number of Documents. The default minimum value is 100.

Grouping Logic

Documents are grouped based on text and location. Note that new groups will appear every time you run Training Data Analysis. To learn more, see Step 4 of our Training a Semi-structured Model article.

Using the Training Data Curator

1. Analyze the data.

The Training Data Curator depends on the results from the document grouping. That's why you should first analyze your data.

For more information about data analysis and groups, see Labeling Anomaly Detection.

• Before analyzing the data, the importance of each document is N/A.

After data analysis is finished, the Training Data Health card shows the quality of your training set.

• Required documents— the number of additional documents required to start model training. This value is lower than the recommended number of documents, as the system needs fewer examples to initiate the training process than it does to achieve optimal model performance.
  • Ineligible/Eligible documents — number of documents excluded/included from the training set. For more information, see Document Eligibility Filtering.
  • Recommended documents— the recommended number of documents for high-quality model training. The minimum requirement is 100, but achieving a robust model typically requires a larger, more diverse, and representative training set. The importance of each document appears in the Training Data Table. Importance is partially determined by grouping or document similarity - similar documents are grouped and assigned high or low importance, helping you avoid redundant annotations.
  • High importance— the documents you should annotate first. They are representative of your data and will help you achieve better performance.
  • Low importance— the documents that are redundant in your training set.

2. Filter the documents by group

• Expand the Filter section and filter the documents by group and by importance.
  • After you’ve selected a group and importance, Apply Filters.

3. Annotate the documents with High importance first.

The more representative your training documents are of real-life documents, the better your model will perform with fewer annotations.

• We recommend double-checking the ones with Low importance.

All annotated documents will be marked as high importance during training data analysis, while similar ones will be marked as low importance. These updates help keyers annotate documents that are most valuable to the model. If you import documents that have already been annotated, they have high importance by default.

Next steps

• Review your training set for inconsistencies to determine whether you need more annotations or corrections. To learn more, see Labeling Anomaly Detection.
• As you annotate documents, their importance becomes High, and unannotated documents may be marked as Low importance. Reanalyze your data as you annotate to see the updated information.

A dataset used to teach the system how to recognize and extract information. It includes documents with labeled fields so the system can learn from real examples.

The suggested minimum and optimal amount of labeled data required to train machine learning models in Hyperscience.

• For Classification models, at least 10 pages per layout are needed, with 1 20 pages per layout recommended.
• For Identification models, the minimum is 100 documents, and the recommended amount is 400 documents.

Providing diverse, high-quality examples ensures better model accuracy and reliability.

A tool in TDM that analyzes your training data to compute the importance of each training document and identify issues such as missing labels, overlapping fields or columns, or inconsistent annotations. This analysis helps you prioritize which documents to annotate and ensures clean, accurate data before you train a model.