Training Data Curator

• Updated on May 16, 2025

• Published on Oct 10, 2024

• 3 minute(s) read

Having a diverse, representative training set is crucial for a high-quality identification model. The Hyperscience application allows you to train a model with fewer annotations with minimal impact on performance.

How data is curated

The Training Data Curator labels each training document as having high or low importance. The importance is calculated by determining which data would best contribute to the model’s performance. For each group of documents, the system labels the most impactful documents as having high importance.

Before using the Training Data Curator, make sure that you've:

• uploaded at least the recommended number of documents. The default minimum value is 100.

Using the Training Data Curator

1. Analyze the data.

The Training Data Curator depends on the results from the document grouping. That's why you should first analyze your data.

For more information about data analysis and groups, see Labeling Anomaly Detection.

• Before analyzing the data, the importance of each document is N/A.

After data analysis is finished, the Training Data Health card shows your training set's quality.

• Required documents— number of additional documents needed to run a model training. This number is lower than the number of recommended documents, as the number of documents the system needs to run the model-training process is lower than the number of documents needed for a high-performance model.

• Ineligible/Eligible documents — number of documents excluded/included from the training set. For more information, see Document Eligibility Filtering.

• Recommended documents— number of documents we recommend using for high-quality model training. The minimum value is 100, but to ensure a robust model, provide a representative and diverse training set, which may require more than 100 documents.

The importance of each document appears in the Training Data Table. Importance is partially determined by grouping or document similarity—similar documents are grouped together and assigned high or low importance, helping you avoid redundant annotations.

• High importance— the documents you should annotate first. They are representative of your data and will help you achieve better performance.

• Low importance— the documents that are redundant in your training set.

2. Click Filters to filter the documents by group and by importance.

After you’ve selected a group and importance, Apply Filters.

.jpg?sv=2026-02-06&spr=https&st=2026-07-27T09%3A02%3A23Z&se=2026-07-27T09%3A15%3A23Z&sr=c&sp=r&sig=BhMv2DDf6pCIYlUtI%2Bgk0wJe0RaL5p3B7MMz%2FF8j2Ko%3D)

3. Annotate the documents with High importance first.

The more representative your training documents are of real-life documents, the better your model will perform with fewer annotations. We recommend double-checking the ones with Low importance.

All annotated documents will be marked as high importance during training data analysis, while similar ones will be marked as low importance. These updates help keyers annotate documents that are most valuable to the model. If you import documents that have already been annotated, they have high importance by default.

Next steps

• Review your training set for inconsistencies to determine whether you need more annotations or corrections. To learn more, see Labeling Anomaly Detection.

• As you annotate documents, their importance becomes High, and unannotated documents may be marked as Low importance. Reanalyze your data as you annotate to see the updated information.

• Based on the data analysis, the system might determine that a document with a Training Status of Never (excluded from the training) is of High importance. In such cases, we recommend double-checking the document.