# Monitoring Model Performance - Published on Sep 24, 2025 - 7 minute(s) read > **Using features for Semi-structured documents** > This article mentions features used in the processing of Semi-structured documents. Your access to those features depends on your license package and pricing plan. > To learn which features are available to your organization and how to add more, contact your Hyperscience representative. After your model has been deployed in production, the next step is to monitor its performance. Machine learning models can degrade over time due to changes in input data, document structure, or annotation quality. That’s why monitoring the health of your models is crucial for your business case over time. Learn more about evaluating models in [Evaluating Model Training Results](https://help.hyperscience.ai/v42/docs/evaluating-model-training-results). In this article, you’ll learn how to: - Recognize common indicators of performance issues. - Use reporting tools to monitor automation and accuracy trends. - Identify when model retraining may be needed. - Monitor performance across Identification and Transcription models. - Validate your training data and annotations. Whether the visual formats of your documents are the same or changing, proactive monitoring helps your models maintain high performance and avoids downstream impact on your workflows. ## Common indicators of performance issues Your model’s performance may not remain consistent over time. As document formats evolve or as new data enters production, the model may become less accurate or require more manual input. Monitoring several key metrics can help you identify whether or not you need to retrain your model. In this section, you’ll learn more about these key metrics and how to monitor them using our reporting tools. ### **Using testing documents** We recommend setting aside 50-100 representative documents for testing your model’s performance. Doing so allows you to evaluate how the model performs on realistic data. - These documents should reflect the variety of inputs you expect in production. - They should **not be seen** by the model (e.g., should not be included in the training documents). - Run your testing documents through the system with a **100% QA Sample Rate** to assess accuracy and automation trends. To learn how to configure the QA settings for Identification and Transcription models in your flow, see [Document Processing Subflow Settings](https://help.hyperscience.ai/v42/docs/document-processing-subflow-settings). This approach helps you establish a reliable performance baseline. ### Decrease in Machine Accuracy > Accuracy > Accuracy indicates how often correct outputs are produced by the system. It measures how effective the model is at making correct predictions relative to the number of predictions made. It is calculated by comparing model predictions to the values that reached consensus during QA. > Accuracy can be influenced by factors like imbalanced datasets or inconsistent annotations. To learn more, see our [Accuracy](https://help.hyperscience.ai/v42/docs/accuracy) article. A drop in accuracy is one of the most common indicators that your model may not perform as expected. Monitoring accuracy helps you assess whether predictions from the model or manual input are aligned with QA outcomes. Learn more about QA in our [What is Quality Assurance?](https://help.hyperscience.ai/v42/docs/what-is-quality-assurance) article. > Target Accuracy > Target Accuracy is a manual setting that defines the minimum accuracy level required for a model to be considered successful - for example, 95% of fields must be correct. > It is specified at the **field level** - if even one occurrence within a field is incorrect or missing, the entire field is considered incorrect. > In contrast, the reported accuracy is measured at the **occurrence level**, where each value is evaluated separately. As a result, reported accuracy is often higher than the target accuracy. Lean how to set your target accuracy in [Document Processing Subflow Settings](https://help.hyperscience.ai/v42/docs/document-processing-subflow-settings). #### Manual Accuracy vs Machine Accuracy Report You can monitor accuracy trends using the Manual Accuracy vs Machine Accuracy report, available on the Accuracy page ( **Reporting** > **Accuracy**). This report shows two key metrics: - **Machine Accuracy**(blue line): Model-only output vs. QA. > Machine Accuracy > Machine Accuracy refers to the ratio of correct model predictions excluding the cases where human review was involved. It represents the model’s standalone performance on tasks that were not manually reviewed. - **Manual Accuracy**(green line): Human-edited output (via Supervision or consensus QA) vs. QA. To learn more, see [Manual Accuracy vs. Machine Accuracy](https://help.hyperscience.ai/v42/docs/manual-accuracy-vs-machine-accuracy). > Monitoring model types > Use this report to identify which model types—Field Identification, Table Identification, Classification, or Transcription—are underperforming. #### Decrease in Automation > Automation > Аutomation is the processing of data without human intervention. We measure it at the field-level—each individual field’s automation is evaluated independently. This approach enables more granular insights into model performance and helps identify exactly where manual review is needed. A decrease in Automation means that more tasks are being routed to Supervision instead of being completed automatically by the model. This happens when the system encounters new, unfamiliar, or ambiguous documents that it’s not confident enough to process without human review. > Automation Rate > The Automation Rate represents the proportion of extracted data with confidence scores exceeding a specified threshold. This threshold is determined by the level of accuracy you want your extracted data to have. To learn more, see [Automation](https://help.hyperscience.ai/v42/docs/automation0). #### Automation Report Track changes for a specific layout and flow with the Automation report on the Processing Time page ( **Reporting** > **Processing Time**). With this report, you can track automation trends for Identification and Transcription models over time. ### Next steps 1. Check new submissions for unexpected documents. - Look for visually different documents, new vendors, or formats that may not be represented in your training data. 2. Confirm that these documents exist in TDM. - If you already have examples of these documents, make sure they are well represented (typically 15-20 samples per vendor) and properly annotated. - If they are not included, add representative examples to your training documents and annotate them. 3. Retrain your model and test it again on documents not seen by the model. To learn more about model retraining, see [Model Validation Tasks](https://help.hyperscience.ai/v42/docs/model-validation-tasks) and [Training a Semi-structured Model](https://help.hyperscience.ai/v42/docs/v41-training-a-semi-structured-model). Learn how to improve your model’s performance in our [Improving Model Performance](https://help.hyperscience.ai/v42/docs/improving-model-performance) article.