Document Drift Management (Layout Triage)

Document Drift Management (Layout Triage)

Document Drift Management (also known as Layout Triage), is a post-processing feature that helps you manage documents that don't match layouts during Classification. When submissions don't meet the Structured Layout Match Threshold or are manually flagged as having incorrect or missing layouts, their pages are marked as unmatched.

This feature replaces the previous "Find Potential Layouts" and "No Layout Found" processes, offering a more streamlined and effective way to handle unmatched pages.

Document Drift Management allows you to determine and create the necessary layouts for these pages, ensuring that similar documents can be accurately classified and processed. In this article, you'll learn how to use Document Drift Management to handle unmatched pages efficiently.

Misclassified vs. Unmatched pages
Misclassified pages are those that the system recognizes but assigns to the wrong layout. You can manually reassign these pages to the correct one. Unmatched pages don't fit any existing layout, so they remain unclassified.

Grouping pages

Document Drift Management works by grouping unmatched pages based on their similarities. It uses a pixel-level analysis to identify pages that share common visual templates. These pages are then automatically clustered into groups, making it easier to identify potential new layouts.

Hyperscience recommends using this feature for Structured use cases.
Document Drift Management effectively handles Structured documents, where consistent formatting is essential for accurate classification.
It can be used for single-page Semi-structured documents (e.g. single-paged invoices), as well.

After the pages are grouped, you can review and adjust them as needed, then create new layouts from these groups.

Using Document Drift Management

To access Document Drift Management, click on Submissions in the left-hand sidebar, and then click the Unmatched tab.

The Unmatched tab displays submissions with pages that did not match any existing layouts during the Classification process. This tab is the starting point for managing and organizing unmatched pages using Document Drift Management.

Step 1 - Unmatched Submissions

The Unmatched Submissions tab displays a table of all submissions with unmatched pages.

Navigating the Unmatched Submissions tab

Filters

Define a specific range of submission IDs by entering values for its beginning and end:

Managing Columns

You can add and remove columns by clicking on the table’s menu.

Columns

The Unmatched Submissions table displays the following columns:

Grouping Pages

Follow the steps below to run a grouping job:

Grouping jobs in Document Drift Management run in the Trainer. If the trainer is busy, your job will be queued.

  1. Select the submissions containing the pages you want to group. Check the pages by using the preview button.
  2. After you’ve made your selections, click Group Pages. The system initiates a visual clustering process, grouping pages based on pixel-level similarities.
  3. After the grouping job is complete, click on the Page Groups tab.

Step 2 - Group Pages

The Page Groups tab displays a list of all clusters of similar pages that have been grouped. It helps you manage these groups, review ungrouped pages, and refine them before creating new layouts. Learn how to navigate the tab and how to use the triage experience below.

You need a minimum of 5 similar pages to create a group. If you have less than 5 similar pages, they will go to the Ungrouped category. However, you can manually create groups with fewer pages.

Navigating the Page Groups tab

The Page Groups table contains the following columns:

Your archived groups can be found in the Archive tab.

Layout Triage

Organize unmatched pages into layouts based on visual similarity using the triage experience.

Best for Structured documents. Similar pages of Semi-structured documents may be grouped together. Extra pages, like fax cover sheets, and those with fewer than 5 similar pages will go to the Ungrouped category.

To start the triage process, select the groups you want to manage, then click Triage Groups.

If you don’t select a group, clicking the Triage Groups button will open the Ungrouped category.

Follow the steps provided in the in-product guidance to begin the triage process.

Step 1: Review group

While pages are automatically grouped based on visual similarities, it's important to manually review and adjust these groups to ensure accuracy. To create a group:

  1. Click on a group in the To Do list on the left to view its contents and start the triage process.
  2. Click on a page to display it in the preview section.
  3. Select multiple pages and click New Group from Selection to create a more precise grouping. After you’ve created a new group from the selected pages, it will appear in the To Do list on the left.

If you manually create a group, similar pages will not be automatically added to it in the future. Ensure manual groups are accurately defined and adjusted as needed.

Merging groups and moving pages
Merging groups or moving pages into an automatic group converts it into a manual group, which will no longer update automatically with new similar unmatched pages.

Step 2: Enter layout metadata

You can define the following properties of the layout:

Each group in the triage experience represents a single-page index, not the entire layout. For example, if you have pages 1, 2, and 3, one group will represent page 1, another group will represent page 2, and so on.

Ensure all properties are filled out, as they are required to proceed with the triage process, and click Next.

Step 3: Provide blank

When creating a new Structured layout, provide a blank version of each page. Using an original form that hasn't been filled is most effective with Structured documents, where consistent formatting helps in accurate grouping.

To set a blank form:

  1. Select the page you want to define as blank form.
  2. Click Set as blank or upload a blank from your local machine by clicking Browse.

Triage Ungrouped

  1. Review pages — Review the pages in the Ungrouped folder and delete any unnecessary pages.
  2. Manually create page groups — Select sets of pages in the Ungrouped folder to manually create additional groups as needed.
  3. Follow the steps above for manually created page groups.

Similar Layout Variation Detection

Structured layouts evolve as document versions change, often resulting in unmatched submissions and reduced automation rates. Similar Layout Variation Detection addresses this issue by allowing you to create layout variations directly within Document Drift Management and providing automated recommendations when a new document closely resembles an existing layout.

Enabling Similar Layout Variation Detection To use Similar Layout Variation Detection, you need to enable it first by going to Administration > System Settings.

Example use case

A common example is the U.S. tax form W-8, which undergoes periodic updates while maintaining its overall structure. With Similar Layout Variation Detection:

Using Similar Layout Variation Detection

If enabled, Similar Layout Variation Detection will be available on the Page Groups tab. You’ll see an additional column called Similar variation that indicates whether a group resembles an existing layout or no similar variations were found.

Fields Overlay

Use the Fields Overlay feature to verify that the document is a variation of an existing layout. Once a similar variation is detected, it will be automatically selected in the Layout Triage experience.

Working across environments

To ensure smooth layout management across lower and production environments:

  1. Detect variations in your production environment.
  2. Create layouts in production.
  3. Download and import into the lower environment as necessary.
  4. Apply changes in the lower environment.
  5. Export and promote to production, ensuring synchronization.

Limitations

Step 3 - Create Potential Layouts

Once you have triaged your groups, they will be ready for layout creation. Find them in the Potential Layouts tab.

Next Steps

Once you’ve created your layout, add it to a release and assign it to a flow.