Evaluation sets
Build test cases with example documents and the expected outcome to validate changes to an IDP template's pipeline: fields, running every case, the monitor and the pass rule.
Evaluation sets
Evaluation sets hold example documents with the expected outcome, to validate pipeline changes. Each case states what the template should do with a document — finish and pass its validations, or fail — and Run all checks every case at once against the current draft.
Use evaluation sets before publishing a version: they show whether a change to a filter, to extraction or to a post-processing step broke documents that already worked.
Test cases
Click Create case to add a case. Each case has:
| Field | Description |
|---|---|
| Name | Required, up to 80 characters. What this case checks, for example "Invoice with discount" or "Wrong kind of receipt". |
| Expected outcome | Expected success: the pipeline should finish and pass its validations. Expected failure: the pipeline should fail at some stage or validation. |
| Example document | The document processed in the test: pdf, png, jpg, jpeg, webp, tiff, tif or bmp, up to 15 MB. |
| Expected output | Optional. Turn on Set expected output and write the final output the pipeline should return, as text or as JSON. Left blank, the case only checks the validations. |
The list shows, for each case, the expected outcome, the document's name, the type of expected output, when the case last ran (or Never run) and the status of its last run.
When you delete a case, its past runs stay in the Runs history.
Run all
Run all opens the Test monitor and asks for confirmation before starting, because each case becomes a real run of the pipeline.
Each case is processed:
- with the current draft configuration, not the active version;
- in the Testing environment, under Runs;
- with the pages in the template's analysis scope and no parameters;
- one case after another, in the same round.
Each case counts toward the workspace's IDP request quota and uses OCR and AI according to the stages, like any other run. Robot validations really run the robot for every case.
A case without an example document ends as Failed.
Test monitor
The monitor follows the round while it happens:
- the progress, for example "3 of 5", with a bar; when it finishes with any failure, it shows the score, for example "3 passed, 2 failed";
- each case with its run status — Queued, Running, Finished or Failed —, the elapsed time, the error message when there is one, and the verdict Passed or Failed.
Click a case to open the run's details, with the trace of each stage. See Runs.
Pass rule
First, the case's run is assessed. It counts as successful when:
- identification, if enabled, accepted the document — a failed identification counts as a failure, with either Reject or Flag;
- every enabled validation passed;
- the expected output, if set, equals the pipeline's final output — the last post-processing step's output or, without one, the extracted text.
Then the case Passed when the outcome matches the expectation: a successful run in an Expected success case, or an unsuccessful run in an Expected failure case. Otherwise, it Failed.
How the expected output is compared depends on its type:
| Type | How it compares |
|---|---|
| JSON | Reads both sides as JSON and compares the values: the order of an object's keys does not matter, the order of a list's items does. An invalid expected JSON never matches. |
| Text | Compares the text without leading and trailing whitespace and with line breaks normalized. |
A run that ends with an error — a missing document, a model failure or an error in a script, for example — shows as Failed with no verdict, even in an Expected failure case. To test rejections, use identification or validations, which fail the result without bringing the run down.