Evaluations
Create evaluation sets with inputs and an expected outcome and test the code project on the Preview environment.
Evaluations
Evaluations are test cases with inputs and expected outputs, run on the Preview environment. Use them to check that a new version of the code still answers as it should before taking it to Production.
To run evaluations, the project needs a Preview device that is online and has a ready deployment. Projects that receive code by .zip have no Preview and therefore cannot run evaluations. See Devices.
If Preview isn't ready, the Evaluations tab tells you why: no Preview device is configured, it is offline, or it doesn't have a ready deployment yet.
Creating a set
Click Create set and fill in:
| Field | Required | Description |
|---|---|---|
| Name | Yes | Up to 80 characters. It can't repeat the name of another set in the project. |
| Description | No | What this set validates. |
| Inputs | No | Key/value pairs sent to the robot as arguments, in the roberty-args-file-path file. Keys can't repeat. |
| Expected outcome | Yes | Must succeed (default): the process must exit with code 0. Must fail: the process must exit with a non-zero code. |
| Validate output | No | On by default. Compares what the robot wrote to the result file with the Expected output — even when the expected outcome is a failure. |
| Expected output | No | Flat JSON. Placeholders like <any> match any value. |
Expected output example:
{ "status": "ok", "processedAt": "<any>" }
To read the inputs and write the result in your code, see Arguments and result and the Official SDK.
Running, duplicating and removing
- Run all — runs every set in the project, one after the other.
- Run — runs only that set. If a run is already in progress, the set doesn't start.
- Duplicate — opens the form filled in with the set's data, to save it as a new set with another name.
- Remove — deletes the set, after a confirmation. This action cannot be undone.
Each set shows whether it expects success or an error, how many arguments it has and when it last ran.
Monitor
When you run sets, a monitor follows each one and shows the overall progress — how many runs finished and how many passed and failed.
| Status | Meaning |
|---|---|
| Scheduling runs… | The runs are being created on the Preview device. |
| Waiting | The run hasn't started yet. |
| Running | The robot is running. |
| Passed | The run finished and met the expected outcome. |
| Rejected | The run finished but did not meet the expected outcome. |
| Finished | The run finished without an evaluation result. |
| Failed | The run failed. |
A set passes when the exit code matches the Expected outcome and, with Validate output on, the actual output matches the Expected output. Open a set in the monitor to compare the expected outcome with the exit code and the actual output.
Evaluation results are not saved. Closing the monitor while something is running stops the run on the device; closing it after it finishes means losing access to the results.
How the robot sees an evaluation
- The run happens on Preview with
roberty-triggerset toevaluation, and the Inputs arrive in the arguments file. See Environment variables. - The exception entry point does not run on evaluation runs.