Skip to main content
Roberty Studio
CODECode

Evaluations

Create evaluation sets with inputs and an expected outcome and test the code project on the Preview environment.

Evaluations

Evaluations are test cases with inputs and expected outputs, run on the Preview environment. Use them to check that a new version of the code still answers as it should before taking it to Production.

info

To run evaluations, the project needs a Preview device that is online and has a ready deployment. Projects that receive code by .zip have no Preview and therefore cannot run evaluations. See Devices.

If Preview isn't ready, the Evaluations tab tells you why: no Preview device is configured, it is offline, or it doesn't have a ready deployment yet.

Creating a set​

Click Create set and fill in:

FieldRequiredDescription
NameYesUp to 80 characters. It can't repeat the name of another set in the project.
DescriptionNoWhat this set validates.
InputsNoKey/value pairs sent to the robot as arguments, in the roberty-args-file-path file. Keys can't repeat.
Expected outcomeYesMust succeed (default): the process must exit with code 0. Must fail: the process must exit with a non-zero code.
Validate outputNoOn by default. Compares what the robot wrote to the result file with the Expected output — even when the expected outcome is a failure.
Expected outputNoFlat JSON. Placeholders like <any> match any value.

Expected output example:

json
{ "status": "ok", "processedAt": "<any>" }

To read the inputs and write the result in your code, see Arguments and result and the Official SDK.

Running, duplicating and removing​

  • Run all — runs every set in the project, one after the other.
  • Run — runs only that set. If a run is already in progress, the set doesn't start.
  • Duplicate — opens the form filled in with the set's data, to save it as a new set with another name.
  • Remove — deletes the set, after a confirmation. This action cannot be undone.

Each set shows whether it expects success or an error, how many arguments it has and when it last ran.

Monitor​

When you run sets, a monitor follows each one and shows the overall progress — how many runs finished and how many passed and failed.

StatusMeaning
Scheduling runs…The runs are being created on the Preview device.
WaitingThe run hasn't started yet.
RunningThe robot is running.
PassedThe run finished and met the expected outcome.
RejectedThe run finished but did not meet the expected outcome.
FinishedThe run finished without an evaluation result.
FailedThe run failed.

A set passes when the exit code matches the Expected outcome and, with Validate output on, the actual output matches the Expected output. Open a set in the monitor to compare the expected outcome with the exit code and the actual output.

warning

Evaluation results are not saved. Closing the monitor while something is running stops the run on the device; closing it after it finishes means losing access to the results.

How the robot sees an evaluation​