Run an eval
Start a run with an assistant, a prompt and a model, then follow it, cancel it or run it again.
A run asks one Smart Assistant every question of an eval set and scores the answers. Each run records the assistant, the prompt and the model it used, so you can always tell what a score measures.
Start a run
-
On the Eval sets page, click New run on the row of the eval set. You can also open the eval set and click New run at the top of the page.
-
Under Smart assistant, select the assistant to evaluate.
-
Smart assistant prompt is filled with the current prompt of the assistant. Leave it as it is to test the assistant your users have today, or edit it to test another prompt.
-
Under Model, choose the AI model that answers. Leave the field empty to use the default model of your platform.
-
Click Run.
The page of the run opens. The answers and the scores appear as they are ready: you do not need to reload the page, and you can leave it and come back later.
What a run uses
-
The prompt and the model you set when you start the run.
-
The content of the assistant. The run searches the content the assistant is limited to (its spaces, templates and tags), as set when the run starts. A later change to the assistant does not affect a run that already exists.
-
Your access rights. The assistant answers as it would in a conversation with you: it reads only the articles you can see. A colleague with other access rights can get different answers on the same eval set.
Each question is asked on its own, as the first message of a new conversation.
Follow a run
Runs are numbered in the order they start: Run 1, Run 2 and so on. A run has one of five statuses.
| Status | Meaning |
|---|---|
| Pending | The run waits to start. |
| Running | The assistant is answering the questions. |
| Completed | The run reached the end. Some questions can still be marked Failed if the assistant could not answer them. |
| Failed | The run stopped on an error. The error message is at the top of the page of the run, and the answers already recorded stay available. |
| Cancelled | An administrator stopped the run. |
Cancel a run
Open the run and click Cancel run while it is pending or running, then confirm.
If the run was pending, it does not start. If it had already started, it stops and keeps the answers it has recorded.
See all the runs of an eval set
Click an eval set on the Eval sets page. Its runs are listed with the most recent first.
| Column | What it shows |
|---|---|
| Run | The number of the run. Click it to read its results. |
| Smart assistant | The assistant evaluated. If it was deleted since, the name it had at the time is shown. |
| Model | The AI model that answered. |
| Timestamp | The date and time the run started. |
| Created by | The administrator who started the run. |
| Score | The score of the run, In progress or Failed. A medal marks the best score of all completed runs. |
The panel on the right shows the description of the eval set and its questions. Click a question to see its key points.
Run the eval again
In the list of runs, click Rerun at the end of the row of the run you want to repeat. The dialog opens with the assistant, the prompt and the model of that run. Start the new run as it is, or change one setting first.
Two cases need your attention:
-
If the assistant was deleted, select another one.
-
If the model is no longer available on your platform, the field is empty and the run uses the default model.
Run the eval again after each change you want to measure: a new prompt, another model, or a change in the content the assistant reads. Then compare the runs.