Read the results of a run
Understand the score of a run, check each answer against its key points and apply the best prompt.
The page of a run shows what the Smart Assistant answered to each question and how the judge scored it. To open it, click an eval set on the Eval sets page, then click a run.
How the score is calculated
-
For a question: the share of its expected key points that the judge found in the answer. With three key points found out of four, the result is 3/4, or 75%.
-
For a run: the average of the scores of its questions. Questions without key points have no score and are not counted.
The score of the run is next to its title, with its status. While the run is in progress, the score changes as the answers arrive.
The colour gives the level at a glance: green from 80%, yellow from 50%, red below.
Choose a view
The two buttons at the top right change the view.
-
List View: each question with the full answer and its key points. Use it to read a run from start to end.
-
Condensed View: one line per question, with its result. Use it to find the weak questions quickly, then click a question to open its answer in a side panel.
Check an answer
For each question, the page shows three things.
-
Answer: the text the assistant wrote, with the score of the question.
-
Sources: click to list the articles and files the assistant read to write its answer. Click a source to preview it. A source that is no longer published, or that you cannot see, is not listed.
-
Expected key points: the verdict of the judge on each one.
| Verdict | Meaning |
|---|---|
| Satisfied (tick) | The judge found the key point in the answer. |
| Not satisfied (cross) | The judge did not find it. The key point is struck through, and the explanation of the judge is below it. |
| Not graded (dash) | The judge gave no verdict, for example because the run was cancelled or the question failed. |
When the result of a question is Failed, the assistant could not answer or the judge could not score the answer. The error message is shown in place of the answer.
Find the cause of a low score
A key point that is not satisfied has three possible causes. The sources and the explanation of the judge tell you which one.
| What you see | Likely cause | What to do |
|---|---|---|
| The answer is right, but the judge did not accept it. | The key point is too strict or too vague. | Rewrite the key point. |
| The right article is not in the sources. | The assistant did not find the information: it is missing, hard to find, or outside the content the assistant is limited to. | Check the content settings of the assistant, then the articles themselves. |
| The right article is in the sources, but the answer is wrong or incomplete. | The prompt does not guide the assistant well enough. | Test another prompt in a new run. |
The judge is an AI model, and it can be wrong. Read its explanation before you act on a verdict.
Review the settings of the run
The panel on the right records how the run was made.
-
Settings: the assistant, the model, the date the run started and the administrator who started it.
-
Prompt: the prompt used for this run, which can differ from the prompt the assistant has today.
Apply the prompt to the assistant
When a run used a better prompt than the one the assistant has, you can copy it to the assistant from the run.
-
Open the run.
-
In the right panel, next to Prompt, click Apply to assistant.
-
Read the confirmation and click Update prompt.
The prompt of the assistant is replaced at once, for everyone who uses it. Past runs are not affected.
Apply to assistant appears only when the prompt of the run differs from the current prompt of the assistant, the assistant still exists, and you can manage Smart Assistants.
Compare with another run
Click Add run to compare at the top of the page and choose a run of the same eval set. See Compare runs.