Evals for Smart Assistants
Score the answers of a Smart Assistant on a fixed list of questions, before and after a change.
When you change the prompt of a Smart Assistant, some answers improve and others get worse. A few questions asked by hand do not show which. An eval asks the assistant the same questions every time and scores the answers, so you can compare a change with what you had before.
How an eval works
An eval has four parts.
| Part | What it is |
|---|---|
| Eval set | A list of questions that you write once and reuse. Each question has its expected key points: the facts a good answer must mention. |
| Run | One pass of an eval set with one assistant. The assistant answers every question, with the prompt and the model chosen for that run. |
| Judge | An AI model that reads each answer and decides, one key point at a time, whether the answer mentions it. |
| Score | For a question, the share of its key points found in the answer. For a run, the average of its questions. |
For example, an eval set for a human resources assistant contains this question:
-
Question: How many days can I work from home each week?
-
Expected key points: "Two days per week" and "The manager approves the days".
If the answer gives the two days but says nothing about the manager, the judge finds one key point out of two. The question scores 50%.
When to run an eval
-
Before you change a prompt. A run can use a different prompt from the one the assistant has today. You see the effect on every question before any user does.
-
To choose a model. Run the same eval set with two models and compare the answers.
-
After a large change in your content. When you reorganise a space or import many articles, run the eval again and check that the score has not dropped.
-
When users report a bad answer. Add their question to the eval set, so that every later run checks it.
Open evals
Open Smart Assistants from the main menu. At the top of the left column, above the list of assistants, click Evaluate assistants.
The Eval sets page lists every eval set of the platform. All platform administrators see the same list.
| Column | What it shows |
|---|---|
| Eval set | The name of the eval set. Click it to open its runs. |
| Questions | The number of questions in the set. |
| Runs | The number of runs started so far. |
| Last run | When the most recent run started. |
| Last result | The score of the most recent run, or its status if it did not complete. |
| Best result | The highest score among all the runs of the set. |
At the end of each row, New run starts a run. The menu next to it has Edit eval and Delete eval.
From a first eval set to a better assistant
-
Create an eval set with your questions and their key points.
-
Run the eval with the assistant as it is today. This first run is your reference.
-
Read the results to find the questions the assistant answers badly, and why.
-
Run the eval again with another prompt or another model, then compare the runs.
-
When a prompt gives better answers, apply it to the assistant from the run.
Questions
Does a run change my assistant?
No. A run only reads the settings of the assistant, even when you test another prompt. The assistant changes only when you click Apply to assistant on a run and confirm.
Who can see the eval sets and their results?
Platform administrators only. Other users do not see Evaluate assistants, and the eval pages are closed to them.