Create an eval set
Write the questions and the expected key points that the judge uses to score each answer.
An eval set is the reference you measure a Smart Assistant against: a list of questions, each with the key points a good answer must mention. The scores are only as reliable as these questions and key points, so write them with care. If evals are new to you, start with Evals for Smart Assistants.
Create the eval set
-
Open Smart Assistants, click Evaluate assistants, then click New eval set.
-
Enter an Eval name. Add a Description if it helps your colleagues: it appears on the page of the eval set.
-
Under Questions, write the first Question, in the words a user would type.
-
Under Expected key points, write one key point in each field. Click Add key point for each additional one.
-
Click Add question and repeat for the other questions.
-
Click Create eval.
The eval set opens, ready for a first run.
An eval set needs a name and at least one question. It can hold up to 200 questions, and each question up to 50 key points. Empty questions and empty key points are removed when you save.
Choose the questions
-
Start from real questions. Use what people really ask: the questions your support team receives, or the conversations behind the feedback on your assistants.
-
Cover what the assistant is for. If it answers on three products, write questions on the three.
-
Include questions it answers badly today. These are the ones a better prompt should fix, and the score will show it.
-
Add a question it must not answer. For a question outside its content, the key point can be "Says that it does not have the information".
Start with a short list, for example ten questions on the most common subjects. Add a question each time a user reports a bad answer.
Write key points the judge can check
The judge takes each key point alone and decides whether the answer mentions it: yes or no. A key point that is easy to check gives a score you can trust.
| Instead of | Write | Why |
|---|---|---|
| "Explains the remote work policy" | "Two days per week" and "The manager approves the days" | One fact per key point. An answer that is half right then gets half the points, not zero. |
| "Gives the deadline" | "The deadline is 30 days after the invoice date" | A precise key point cannot be satisfied by a vague answer. |
| "The answer is clear and friendly" | "Gives the link to the request form" | Describe the content of the answer, not its style. |
A question can have no key points. The assistant still answers it and you can read the answer, but the question gets no score and does not count in the score of the run.
Edit an eval set
On the Eval sets page, open the menu at the end of the row and choose Edit eval. You can also open the eval set and click Edit. Change the name, the questions or the key points, then click Save.
Runs that already exist keep the questions and the results they recorded. After an edit, older and newer runs are no longer scored on the same questions.
Delete an eval set
On the Eval sets page, open the menu at the end of the row, choose Delete eval and confirm. The same action is in the More actions menu on the page of the eval set.
The runs of the eval set are removed with it.