Compare runs
Put several runs of an eval set side by side to see which prompt or model gives better answers.
A single score says how good a run is. A comparison says what a change did: which questions a new prompt fixed, and which ones it broke. The Compare runs page puts the runs of one eval set side by side, question by question.
Open a comparison
There are two ways to start.
-
From the eval set: open the eval set, tick two or more runs in the list, then click Compare runs in the bar that appears.
-
From a run: open the run, click Add run to compare and choose a second run.
On the comparison page, click Add run to compare to add another column. The menu lists the 50 most recent runs of the eval set, each with its score and its model.
You can only compare runs of the same eval set.
Read the comparison
At the top, five figures summarise the comparison: Smart assistants, Questions, Total runs, Models tried and Prompts. Smart assistants, Models tried and Prompts count the different assistants, models and prompts among the runs. If Prompts shows 2, the runs used two different prompts.
Below, each run has its own column, with the most recent first.
-
The header gives the number of the run and its score.
-
Parameters and Prompt show the assistant, the model, the date and the prompt of each run.
-
Each following row is one question, with the answer of each run, its sources and the verdict on each key point.
To see every result at once, switch to Condensed View at the top right. Each cell then shows the result of one run for one question, for example 2/4, and a medal marks the best score of the compared runs. Click a result to open the answer in a side panel, then use the arrows to move to the same question in the next run.
If a cell says "No answer for this question in this run.", the question was added to the eval set after that run, or removed before it.
Change one thing at a time
When two runs differ in several settings, you cannot tell which one changed the score. Keep everything identical except the setting you test. Rerun on the page of the eval set helps: it opens a new run with the settings of an earlier one, ready for a single change.
| To find out | Compare |
|---|---|
| Whether a new prompt is better | Two runs with the same assistant and model, and different prompts. |
| Which model answers better | Two runs with the same assistant and prompt, and different models. |
| Whether a content change helped | The same settings, run before and after the change. |
| Which assistant suits a set of questions | Two assistants limited to different content, with the same model. |
After the comparison
Open the run with the best answers, check the questions that still fail, and apply its prompt to the assistant.