Evaluation Runs

Open the run dialog

On Evaluations → Runs, click Run evaluation. The New Evaluation Run dialog opens.

Name it and pick the data

Enter a Name, then choose a Dataset and Dataset version. Only datasets with at least one released version appear — release a draft first if yours is missing (see Datasets). Pick a released version: a run started on a draft fails with Dataset version is not released.

Choose the run mode

• With workflow — Re-run rows through agents, then score. Each dataset row is fed into one or more workflows, and metrics can score the fresh outputs. Rows captured from traces are the exception: they are scored as recorded, without running the workflow.
• Dataset only — Score the dataset directly. No workflow executes; metrics read what was recorded. Use this for datasets captured from traces. Rows without trace fields (input, output, status, trace_id) are not scored in this mode — their tasks fail.

(With workflow) add agents

Click Add agent and pick a workflow (Agent) and an Agent version. Then fill in its Input Mappings: each field of the workflow's Input node gets a selector drawn from your dataset columns, like $.dataset.question.

Add metrics and map their inputs

Click Add metric and pick a saved Metric. For every metric input, choose a source:

• $.dataset.<column> — a dataset field.
• $.workflow.<field> — a field of the chosen agent version's Output node (with-workflow mode). Only Output node fields are listed; to score another node's result, such as the documents a retriever returned, add it to the Output node — see Score intermediate results.
• $.trace — the full recorded trace, offered in dataset-only mode when items carry input, output, status, and trace_id fields. It's passed as text — every node's input and output, with strings longer than 2,000 characters truncated — so it suits LLM-as-a-judge metrics.

Selectors are picked from a list. Nested paths such as $.dataset.input.question also work, but only in runs started through the API.

Run and watch the results

Start the run. It appears in the runs list with NAME, STATUS, STARTED BY, and STARTED AT columns; statuses progress through pending, running, and end at succeeded, failed, or canceled.

Open the run to see the Evaluations results table: one row per dataset item with the item's fields and one column per metric score, plus run status. Click Download results to save the full result set as JSON (disabled while the run is still running).

Expose the retriever's output

In the workflow, open the Output node and click Add field. Name the field kb_content, type / in its value, and pick content under your retriever node — the picker lists every node upstream of the Output node, and the value becomes $.knowledge-base-retriever.output.content (using your retriever node's name). Save a new version.

The retriever's content is one text block with every retrieved chunk and its source metadata (such as title, source, and the relevance score). Its documents output holds the same results as a list of {id, content, metadata, score} objects.

Add the expected source to the dataset

Give the dataset a question column and a source_document column holding the document each answer should be grounded in, then Release the version.

Write the metric

Create an LLM-as-a-judge metric whose instructions reference all three inputs:

Decide whether the answer was built from the correct source.

Question: {{question}}
Expected source document: {{source_document}}
Documents the retriever returned: {{kb_retrieved_document}}

Score 1 if the expected source document is among the retrieved documents, otherwise 0.
Respond exactly as {"score": X}.

Map the inputs in the run

Start a With workflow run, add the agent at the version you just saved, and map its question input to $.dataset.question. Then add the metric and map: