Datasets

A dataset is a table of test items — each item is a JSON object whose keys are the dataset's columns. Datasets are versioned: you edit a draft version, release it to freeze it for evaluation runs, and fork a released version when you need to change it again. Manage them on the DATASETS tab of Evaluations.

Create a dataset

Add the dataset

On Evaluations → DATASETS, click Add new dataset. Give it a Name, optionally a Description, and optionally drop a JSON file with initial entries (Upload from file — a Sample JSON link shows the expected shape). Click Create.

A new dataset starts with a draft version (shown as v1-draft on the dataset page).

Add columns and entries

Open the dataset. While the active version is a Draft, the Dataset Entries table is editable:

• New column — add a field to the version's schema (for example question, context, ground_truth_answer).
• New dataset entry — add a row by filling in each column.
• Upload entries — upload a JSON file of items; a Sample JSON link in the dialog shows the expected shape (an array of objects whose keys match your columns).

Release the version

Click Release to freeze the version. Released versions are immutable — entry and column editing is disabled — and only datasets with at least one released version can be selected in an evaluation run. The version label switches from v1-draft (Draft) to v1 (Released).

Iterate with new versions

• New dataset version creates a fresh draft on the same dataset.
• Fork (available on a released version) copies it into a new draft so you can amend the data without touching the frozen version.

Switch between versions with the version selector on the dataset card.

Add items from traces

Every trace can become a dataset item. In the trace side sheet, open the evaluation section and click Add to dataset, then pick a Dataset and a draft Version in the Add trace to dataset dialog — only draft versions are offered, since released versions are immutable.

Trace-derived items carry input, output, status, and trace_id fields. A dataset made of such items can be scored directly in Dataset only run mode, where metrics can also reference the full recorded trace via the $.trace selector.

Datasets via the API

All UI actions map to management API endpoints:

# Create a dataset
curl -X POST "https://api.getdynamiq.ai/v1/datasets" \
  -H "Authorization: Bearer $DYNAMIQ_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "rag-regression", "description": "RAG regression set", "project_id": "<your-project-id>"}'

# Create a new draft version (schema is optional)
curl -X POST "https://api.getdynamiq.ai/v1/datasets/<dataset-id>/versions" \
  -H "Authorization: Bearer $DYNAMIQ_ACCESS_KEY"

# Add items to a draft version (JSON body)
curl -X POST "https://api.getdynamiq.ai/v1/dataset-versions/<version-id>/items" \
  -H "Authorization: Bearer $DYNAMIQ_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {
        "question": "What is the capital of France?",
        "context": "Paris is the capital city of France.",
        "ground_truth_answer": "Paris is the capital of France."
      }
    ]
  }'

# Release / fork a version
curl -X POST "https://api.getdynamiq.ai/v1/dataset-versions/<version-id>/release" \
  -H "Authorization: Bearer $DYNAMIQ_ACCESS_KEY"
curl -X POST "https://api.getdynamiq.ai/v1/dataset-versions/<version-id>/fork" \
  -H "Authorization: Bearer $DYNAMIQ_ACCESS_KEY"

# Create an item from a trace
curl -X POST "https://api.getdynamiq.ai/v1/dataset-items/from-trace" \
  -H "Authorization: Bearer $DYNAMIQ_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "<dataset-id>",
    "dataset_version_id": "<draft-version-id>",
    "trace_id": "<trace-id>"
  }'

File upload to POST /v1/dataset-versions/{dataset_version_id}/items also accepts multipart/form-data with a file field — .json (an array of objects) or .jsonl (one object per line). Other formats, including CSV, are rejected with unsupported file format.

Useful reads:

• GET /v1/datasets?project_id=... and GET /v1/dataset-versions?dataset_id=... — list datasets and versions.
• GET /v1/dataset-items?dataset_version_id=... — page through a version's items; PUT /v1/dataset-items/{dataset_item_id} updates one with {"data": {...}}.
• GET /v1/dataset-versions/{dataset_version_id}/download?format=json — download a version (json or jsonl).
• PUT /v1/dataset-versions/{dataset_version_id}/schema — add or delete columns with {"add": [{"name": "...", "schema": {...}}], "delete": [{"name": "...", "delete_from_items": true}]}.

Next steps