Judgement

Ask typed questions about a record and get calibrated answers — three judges, the confidence measure, thresholds, and routing into Choice and Decision Table.

Judgement asks yes/no, choice and score questions about a piece of state and returns an answer, a probability distribution and a confidence for each one. One judge answers them, held in the node the way an agent holds its LLM: a SystemOne, which is a TypeSafe model built for the job, or an LLM or an agent node. The outputs are shaped for the nodes that route on them, so a Choice or a DecisionTable reads them without a transform.

A first judgement

The node runs standalone like any other, which is the quickest way to try a question set:

from dynamiq.connections import TypeSafe
from dynamiq.nodes.detectors import SystemOne
from dynamiq.nodes.tools import Judgement, JudgementOption, JudgementQuestion

triage = Judgement(
    id="triage",
    name="triage",
    judge=SystemOne(connection=TypeSafe()),  # reads TYPESAFE_API_KEY
    min_confidence=0.8,
    questions=[
        JudgementQuestion(
            name="angry",
            type="noul",
            instructions="Is the customer angry?",
        ),
        JudgementQuestion(
            name="team",
            type="choice",
            instructions="Which team should own this ticket?",
            options=[
                JudgementOption(name="billing", description="A charge, refund or invoice problem"),
                JudgementOption(name="support", description="A product usage question"),
                JudgementOption(name="sales", description="A question about buying or upgrading"),
            ],
        ),
        JudgementQuestion(
            name="severity",
            type="score",
            instructions="How severe is this for the customer?",
            options=[
                JudgementOption(name="none", description="No real impact"),
                JudgementOption(name="mild", description="Mild annoyance"),
                JudgementOption(name="blocking", description="Blocking their work"),
                JudgementOption(name="critical", description="Money lost or account at risk"),
            ],
        ),
    ],
)

result = triage.run(
    input_data={
        "state": (
            "I have been charged twice for my subscription this month and nobody has "
            "replied to my two previous emails. I want a refund today."
        )
    }
)
print(result.output["decisions"])
print(result.output["confidence"], result.output["needs_review"])

{'angry': True, 'team': 'billing', 'severity': 'critical'}
0.81 False

Set id to something readable: downstream selectors reference a node by its id, not its name.

Question types

`type`	Answer in `decisions`	Extra fields in `answers`	Options
`noul`	`True` / `False`	`probability` of yes	none
`choice`	the option name	`probabilities` per option	2 to 255
`score`	the level name	`score`, `index`, `probabilities`	2 to 10, lowest first

A score returns the probability-weighted mean of the level indices as score, so a run torn between two neighbouring levels lands between them, while decisions still names the most likely one.

A yes/no question also takes yes_when and no_when, the wording of each decision. Give every question a name of letters, digits and underscores: it is the key the answer is published under.

Choosing a judge

from dynamiq.connections import Anthropic as AnthropicConnection
from dynamiq.nodes.llms import Anthropic

# An LLM judge answers against a JSON schema built from the questions.
triage = Judgement(
    id="triage",
    name="triage",
    judge=Anthropic(connection=AnthropicConnection(), model="claude-opus-5", temperature=0),
    questions=[...],
)

An agent judge is the same, with tools: it may look something up before answering, and the tool calls come back under evidence.

The judge slot takes exactly one node, and only a SystemOne, an LLM or an agent is accepted; a node built without one is refused. Each judge carries its own settings — a SystemOne its TypeSafe connection, model and timeout, an LLM its connection and model — so switching judges is switching one node and nothing on Judgement changes.

SystemOne also runs on its own. Given a state and questions in wire form it returns the service's answers unread, which is useful for inspecting a raw distribution:

from dynamiq.nodes.detectors import SystemOne

answers = SystemOne(connection=TypeSafe()).run(
    input_data={
        "state": "I was charged twice",
        "questions": {"angry": {"type": "noul", "instructions": "Is the customer angry?"}},
    }
)
print(answers.output["answers"]["angry"])

An agent judge answers in whichever inference mode the agent is configured for. If a model returns an empty reply in the agent's default Thought/Action format, set the agent's inference_mode to FUNCTION_CALLING, which answers through a tool call instead.

Feeding the node

Pass a single state — a string, a dict or a list — or declare input_fields and map them from upstream outputs the way Rules takes its inputs:

triage = Judgement(
    id="triage",
    name="triage",
    judge=SystemOne(connection=TypeSafe()),
    input_fields=[NamedField(name="ticket"), NamedField(name="account")],
    questions=[...],
)

The state is sent whole, so keep it to what the questions need. System One accepts about 32,000 tokens of state per request and the node checks the size before it calls.

Outputs

Key	Type	Holds
`decisions`	`dict`	One entry per question — the part you route on
`answers`	`dict`	The full distribution per question, with `confidence`
`confidence`	`float`	The lowest confidence of any question
`needs_review`	`bool`	True when any question fell below `min_confidence`
`low_confidence`	`list`	The names of those questions
`content`	`str`	The answers as a short report, which is what an agent reads
`rationale`	`dict`	Per question, when `include_rationale` is on
`evidence`	`list`	The tool calls an agent judge made
`model`, `backend`, `confidence_source`	`str`	Which model answered, which judge, where the probabilities came from
`usage`	`dict`	`input_tokens`, `output_tokens`, `cost_usd`, in the shape LLM nodes report

Confidence and review

Confidence says how concentrated a distribution is, from 0 when every outcome is equally likely to 1 when one outcome takes everything. For n outcomes it is (n × max − 1) / (n − 1), and a yes/no answer is the two-outcome case, |2p − 1|. It is computed the same way whichever judge answered, so a threshold tuned on one judge still means something after you switch to another.

The node-level confidence is the lowest of the per-question confidences. min_confidence marks a run for review: any question below it is listed in low_confidence and sets needs_review. Leave it unset and nothing is ever flagged. noul_threshold, 0.5 by default, is the probability at which a yes/no answer becomes True.

For an LLM or agent judge, confidence_mode chooses where the probabilities come from:

verbalized (the default) asks the judge to state its own probabilities in one call.
sampling runs the judge samples times and uses how often the answers agree.

Sampling costs one call per sample and needs at least 2; it is refused with a SystemOne judge, which already returns calibrated probabilities from one call. With few samples it saturates — three answers that agree give a confidence of 1.00 whether the question was easy or the judge was lucky.

Routing on the answers

from dynamiq import Workflow
from dynamiq.flows import Flow
from dynamiq.nodes.node import InputTransformer, NodeDependency
from dynamiq.nodes.operators import Choice
from dynamiq.nodes.operators.operators import ChoiceOption
from dynamiq.nodes.types import ChoiceCondition, ConditionOperator
from dynamiq.runnables import RunnableConfig

route = Choice(
    id="route",
    name="route",
    options=[
        ChoiceOption(
            id="needs_human",
            name="needs_human",
            condition=ChoiceCondition(
                variable="$.needs_review", operator=ConditionOperator.BOOLEAN_EQUALS, value=True
            ),
        ),
        ChoiceOption(
            id="angry_billing",
            name="angry_billing",
            condition=ChoiceCondition(
                operator=ConditionOperator.AND,
                operands=[
                    ChoiceCondition(
                        variable="$.decisions.angry",
                        operator=ConditionOperator.BOOLEAN_EQUALS,
                        value=True,
                    ),
                    ChoiceCondition(
                        variable="$.decisions.team",
                        operator=ConditionOperator.STRING_EQUALS,
                        value="billing",
                    ),
                ],
            ),
        ),
    ],
    depends=[NodeDependency(node=triage)],
    input_transformer=InputTransformer(
        selector={
            "decisions": "$.triage.output.decisions",
            "confidence": "$.triage.output.confidence",
            "needs_review": "$.triage.output.needs_review",
        }
    ),
)

workflow = Workflow(flow=Flow(nodes=[triage, route]))
result = workflow.run(input_data={"state": ticket}, config=RunnableConfig(callbacks=[]))

A yes/no decision is a boolean, a choice and a level are strings, and confidence is a number, so each takes the matching operator family: BOOLEAN_EQUALS, STRING_EQUALS, and the numeric comparisons.

Two details save debugging time. A selector path names the upstream node by its id, and a path that does not resolve produces null rather than an error — a condition on null is false, so a mistyped path reads as a branch that never fires. The Choice result is keyed by option id too, so set explicit ids when you read the result by hand:

branches = result.output[route.id]["output"]
taken = {option.name: branches[option.id]["output"] for option in route.options}

A DecisionTable reads the same values as typed input columns, which scales better past a handful of rules.

As an agent tool

Give the node to an Agent in its tools and it reads content, the short report. Unless allow_agent_questions is False, the agent may add questions of its own at call time; they are merged with the configured ones by name.

Failures and limits

A judgement that cannot be trusted fails rather than guessing. A rejected API key, a malformed answer or a question the judge did not answer raises a tool error naming the question. Requests the server asks to retry — 408, 429, 529 and the rest of the 5xx family — are retried automatically, honouring Retry-After; an authentication failure is not retried. SystemOne.timeout, 30 seconds by default, bounds a single request.

YAML

triage:
  type: dynamiq.nodes.tools.Judgement
  name: triage
  judge:
    type: dynamiq.nodes.detectors.SystemOne
    name: system-one
    connection: typesafe_conn
    model: jev-latest
  min_confidence: 0.8
  questions:
    - id: q1
      name: is_urgent
      type: noul
      instructions: Does this need attention today?
      yes_when: Needs attention today
      no_when: Can wait
    - id: q2
      name: team
      type: choice
      instructions: Which team should own this ticket?
      options:
        - { id: o1, name: billing, description: A charge, refund or invoice problem }
        - { id: o2, name: technical, description: A bug or an outage }
        - { id: o3, name: sales, description: A question about buying or upgrading }
  input_transformer:
    selector:
      ticket: $.start.output.ticket

examples/components/core/dag/judgement_workflow.yaml in the SDK repository routes support tickets through a Judgement and a Decision Table, and run_judgement_workflow.py runs it.