Python Code Metrics

Creating a Python-Based Metric in Dynamiq: A Quick Guide

The Power of Custom Python Metrics

One of Dynamiq's most powerful features is the ability to create custom Python-based metrics. This functionality gives you unparalleled flexibility in evaluating your AI workflows. Rather than being limited to pre-defined evaluation criteria, you can implement exactly the logic you need to assess your specific use cases.

Creating Your Python Metric

To create a Python metric in Dynamiq:

1. Navigate to Evaluations → Metrics → Create new Metric

2. Select the Python tab

When you arrive at the Python tab, you'll see an example metric to help you get started. For even faster development, you can choose from several pre-built templates:

• Exact Match

• Email Presence

• Phone Presence

• String Presence

• Arithmetic Sum

• JSON Validity Check

How Python Metrics Work

Let's look at a simple example - the "Exact Match" metric:

def evaluate(answer: str, expected: str) -> int:
    if answer == expected:
        score = 1
    else:
        score = 0
    return score

This code checks if an answer exactly matches the expected value, returning a score of 1 for a match or 0 otherwise.

Important Note on Function Arguments

When you define a Python metric, Dynamiq automatically extracts the function arguments (in this case, answer and expected). During evaluation runs, you'll need to map these arguments to specific fields from your dataset or workflow output.

This mapping system allows your metrics to dynamically access the relevant data during evaluation.

Technical Implementation Details

Under the hood, Dynamiq uses RestrictedPython to safely execute your metric code. This means some operations may be prohibited for security reasons.

For a detailed overview of allowed Python libraries and implementation details, you can review the official Dynamiq GitHub repository.

Taking Your Evaluations to the Next Level

With custom Python metrics at your fingertips, you can now create sophisticated evaluation pipelines tailored to your exact requirements. This functionality enables you to:

• Implement domain-specific evaluation logic

• Create composite metrics that consider multiple factors

• Evaluate complex data structures and relationships

• Build metrics that align perfectly with your business objectives

By leveraging the full power of Python within your evaluation pipeline, you can gain deeper insights into your AI applications and make data-driven improvements with confidence.