Python Code metrics

Creating a Python-Based Metric in Dynamiq: A Quick Guide

In our ongoing exploration of Dynamiq's evaluation capabilities, this article demonstrates how to create a custom Python-based metric. This allows you to execute specific logic to assess your AI workflows. We will guide you on implementing the “ExactMatch” metric, which determines if an answer precisely matches the expected result.

Steps to Create a Python-Based Metric

Access the Metric Creation Interface

1. Navigate to Create a Metric:

   • Go to Evaluations -> Metrics -> Create a Metric.

Select Python Option

1. Choose the Python Tab:

   • In the metrics creation interface, locate and click on the Python tab. This section lets you input custom Python code for defining metric logic.

Define the Metric

1. Set Metric Name:

   • Enter a descriptive name such as "ExactMatch" to identify the metric easily.

2. Enter Source Code:

   • Input your Python code in the provided editor. For the ExactMatch metric, use the following code:

def evaluate(answer: str, expected: str) -> int:
    if answer == expected:
        score = 1
    else:
        score = 0
    return score

• This code checks if the answer matches the expected value and returns 1 for a match and 0 otherwise.

1. Specify Inputs:

   • Ensure that the input fields like answer and expected are correctly mapped, as they are integral to your metric logic.

Finalize and Create the Metric

1. Create Metric:

   • Once the source code and inputs are correctly configured, click the Create button to add the Python-based metric to your evaluation toolkit.

Integrate Python Metric into Evaluation Runs

• Incorporate your newly created Python metric into evaluation runs to measure AI workflow performance effectively. Use it alongside existing workflows and datasets to obtain comprehensive insights.

Conclusion

By integrating custom Python code into your metrics, you have the flexibility to define precise evaluation criteria tailored to your specific needs. The ExactMatch metric exemplifies how straightforward code can enhance your evaluation process in Dynamiq. Leverage this functionality to explore more advanced custom metrics for your workflows.

For further information on integrating and managing your metrics effectively, refer to our previous documentation resources.