# Workflows ## **Section 3: Implementing the Workflows** In this section, we'll set up the two workflows that will generate answers for evaluation using the dataset we prepared in **Section 2**. These workflows are essential for testing the effectiveness of our evaluation metrics and demonstrating how different approaches can impact the quality of the answers. *** ### **Overview of the Workflows** We'll implement two distinct workflows: 1. **Workflow A: Perfect Answering Workflow** * Designed to produce accurate, high-quality answers that closely align with the ground truth. * Mimics an ideal AI assistant. 2. **Workflow B: Imperfect Answering Workflow** * Intentionally introduces mistakes and suboptimal answers. * Helps test the evaluation metrics' ability to detect and score various issues. By comparing the outputs of these workflows, we'll be able to assess how effectively our evaluation framework differentiates between high-quality and lower-quality answers. *** ### **Implementing Workflow A: Perfect Answering Workflow** #### **Objective** * To generate precise, accurate, and well-articulated answers that align closely with the ground truth. #### **Approach** * Provide clear instructions to the AI model to ensure high-quality responses. * Encourage accuracy, completeness, and clarity in the answers. #### **Workflow A Prompt** Here's the prompt that will guide the AI model in Workflow A: ```markdown You are an expert assistant providing precise and accurate answers to questions. Ensure that your answers are correct, concise, and, where appropriate, include brief explanations to enhance understanding. **Instructions:** - Provide accurate information in response to the question. - Keep the answer clear and concise. - Include a brief explanation if it adds value. - Do not include irrelevant information. - Use proper grammar and spelling. - Maintain a professional tone. **Question:** {{question}} **Answer:** ``` **Notes:** * **Variable Placeholder:** Replace `{{question}}` with the actual question from the dataset. * **Consistency:** Maintain a consistent format for all inputs to ensure uniform outputs. * **Tone and Style:** Encourage professionalism and clarity in the answers. #### **Implementation Steps** 1. **Prepare the Input:** * For each question in your dataset, create an input to the AI model using the Workflow A prompt. * Ensure the `{{question}}` placeholder is correctly replaced. 2. **Run the Workflow:** * Input the prepared prompt into the AI model (e.g., GPT-based model) to generate the answer. * Collect the generated answer for evaluation. #### **Example** **Input Prompt:** ```markdown You are an expert assistant providing precise and accurate answers to questions. Ensure that your answers are correct, concise, and, where appropriate, include brief explanations to enhance understanding. **Instructions:** - Provide accurate information in response to the question. - Keep the answer clear and concise. - Include a brief explanation if it adds value. - Do not include irrelevant information. - Use proper grammar and spelling. - Maintain a professional tone. **Question:** What is the capital of Japan? **Answer:** ``` **Generated Answer:** "Tokyo." *** ### **Implementing Workflow B: Imperfect Answering Workflow** #### **Objective** * To generate answers with intentional mistakes or issues, such as factual errors, incomplete information, or unclear language. * To test the evaluation metrics' ability to detect and score these imperfections. #### **Approach** * Provide instructions that allow the AI model to include errors or suboptimal content in the answers. * Simulate potential real-world mistakes that an AI assistant might make. #### **Workflow B Prompt** Here's the prompt that will guide the AI model in Workflow B: ```markdown You are an assistant providing answers to questions, but you sometimes make mistakes or include irrelevant information. **Instructions:** - Provide an answer to the question, but you may include: - Minor factual errors. - Incomplete or insufficient information. - Irrelevant or off-topic details. - Grammatical or spelling mistakes. - Do not fully disregard the question. - Aim for a casual tone. **Question:** {{question}} **Answer:** ``` **Notes:** * **Variable Placeholder:** Replace `{{question}}` with the actual question from the dataset. * **Intentional Imperfections:** The prompt allows the AI model to introduce mistakes, helping to test the evaluation framework. #### **Implementation Steps** 1. **Prepare the Input:** * For each question in your dataset, create an input to the AI model using the Workflow B prompt. * Ensure the `{{question}}` placeholder is correctly replaced. 2. **Run the Workflow:** * Input the prepared prompt into the AI model to generate the answer. * Collect the generated answer for evaluation. 3. **Automate the Process (Optional):** * Utilize a tool like Dynamiq to automate the process for all questions in the dataset. #### **Example** **Input Prompt:** ```markdown You are an assistant providing answers to questions, but you sometimes make mistakes or include irrelevant information. **Instructions:** - Provide an answer to the question, but you may include: - Minor factual errors. - Incomplete or insufficient information. - Irrelevant or off-topic details. - Grammatical or spelling mistakes. - Do not fully disregard the question. - Aim for a casual tone. **Question:** What is the capital of Japan? **Answer:** ``` **Generated Answer:** "The capital of Japan is Kyoto, a beautiful city with lots of history." *** ### **Running the Workflows** #### **Preparation** * **Ensure Access to AI Model:** Make sure you have access to an AI language model capable of generating the answers (e.g., OpenAI's GPT-3.5-Turbo or GPT-4). *** **Next Steps:** * Proceed to **Section 4: Evaluating the Workflow Outputs**, where we'll assess the generated answers using our evaluation metrics. * Prepare the collected answers and ensure they're ready for the evaluation process. *** *Excited to see how our workflows perform under evaluation? Let's move forward and find out!* 🚀 --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.getdynamiq.ai/evaluations-code/workflows.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.