[Azure AI Studio ]生成式人工智能的评估和监控指标

{"question":"Which tent is the most waterproof?","context":"From our product list, the Alpine Explorer tent is the most waterproof. 
The Adventure Dining Table has higher weight.","answer":"The Alpine Explorer Tent is the most waterproof.","ground_truth":"The Alpine 
Explorer Tent has the highest rainfly waterproof rating at 3000m"} 

In this context, users engage in conversational interactions, either through a series of turns or in a single exchange. The generative AI model, equipped with retrieval mechanisms, generates responses and can access and incorporate information from external sources, such as documents. The Retrieval Augmented Generation (RAG) model enhances the quality and relevance of responses by using external documents and knowledge.

The test set format will follow this data format:

{"messages":[{"role":"user","content":"How can I check the status of my online order?"},{"content":"Hi Sarah Lee! To check the status of 
your online order for previous purchases such as the TrailMaster X4 Tent or the CozyNights Sleeping Bag, please refer to your email for 
order confirmation and tracking information. If you need further assistance, feel free to contact our customer support at 
support@contosotrek.com or give us a call at 1-800-555-1234.
","role":"assistant","context":{"citations":[{"id":"cHJvZHVjdF9pbmZvXzYubWQz","title":"Information about product item_number: 6",
"content":"# Information about product item_number: 6\n\nIt's essential to check local regulations before using the EcoFire 
Camping Stove, as some areas may have restrictions on open fires or require a specific type of stove.\n\n30) How do I clean and 
maintain the EcoFire Camping Stove?\n   To clean the EcoFire Camping Stove, allow it to cool completely, then wipe away any ash 
or debris with a brush or cloth. Store the stove in a dry place when not in use."}]}}]}

Built-in prompt used by the Large Language Model judge to score this metric:

You will be presented with a CONTEXT and an ANSWER about that CONTEXT. You need to decide whether the ANSWER is entailed 
by the CONTEXT by choosing one of the following rating: 

1. 5: The ANSWER follows logically from the information contained in the CONTEXT. 

2. 1: The ANSWER is logically false from the information contained in the CONTEXT. 

3. an integer score between 1 and 5 and if such integer score does not exist,  

use 1: It is not possible to determine whether the ANSWER is true or false without further information. 

Read the passage of information thoroughly and select the correct answer from the three answer labels. 

Read the CONTEXT thoroughly to ensure you know what the CONTEXT entails.  

Note the ANSWER is generated by a computer system, it can contain certain symbols, which should not 
be a negative factor in the evaluation. 

Built-in prompt used by the Large Language Model judge to score this metric (For question answering data format):

Relevance measures how well the answer addresses the main aspects of the question, based on the context. Consider whether 
all and only the important aspects are contained in the answer when evaluating relevance. Given the context and question,
 score the relevance of the answer between one to five stars using the following rating scale: 

One star: the answer completely lacks relevance 

Two stars: the answer mostly lacks relevance 

Three stars: the answer is partially relevant 

Four stars: the answer is mostly relevant 

Five stars: the answer has perfect relevance 

This rating value should always be an integer between 1 and 5. So the rating produced should be 1 or 2 or 3 or 4 or 5. 

Built-in prompt used by the Large Language Model judge to score this metric (For conversation data format) (without Ground Truth available):

You will be provided a question, a conversation history, fetched documents related to the question and a response to the 
question in the {DOMAIN} domain. Your task is to evaluate the quality of the provided response by following the steps below:  
 
- Understand the context of the question based on the conversation history.  
 
- Generate a reference answer that is only based on the conversation history, question, and fetched documents. Don't 
generate the reference answer based on your own knowledge.  
 
- You need to rate the provided response according to the reference answer if it's available on a scale of 1 (poor) to 
5 (excellent), based on the below criteria:  
 
5 - Ideal: The provided response includes all information necessary to answer the question based on the reference answer 
and conversation history. Please be strict about giving a 5 score.  
 
4 - Mostly Relevant: The provided response is mostly relevant, although it might be a little too narrow or too broad 
based on the reference answer and conversation history.  
 
3 - Somewhat Relevant: The provided response might be partly helpful but might be hard to read or contain other 
irrelevant content based on the reference answer and conversation history.  
 
2 - Barely Relevant: The provided response is barely relevant, perhaps shown as a last resort based on the reference
 answer and conversation history.  
 
1 - Completely Irrelevant: The provided response should never be used for answering this question based on the
 reference answer and conversation history.  
 
- You need to rate the provided response to be 5, if the reference answer can not be generated since no relevant 
documents were retrieved.  
 
- You need to first provide a scoring reason for the evaluation according to the above criteria, and then provide 
a score for the quality of the provided response.  
 
- You need to translate the provided response into English if it's in another language. 

- Your final response must include both the reference answer and the evaluation result. The evaluation result 
should be written in English.  

Built-in prompt used by the Large Language Model judge to score this metric (For conversation data format) (with Ground Truth available):

Your task is to score the relevance between a generated answer and the question based on the ground truth answer in 
the range between 1 and 5, and please also provide the scoring reason.  
 
Your primary focus should be on determining whether the generated answer contains sufficient information to address 
the given question according to the ground truth answer.   
 
If the generated answer fails to provide enough relevant information or contains excessive extraneous information,
 then you should reduce the score accordingly.  
 
If the generated answer contradicts the ground truth answer, it will receive a low score of 1-2.   
 
For example, for question "Is the sky blue?", the ground truth answer is "Yes, the sky is blue." and the generated 
answer is "No, the sky is not blue.".   
 
In this example, the generated answer contradicts the ground truth answer by stating that the sky is not blue, when 
in fact it is blue.   
 
This inconsistency would result in a low score of 1-2, and the reason for the low score would reflect the contradiction
 between the generated answer and the ground truth answer.  
 
Please provide a clear reason for the low score, explaining how the generated answer contradicts the ground truth answer.  
 
Labeling standards are as following:  
 
5 - ideal, should include all information to answer the question comparing to the ground truth answer， and the generated 
answer is consistent with the ground truth answer  
 
4 - mostly relevant, although it might be a little too narrow or too broad comparing to the ground truth answer, and the 
generated answer is consistent with the ground truth answer  
 
3 - somewhat relevant, might be partly helpful but might be hard to read or contain other irrelevant content comparing to
 the ground truth answer, and the generated answer is consistent with the ground truth answer  
 
2 - barely relevant, perhaps shown as a last resort comparing to the ground truth answer, and the generated answer
 contradicts with the ground truth answer  
 
1 - completely irrelevant, should never be used for answering this question comparing to the ground truth answer, and 
the generated answer contradicts with the ground truth answer  

Built-in prompt used by the Large Language Model judge to score this metric:

Coherence of an answer is measured by how well all the sentences fit together and sound naturally as a whole. Consider 
the overall quality of the answer when evaluating coherence. Given the question and answer, score the coherence of answer 
between one to five stars using the following rating scale: 

One star: the answer completely lacks coherence 

Two stars: the answer mostly lacks coherence 

Three stars: the answer is partially coherent 

Four stars: the answer is mostly coherent 

Five stars: the answer has perfect coherency 

This rating value should always be an integer between 1 and 5. So the rating produced should be 1 or 2 or 3 or 4 or 5. 

Built-in prompt used by the Large Language Model judge to score this metric:

Fluency measures the quality of individual sentences in the answer, and whether they are well-written and grammatically correct.
 Consider the quality of individual sentences when evaluating fluency. Given the question and answer, score the fluency of the 
 answer between one to five stars using the following rating scale: 

One star: the answer completely lacks fluency 

Two stars: the answer mostly lacks fluency 

Three stars: the answer is partially fluent 

Four stars: the answer is mostly fluent 

Five stars: the answer has perfect fluency 

This rating value should always be an integer between 1 and 5. So the rating produced should be 1 or 2 or 3 or 4 or 5. 

Built-in prompt used by the Large Language Model judge to score this metric:

A chat history between user and bot is shown below 

A list of documents is shown below in json format, and each document has one unique id.  

These listed documents are used as contex to answer the given question. 

The task is to score the relevance between the documents and the potential answer to the given question in 
the range of 1 to 5.  

1 means none of the documents is relevant to the question at all. 5 means either one of the document or combination of 
a few documents is ideal for answering the given question. 

Think through step by step: 

- Summarize each given document first 

- Determine the underlying intent of the given question, when the question is ambiguous, refer to the given chat history  

- Measure how suitable each document to the given question, list the document id and the corresponding relevance score.  

- Summarize the overall relevance of given list of documents to the given question after # Overall Reason, note that 
the answer to the question can soley from single document or a combination of multiple documents.  

- Finally, output "# Result" followed by a score from 1 to 5.  

  

# Question 

{{ query }} 

# Chat History 

{{ history }} 

# Documents 

---BEGIN RETRIEVED DOCUMENTS--- 

{{ FullBody }} 

---END RETRIEVED DOCUMENTS--- 

Built-in prompt used by the Large Language Model judge to score this metric:

GPT-Similarity, as a metric, measures the similarity between the predicted answer and the correct answer. If the information 
and content in the predicted answer is similar or equivalent to the correct answer, then the value of the Equivalence metric
 should be high, else it should be low. Given the question, correct answer, and predicted answer, determine the value of 
 Equivalence metric using the following rating scale: 

One star: the predicted answer is not at all similar to the correct answer 

Two stars: the predicted answer is mostly not similar to the correct answer 

Three stars: the predicted answer is somewhat similar to the correct answer 

Four stars: the predicted answer is mostly similar to the correct answer 

Five stars: the predicted answer is completely similar to the correct answer 

This rating value should always be an integer between 1 and 5. So the rating produced should be 1 or 2 or 3 or 4 or 5. 

• Evaluate your generative AI apps via the playground
• Evaluate your generative AI apps with the Azure AI Studio
• View the evaluation results
• Transparency Note for Azure AI Studio safety evaluations