【Azure OpenAI 】Azure OpenAI服务内容过滤

Detecting indirect attacks requires using document delimiters when constructing the prompt. See the Document embedding in prompts section to learn more.

Azure OpenAI Service includes default safety settings applied to all models, excluding Azure OpenAI Whisper. These configurations provide you with a responsible experience by default, including content filtering models, blocklists, prompt transformation, content credentials, and others. Read more about it here.

All customers can also configure content filters and create custom safety policies that are tailored to their use case requirements. The configurability feature allows customers to adjust the settings, separately for prompts and completions, to filter content for each content category at different severity levels as described in the table below. Content detected at the 'safe' severity level is labeled in annotations but is not subject to filtering and isn't configurable.

¹ For Azure OpenAI models, only customers who have been approved for modified content filtering have full content filtering control and can turn off content filters. Apply for modified content filters via this form: Azure OpenAI Limited Access Review: Modified Content Filters. For Azure Government customers, apply for modified content filters via this form: Azure Government - Request Modified Content Filtering for Azure OpenAI Service.

Configurable content filters for inputs (prompts) and outputs (completions) are available for the following Azure OpenAI models:

• GPT model series
• GPT-4 Turbo Vision GA^* (turbo-2024-04-09)
• GPT-4o
• GPT-4o mini
• DALL-E 2 and 3

Configurable content filters are not available for

• o1-preview
• o1-mini

^*Only available for GPT-4 Turbo Vision GA, does not apply to GPT-4 Turbo Vision preview

Content filtering configurations are created within a Resource in Azure AI Studio, and can be associated with Deployments. Learn more about configurability here.

Customers are responsible for ensuring that applications integrating Azure OpenAI comply with the Code of Conduct.

When the content filtering system detects harmful content, you receive either an error on the API call if the prompt was deemed inappropriate, or the finish_reason on the response will be content_filter to signify that some of the completion was filtered. When building your application or system, you'll want to account for these scenarios where the content returned by the Completions API is filtered, which might result in content that is incomplete. How you act on this information will be application specific. The behavior can be summarized in the following points:

• Prompts that are classified at a filtered category and severity level will return an HTTP 400 error.
• Non-streaming completions calls won't return any content when the content is filtered. The finish_reason value is set to content_filter. In rare cases with longer responses, a partial result can be returned. In these cases, the finish_reason is updated.
• For streaming completions calls, segments are returned back to the user as they're completed. The service continues streaming until either reaching a stop token, length, or when content that is classified at a filtered category and severity level is detected.

The table below outlines the various ways content filtering can appear:

Example request payload:

{
    "prompt":"Text example", 
    "n": 3,
    "stream": false
}

Example response JSON:

{
    "id": "example-id",
    "object": "text_completion",
    "created": 1653666286,
    "model": "davinci",
    "choices": [
        {
            "text": "Response generated text",
            "index": 0,
            "finish_reason": "stop",
            "logprobs": null
        }
    ]
}

Example request payload:

{
    "prompt":"Text example",
    "n": 3,
    "stream": false
}

Example response JSON:

{
    "id": "example",
    "object": "text_completion",
    "created": 1653666831,
    "model": "ada",
    "choices": [
        {
            "text": "returned text 1",
            "index": 0,
            "finish_reason": "length",
            "logprobs": null
        },
        {
            "text": "returned text 2",
            "index": 1,
            "finish_reason": "content_filter",
            "logprobs": null
        }
    ]
}

Example request payload:

{
    "prompt":"Content that triggered the filtering model"
}

Example response JSON:

"error": {
    "message": "The response was filtered",
    "type": null,
    "param": "prompt",
    "code": "content_filter",
    "status": 400
}

Example request payload:

{
    "prompt":"Text example",
    "n": 3,
    "stream": true
}

Example response JSON:

{
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1653670914,
    "model": "ada",
    "choices": [
        {
            "text": "last part of generation",
            "index": 2,
            "finish_reason": "stop",
            "logprobs": null
        }
    ]
}

Example request payload:

{
    "prompt":"Text example",
    "n": 3,
    "stream": true
}

Example response JSON:

 {
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1653670515,
    "model": "ada",
    "choices": [
        {
            "text": "Last part of generated text streamed back",
            "index": 2,
            "finish_reason": "content_filter",
            "logprobs": null
        }
    ]
}

Example request payload:

{
    "prompt":"Text example",
    "n": 1,
    "stream": false
}

Example response JSON:

{
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1652294703,
    "model": "ada",
    "choices": [
        {
            "text": "generated text",
            "index": 0,
            "finish_reason": "length",
            "logprobs": null,
            "content_filter_result": {
                "error": {
                    "code": "content_filter_error",
                    "message": "The contents are not filtered"
                }
            }
        }
    ]
}

When annotations are enabled as shown in the code snippet below, the following information is returned via the API for the categories hate and fairness, sexual, violence, and self-harm:

• content filtering category (hate, sexual, violence, self_harm)
• the severity level (safe, low, medium, or high) within each content category
• filtering status (true or false).

Optional models can be enabled in annotate (returns information when content was flagged, but not filtered) or filter mode (returns information when content was flagged and filtered).

When annotations are enabled as shown in the code snippets below, the following information is returned by the API for optional models:

When displaying code in your application, we strongly recommend that the application also displays the example citation from the annotations. Compliance with the cited license may also be required for Customer Copyright Commitment coverage.

See the following table for the annotation availability in each API version:

# os.getenv() for the endpoint and key assumes that you are using environment variables.

import os
from openai import AzureOpenAI
client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),  
    api_version="2024-03-01-preview",
    azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT") 
    )

response = client.completions.create(
    model="gpt-35-turbo-instruct", # model = "deployment_name".
    prompt="{Example prompt where a severity level of low is detected}" 
    # Content that is detected at severity level medium or high is filtered, 
    # while content detected at severity level low isn't filtered by the content filters.
)

print(response.model_dump_json(indent=2))

{ 
  "choices": [ 
    { 
      "content_filter_results": { 
        "hate": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "protected_material_code": { 
          "citation": { 
            "URL": " https://github.com/username/repository-name/path/to/file-example.txt", 
            "license": "EXAMPLE-LICENSE" 
          }, 
          "detected": true,
          "filtered": false 
        }, 
        "protected_material_text": { 
          "detected": false, 
          "filtered": false 
        }, 
        "self_harm": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "sexual": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "violence": { 
          "filtered": false, 
          "severity": "safe" 
        } 
      }, 
      "finish_reason": "stop", 
      "index": 0, 
      "message": { 
        "content": "Example model response will be returned ", 
        "role": "assistant" 
      } 
    } 
  ], 
  "created": 1699386280, 
  "id": "chatcmpl-8IMI4HzcmcK6I77vpOJCPt0Vcf8zJ", 
  "model": "gpt-35-turbo-instruct", 
  "object": "text.completion",
  "usage": { 
    "completion_tokens": 40, 
    "prompt_tokens": 11, 
    "total_tokens": 417 
  },  
  "prompt_filter_results": [ 
    { 
      "content_filter_results": { 
        "hate": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "jailbreak": { 
          "detected": false, 
          "filtered": false 
        }, 
        "profanity": { 
          "detected": false, 
          "filtered": false 
        }, 
        "self_harm": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "sexual": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "violence": { 
          "filtered": false, 
          "severity": "safe" 
        } 
      }, 
      "prompt_index": 0 
    } 
  ]
} 

For details on the inference REST API endpoints for Azure OpenAI and how to create Chat and Completions, follow Azure OpenAI Service REST API reference guidance. Annotations are returned for all scenarios when using any preview API version starting from 2023-06-01-preview, as well as the GA API version 2024-02-01.

{
    "error": {
        "message": "The response was filtered due to the prompt triggering Azure Content management policy. 
                   Please modify your prompt and retry. To learn more about our content filtering policies
                   please read our documentation: https://go.microsoft.com/fwlink/?linkid=21298766",
        "type": null,
        "param": "prompt",
        "code": "content_filter",
        "status": 400,
        "innererror": {
            "code": "ResponsibleAIPolicyViolation",
            "content_filter_result": {
                "hate": {
                    "filtered": true,
                    "severity": "high"
                },
                "self-harm": {
                    "filtered": true,
                    "severity": "high"
                },
                "sexual": {
                    "filtered": false,
                    "severity": "safe"
                },
                "violence": {
                    "filtered":true,
                    "severity": "medium"
                }
            }
        }
    }
}

A key aspect of Azure OpenAI's Responsible AI measures is the content safety system. This system runs alongside the core GPT model to monitor any irregularities in the model input and output. Its performance is improved when it can differentiate between various elements of your prompt like system input, user input, and AI assistant's output.

For enhanced detection capabilities, prompts should be formatted according to the following recommended methods.

The Chat Completion API is structured by definition. It consists of a list of messages, each with an assigned role.

The safety system parses this structured format and applies the following behavior:

• On the latest “user” content, the following categories of RAI Risks will be detected:
  
  • Hate
  • Sexual
  • Violence
  • Self-Harm
  • Prompt shields (optional)

This is an example message array:

{"role": "system", "content": "Provide some context and/or instructions to the model."}, 
{"role": "user", "content": "Example question goes here."}, 
{"role": "assistant", "content": "Example answer goes here."}, 
{"role": "user", "content": "First question/message for the model to actually respond to."} 

In addition to detection on last user content, Azure OpenAI also supports the detection of specific risks inside context documents via Prompt Shields – Indirect Prompt Attack Detection. You should identify parts of the input that are a document (for example, retrieved website, email, etc.) with the following document delimiter.

<documents> 
*insert your document content here* 
</documents>

When you do so, the following options are available for detection on tagged documents:

• On each tagged “document” content, detect the following categories:
  
  • Indirect attacks (optional)

Here's an example chat completion messages array:

{"role": "system", "content": "Provide some context and/or instructions to the model, including document context. 
\"\"\" <documents>\n*insert your document content here*\n</documents> \"\"\""}, 

{"role": "user", "content": "First question/message for the model to actually respond to."} 

When you tag unvetted documents for detection, the document content should be JSON-escaped to ensure successful parsing by the Azure OpenAI safety system.

For example, see the following email body:

Hello Josè, 

I hope this email finds you well today.

With JSON escaping, it would read:

Hello Jos\u00E9,\nI hope this email finds you well today. 

The escaped text in a chat completion context would read:

{"role": "system", "content": "Provide some context and/or instructions to the model, including document context. 
\"\"\" <documents>\n Hello Jos\\u00E9,\\nI hope this email finds you well today. \n</documents> \"\"\""}, 

{"role": "user", "content": "First question/message for the model to actually respond to."}

This section describes the Azure OpenAI content streaming experience and options. Customers can receive content from the API as it's generated, instead of waiting for chunks of content that have been verified to pass your content filters.

The content filtering system is integrated and enabled by default for all customers. In the default streaming scenario, completion content is buffered, the content filtering system runs on the buffered content, and – depending on the content filtering configuration – content is either returned to the user if it doesn't violate the content filtering policy (Microsoft's default or a custom user configuration), or it’s immediately blocked and returns a content filtering error, without returning the harmful completion content. This process is repeated until the end of the stream. Content is fully vetted according to the content filtering policy before it's returned to the user. Content isn't returned token-by-token in this case, but in “content chunks” of the respective buffer size.

Customers can choose the Asynchronous Filter as an extra option, providing a new streaming experience. In this case, content filters are run asynchronously, and completion content is returned immediately with a smooth token-by-token streaming experience. No content is buffered, which allows for a fast streaming experience with zero latency associated with content safety.

Customers must understand that while the feature improves latency, it's a trade-off against the safety and real-time vetting of smaller sections of model output. Because content filters are run asynchronously, content moderation messages and policy violation signals are delayed, which means some sections of harmful content that would otherwise have been filtered immediately could be displayed to the user.

Annotations: Annotations and content moderation messages are continuously returned during the stream. We strongly recommend you consume annotations in your app and implement other AI content safety mechanisms, such as redacting content or returning other safety information to the user.

Content filtering signal: The content filtering error signal is delayed. If there is a policy violation, it’s returned as soon as it’s available, and the stream is stopped. The content filtering signal is guaranteed within a ~1,000-character window of the policy-violating content.

Customer Copyright Commitment: Content that is retroactively flagged as protected material may not be eligible for Customer Copyright Commitment coverage.

To enable Asynchronous Filter in Azure OpenAI Studio, follow the Content filter how-to guide to create a new content filtering configuration, and select Asynchronous Filter in the Streaming section.

This is the same as default annotations.

data: { 
    "id": "", 
    "object": "", 
    "created": 0, 
    "model": "", 
    "prompt_filter_results": [ 
        { 
            "prompt_index": 0, 
            "content_filter_results": { ... } 
        } 
    ], 
    "choices": [], 
    "usage": null 
} 

Completion messages are forwarded immediately. No moderation is performed first, and no annotations are provided initially.

data: { 
    "id": "chatcmpl-7rAJvsS1QQCDuZYDDdQuMJVMV3x3N", 
    "object": "chat.completion.chunk", 
    "created": 1692905411, 
    "model": "gpt-35-turbo", 
    "choices": [ 
        { 
            "index": 0, 
            "finish_reason": null, 
            "delta": { 
                "content": "Color" 
            } 
        } 
    ], 
    "usage": null 
} 

The text field will always be an empty string, indicating no new tokens. Annotations will only be relevant to already-sent tokens. There may be multiple annotation messages referring to the same tokens.

"start_offset" and "end_offset" are low-granularity offsets in text (with 0 at beginning of prompt) to mark which text the annotation is relevant to.

"check_offset" represents how much text has been fully moderated. It's an exclusive lower bound on the "end_offset" values of future annotations. It's non-decreasing.

data: { 
    "id": "", 
    "object": "", 
    "created": 0, 
    "model": "", 
    "choices": [ 
        { 
            "index": 0, 
            "finish_reason": null, 
            "content_filter_results": { ... }, 
            "content_filter_raw": [ ... ], 
            "content_filter_offsets": { 
                "check_offset": 44, 
                "start_offset": 44, 
                "end_offset": 198 
            } 
        } 
    ], 
    "usage": null 
} 

Below is a real chat completion response using Asynchronous Filter. Note how the prompt annotations aren't changed, completion tokens are sent without annotations, and new annotation messages are sent without tokens—they're instead associated with certain content filter offsets.

{"temperature": 0, "frequency_penalty": 0, "presence_penalty": 1.0, "top_p": 1.0, "max_tokens": 800, "messages": [{"role": "user", "content": "What is color?"}], "stream": true}

data: {"id":"","object":"","created":0,"model":"","prompt_annotations":[{"prompt_index":0,"content_filter_results":
{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},
"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[],"usage":null} 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,
"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null} 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,
"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":"Color"}}],"usage":null} 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,
"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" is"}}],"usage":null} 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,
"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" a"}}],"usage":null} 

... 

data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":null,
"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},
"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},
"content_filter_offsets":{"check_offset":44,"start_offset":44,"end_offset":198}}],"usage":null} 

... 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,
"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":"stop","delta":{}}],"usage":null} 

data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":null,
"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},
"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},
"content_filter_offsets":{"check_offset":506,"start_offset":44,"end_offset":571}}],"usage":null} 

data: [DONE] 

{"temperature": 0, "frequency_penalty": 0, "presence_penalty": 1.0, "top_p": 1.0, "max_tokens": 800, 

"messages": [{"role": "user", "content": "Tell me the lyrics to \"Hey Jude\"."}], "stream": true}

data: {"id":"","object":"","created":0,"model":"","prompt_filter_results":[{"prompt_index":0,"content_filter_results":
{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},
"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[],"usage":null} 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,"model":
"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null} 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,"model":
"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":"Hey"}}],"usage":null} 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,
"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" Jude"}}],"usage":null} 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,
"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":","}}],"usage":null} 

... 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,
"model":"gpt-35- 

turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" better"}}],"usage":null} 

data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":null,
"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},
"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},
"content_filter_offsets":{"check_offset":65,"start_offset":65,"end_offset":1056}}],"usage":null} 

data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":"content_filter",
"content_filter_results":{"protected_material_text":{"detected":true,"filtered":true}},
"content_filter_offsets":{"check_offset":65,"start_offset":65,"end_offset":1056}}],"usage":null} 

data: [DONE]