Set up Runtime for Gateway Interception

AI Defense runtime gateway enforcement

The gateway-deployed approach to AI Defense runtime enables AI Defense to monitor traffic to and from your AI model or application. In this approach, the AI Defense Gateway acts as a proxy between your AI application and its users.

Policies and gateway-deployed AI Defense runtime

Gateway-deployed AI Defense runtime will log an event each time your policy is triggered, and your policy rules can be configured to block any prompt or response that violates your policy.

If you wish to perform AI Defense runtime evaluation via the API, see the Set up Runtime for the Inspection API, instead.

Supported model endpoint types

The following model endpoint types are supported.

AWS Bedrock (model URLS like bedrock-runtime.*.amazonaws.com)
OpenAI (model URLS like api.openai.com)
Azure OpenAI (model URLS like *.openai.azure.com)
Google Vertex (model URLS like *aiplatform.googleapis.com)
Cohere (model URLS like api.cohere.com)
Mistral (model URLS like api.mistral.ai)

You may not apply AI Defense runtime to an arbitrary or self-hosted model endpoint address.

Set up gateway-enforced AI Defense runtime protection

To use the AI Defense runtime gateway-enforced approach, you will create an application and connection in AI Defense, add a policy to the connection, and set your AI application to use the AI Defense Gateway address as the LLM API endpoint.

You do not need to deploy a gateway or other monitoring tool in order to intercept prompts and responses. Gateway deployment is cloud-based and automatic.

To set this up:

Create a gateway-style application in the AI Defense UI, The application is a wrapper that provides a connection guide and allows you to look up your model's AI Defense runtime protection and policy later in the AI Defense UI.
In the application, create a connection to your LLM API.
Add a policy to the connection.
Use the Connection Guide to find out all the details for connecting to the AI Defense Gateway address. Ensure that your AI application uses this AI Defense Gateway endpoint rather than your LLM's endpoint.
Allowlist the public egress IP address for the AI Defense gateway service, as shown below.

Each time a prompt or response triggers your policy, your policy rules will log it to the Event log, and if the prompt or response represents a potential risk, AI Defense blocks it.

IP addresses to allowlist for the AI Defense gateway

In your environment's ACL configuration, allowlist the public egress IP address for the AI Defense gateway service. Failure to do this will result in a 403 error. Allowlist the IP address for your region:

us-west-2: 52.12.28.51/32
eu-central-1: 63.176.154.159/32
ap-northeast-1: 35.79.78.177/32