Monitor AI Threats and Events

The Events section offers a comprehensive view of Runtime evaluation results for user prompts and LLM responses in your AI Defense environment. It includes detailed event logs capturing AI-related activities such as detected prompts, responses, and policy rule matches. Advanced filtering options allow you to filter the Events view by time period, application, or event type, enabling targeted monitoring and efficient analysis of AI events.

Runtime enforcement points

AI runtime protection supports the following types of enforcement points:

The AI Defense Gateway is a cloud-based gateway that intercepts prompt and response traffic on the connection. Runtime protection evaluates content based on the policy you applied to the connection. Evaluation results appear in the Events log in AI Defense.
Multicloud Defense monitoring intercepts prompt and response traffic in the VPC. Runtime protection evaluates content based on the Multicloud Defense AI Guardrails profile you applied to the VPC. Evaluation results appear in Multicloud Defense logs.
API-enforced runtime protection is performed as its name suggests: when you invoke the API endpoint. You send prompt and/or response data to the AI Defense Inspection API endpoint, and the API returns a response with its evaluation. An event is also logged.

Invoke API-enforced AI Runtime evaluation

How API-enforced evaluation works

The typical API-enforced runtime protection usage pattern is as follows:

Your application calls the AI Defense inspect conversations API endpoint and provides the user prompt and/or model response for evaluation.
AI Runtime makes its evaluation based on the policy you've applied to the connection, or based on the rules you specified in the API request.
Based on Runtime's evaluation results, your application can return your model's response as-is, block the response, or modify it. It's important to note that in API-enforced usage, Runtime does not block prompts or responses; that responsibility is left to your application.

Learn more about the AI Defense Inspection API at https://developer.cisco.com/docs/ai-defense/.

Prerequisite

Set up AI Runtime as explained in Set up Runtime for the Inspection API.

Procedure

To test prompts and responses against your policy, call the AI Defense Runtime API endpoint as follows:

Use the API key you generated in Set up API-enforced Runtime.
Call the AI Defense "inspect conversations" API endpoint with your prompt and/or response. See the documentation for the AI Runtime endpoint at Inspect conversations - AI Defense - Cisco DevNet .

Your Runtime API endpoint address will be similar to the following. The example URL below includes the “us.” subdomain that specifies the us-west-2 AWS region. Replace this subdomain with the code for your region. For example:

https://us.api.inspect.aidefense.security.cisco.com/api/v1/inspect/chat

The regional URLs are:

https://us.api.inspect.aidefense.security.cisco.com for the us-west-2 region
https://ap.api.inspect.aidefense.security.cisco.com for the ap-ne-1 region
https://eu.api.inspect.aidefense.security.cisco.com for the eu-central-1 region

Important! When you use the API to check compliance with a policy, violations are reported as Events and in the API response body. In contrast, when you use the API to check compliance with a rule or rules, violations are returned only in the API response body.

Event log contents

The Events log contains:

Event Time: Timestamp indicating when the event occurred, enabling precise tracking and analysis.
Rule Action: Specifies the action taken by the system, such as block, allow, or alert, based on the guardrail or policy applied.
Message Type: Identifies whether the captured message is a prompt, response, or both, providing context to the event.
Application: The associated application where the event originated, offering insight into usage patterns and activity sources.
Model: Specifies the AI model involved in the interaction, helping to pinpoint the source of the AI activity.
Rule Name: The name of the policy or guardrail rule that triggered the event, aiding in understanding the enforcement mechanisms.

Filter the Events list

You can filter the events log view by clicking the settings icon on the right top corner of the table. You can select from

Event time (UTC): When the policy violation happened
Rule action: Whether the policy blocked the content or only logged it
Message type: Type of content examined
Application: Which of your applications this traffic belongs to. See Applications and connections.
Model: Which AI model the user was interacting with
Rule name: Name of the policy rule that was violated

Access to and storage of event logs

Event logs are visible to AI Defense users with the Admin or Analyst role. Event logs and other user data is securely stored in the AWS us-west2 region for customers in the Americas, in the AWS eu-central1 region for European customers, and in the AWS apne-1 region for customers in Asia and the Pacific region. Data sovereignty is not currently configurable.