Platform / Inference & Model Governance
Govern the
model call. Keep your gateway.
Which models may be called, what may be sent, what may come back, what may happen next. Four decisions on one call, under one policy, in one record — without standing in the token path. Your gateway keeps the connection, the keys and the retries. GovernorAI returns the verdict.
- Provider keys never leave your gateway
- Policy in your engine: native or OPA/Rego
- In-band refusals
- Portkey, LiteLLM, Envoy, NGINX, Apigee, APIM
# model registry — the governance record
name: gpt-4o
provider: openai
status: approved # pending_review | approved
# restricted | deprecated | blocked
risk_tier: medium
approved_by: a.okafor # written on the status change
approved_at: 2026-03-04T11:20Z
# model policy — evaluated at the seam
allowed_models: [ gpt-4o, claude-sonnet ]
blocked_models: [ legacy-finetune-v1 ]
max_risk_tier: high
require_approval_above_tier: mediumThe caller keeps the connection and the credentials. GovernorAI returns a verdict.
WHY IT IS DIFFERENT
Three things no proxy can say.
Most controls on inference arrive as a new hop: re-point your apps at a new hostname, hand over the provider keys, and route every model call through a new vendor. GovernorAI is built the other way round.
Consulted, not inserted
GovernorAI registers as a guardrail on the gateway that is already there and answers a question. No new hostname, no key handoff, no vendor between your gateway and your provider.
Portkey webhook · LiteLLM guardrail · Envoy ext_proc · forward-authFour decisions, one chain
A prompt that leaks an identifier, a completion carrying a planted instruction, and a tool call that writes to a system of record are one call, not three products. The same decision core decides all four, against the same policy, into the same record.
model → request → response → actionA refusal inside the gateway’s deadline
Most gateways default to allow when a guardrail times out. The GovernorAI check endpoint never returns a 5xx: a deny, a pending approval and an internal fault all land in-band as a refusal, inside a deadline shorter than the gateway's timeout. Set the gateway's own check to deny on failure too.
HTTP 200 + verdict:false · 2.5 s default deadlineIn the product
See inference governance in the console.
Recorded on a demo tenant: what each seam can inspect and enforce, an inline decision with the detector that fired, the gateway decision log, and the approved-model inventory.
Inference, step by step
Four screens from a recorded demo tenant.
WHERE IT SITS
Where it sits changes what you get.
An inline inference proxy and a consulted guardrail look similar on a slide. They are not similar in a key rotation, an audit, or an outage.
| Dimension | Inline proxy or AI firewall | GovernorAI |
|---|---|---|
| In a key rotation | ||
| Where it sits | In front of the provider; apps are re-pointed to a new hostname | Beside the gateway you already run; nothing is re-pointed |
| Your provider keys | Handed to the vendor | Stay in your gateway |
| Provider connection | Re-routed through the vendor, with the vendor's retries and timeouts | Stays in your gateway, with the routing and retries you already tuned |
| In an audit | ||
| What comes back | A flag or a score | A verdict — allow, deny, shape, or hold for approval — with the rule that fired and an evidence entry |
| What is governed | The prompt and the completion | The model registry, the prompt, the completion and the resulting tool call |
| Policy | The vendor's rules, in the vendor's console | Yours — GovernorAI native or OPA/Rego, including the OPA you already run |
| Record | Logs | A record of what was decided and what was executed, mapped to controls |
| In an outage | ||
| When the control itself fails | Often fails open | Fails closed: refusals arrive in-band, inside the gateway's timeout, so a fault never reads as a pass |
THE DECISION CHAIN
Four decisions, in order.
The risk moves along the call. So does the decision.
Which models may be called
Every model is a registry row with a status and an approver. Models found by Shadow AI discovery land as pending review — governed before they are approved.
status · risk_tier · approved_byWhat may be sent
The prompt is a governed surface. Secrets are redacted, regulated identifiers masked, and prompt injection is denied outright because it cannot be safely rewritten.
llm.chat.requestWhat may come back
The completion is inspected on the way back. Output of a governed retrieval call can be declared untrusted and scanned as a payload, not read as an answer.
llm.chat.responseWhat may then happen
What the model said becomes a tool call. The same core decides it: allow, deny, hold for a human, or narrow the arguments and re-verify them before dispatch.
gateway/executeWhy one chain. The registry is the record of approval; the call is where a model is actually refused. Every usage record carries the decision that was reached, and the agent-to-model link is a declared association, not a guess from recent traffic — so a quiet agent never silently drops out of the governance graph.
gpt-4o is an approved registry row — risk tier 2, approved by the model review board.
“Refund order 8841 for card ending 4417…” — the card number is masked before the prompt leaves the gateway.
The completion is clean. The model proposes a $12,400 refund and a refund.create call.
Policy requires finance approval above $10,000. The call is held and not dispatched. A named approver's decision is recorded; it does not release the call.
DEPLOYMENT
A guardrail entry in the config you already maintain.
Nothing about the model call moves. The key stays where it is, the model list stays where it is, and GovernorAI is reachable at an address the gateway calls.
- PortkeyNow Prisma AIRS AI Gateway. Bring-your-own-guardrail webhook on the before- and after-request hooks. Config only; no code on the Portkey side.
- LiteLLMGuardrail wired at pre-call, during-call and post-call. Blocks on an outage unless you deliberately set fail-open.
- Envoy familyExternal-processing filter for Envoy Gateway, Istio, Gloo, Envoy AI Gateway and agentgateway. Redaction applied as a body mutation.
- Forward-authNGINX auth_request, Traefik, Apigee, Azure APIM, Lambda authorizer. Allow or deny; shaping is flagged, not silently skipped.
- Native seamsAWS Bedrock, Azure, Google Cloud, MCP, and the SDK wrapper — the same decision core, declared seam by seam.
LiteLLM · config.yaml
# the model list is untouched — your key stays yours model_list: - model_name: gpt-4o litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY guardrails: # govern the INPUT before the model call - guardrail_name: "governorai-pre" litellm_params: guardrail: governor_guardrail.GovernorAIGuardrail mode: "pre_call" api_base: os.environ/GOVERNOR_LLMGUARD_URL fail_open: false # fail CLOSED on outage # govern a fully buffered response; # streamed responses are recorded post-call - guardrail_name: "governorai-post" litellm_params: guardrail: governor_guardrail.GovernorAIGuardrail mode: "post_call" api_base: os.environ/GOVERNOR_LLMGUARD_URL fail_open: false
BY TEAM
What each team gets.
Security
Per-call authorization on the model path, with pause-for-approval and a deny that holds when the control itself is unreachable.
Security use casesPlatform & AI engineering
No re-pointing, no new hostname, one config entry on the gateway. Write policy in GovernorAI's native engine or OPA/Rego — or keep the OPA you already run.
Platform use casesGRC
Each recorded decision — the policy that applied, whether approval was required, and what was actually executed — in one record. Declare a surface out of scope and attest from the record that no decision touched it.
Evidence & complianceBring your gateway config
See it against your gateway.
Bring your Portkey, LiteLLM or Envoy config to a 30-minute briefing, or run the quickstart yourself. Design partners get a six-week evaluation.