How AxonLLM is built

Request Pipeline

What happens to a request

The normal non-streaming path through GatewayAgent.handle_chat_completion is ordered so authority and admission run before provider spend, accounting runs immediately after the provider result, and output policy runs before release. Streaming uses the same controls with the lifecycle differences called out below.

1. Parse the request and extract context — project, user, and model resolved from the token or API key
2. Validate — skipped for smart and ensemble routing, where the model is not yet chosen
3. Enforce quotas and cap max_tokens — the policy hierarchy resolves org → business unit → project → env
4. Detect prompt injection — NFKD normalization and homoglyph-resistant patterns; audited and dispatched even when allowed
5. Redact PII — replaced with tokens, and the mapping held for step 11.5
6. Rate limit — sliding window per project and user
7. Check model access — project allow-list intersected with the user's
8. Check budgets — project and user, tightest limit wins
9. Apply request guardrails
10. Look up the cache — exact key first, then a semantic match on the reworded question; a hit returns here and never reaches a provider
11. Route by region — hub to spoke, honouring data-residency zones; skipped in single-region deploys
12. Route and execute — direct with a fallback chain, smart (classify → leaderboard → cost/quality), or ensemble (panel → quorum → synthesis)
13. Finalize usage, cost, and reserved spend — provider usage is durable before output policy can withhold a response
14. Apply response guardrails — only block changes execution today; warn/redact rules record matches
15. Re-inject PII — the original values restored into the response, per chunk when streaming
16. Append audit metadata — tenant SHA-256 hash chain without prompt or response bodies
17. Budget status and session storage — recorded after the response is complete
18. Stream, if asked — eligible routes relay native chunks; output inspection buffers, and other routes can emit simulated chunks
19. Write the cache — only eligible non-streaming, post-policy responses are stored
20. Return — JSON, with routing and cache metadata attached

Infrastructure

AWS services it deploys onto

The production topology separates the AgentCore data plane from a private Fargate web control plane. They share canonical DynamoDB authority but have different routes and IAM permissions.

Components

What's inside the gateway

Wired in one place — build_gateway_components() in src/gateway/bootstrap.py. Every component takes its collaborators as constructor arguments, which is what makes the persistence layer optional rather than assumed.