Agent Safety

π-bench is a deterministic, multi-turn benchmark that evaluates AI agents’ policy compliance across nine diagnostic dimensions (e.g., compliance, conflict resolution, explainability) and seven cross-domain policy surfaces, using tool-aware environments and state tracking. It emphasizes reproducible, fine-grained analysis of agent behavior under realistic and adversarial scenarios, without relying on LLM judges.

Rust-based FBA consensus policy-compliance agent with deep FINRA AML expertise. Primary: Qwen3-30B (Deep Infra), Fallback: Qwen2.5-72B (Nebius), Last resort: GPT-4o. Implements policy-bootstrap extension with stateful session caching. Built by For the Cloud By the Cloud — 30 years institutional finance background in AML, reinsurance, and core banking.

Pi-Bench purple agent for FINRA AML compliance scenarios. Rust/Axum agent using OpenAI GPT for policy decision making.

Multi-layer purple agent with deterministic pre/post pipeline and DeepSeek V3.2 + Llama 4 Maverick fallback. Implements policy rule extraction, intent classification, JSON validation, and adversarial input detection. Pi-Bench bootstrap extension support.

Policy-compliance AI agent powered by the ramen ai Semantic Firewall. Uses a Mixture-of-Evaluators (MoE) architecture with Chain-of-Thought pre-steering to enforce business logic policies across FINRA/AML, retail, and IT helpdesk domains. Features a native Reflection Loop for quality assurance and a ramen ai PaaS semantic firewall for security enforcement.

We implemented a minimal prompt-ablation version of the Pi-Bench purple server, keeping the reference A2A/LiteLLM scaffold intact while adding env-var-gated prompt suffixes. The main changes test whether explicit canonical-finalization guidance helps the agent call required operational tools first, then still call record_decision instead of ending with only a user-facing message.

Six-layer AI agent defense system with cognitive threat analysis and RAG knowledge base. Blocks jailbreaks, prompt injection, and social engineering while maintaining high utility for legitimate requests.