guardrails-api

Purpose

guardrails-api is the guardrails-plane orchestrator. Receives a text (or structured payload) from openbox-core, fans out to 3 CPU-based ML model hosts (PII / NSFW / toxicity), aggregates results, and returns a verdict. Optionally calls llamafirewall-server for deep prompt-injection detection (GPU-backed).

Plane: Guardrails plane.

Ports & endpoints

Port	Protocol	Purpose	Exposed via
8000	HTTP/REST	Guardrails API (/v1/scan, /v1/verdict)	Istio Gateway (optional; typically cluster-internal only)
8000/health	HTTP	Liveness	K8s kubelet
9464	HTTP	Prometheus scrape	Cluster-internal

Typical request: POST /v1/scan with JSON body {"text": "..."} → response includes verdicts per model.

Runtime

Property	Value
Container count per pod	1
Resource requests	500m CPU / 1 Gi RAM
Resource limits	1 vCPU / 2 Gi RAM
HPA min → max replicas	2 → 10 (most-scaled service)
HPA target	CPU 60%
Node placement	Dedicated guardrails node group in prod (c7i.xlarge in reference); general CPU pool acceptable
StatefulSet?	No

Why min 2? guardrails-api sits on the hot path for every agent action — single-pod restart = brief agent stall. Prod baseline is 2, scales up to 10 under bursts.

Configuration

Chart values keys (top-level guardrails-api.*)

Key	Purpose
image.repository / image.tag	Container image
autoscaling.{minReplicas,maxReplicas,targetCPUUtilizationPercentage}	HPA — do not lower minReplicas below 2 in prod
resources	Requests/limits
llamafirewall.enabled	Toggle deep prompt-injection detector (requires GPU node + vllm)
llamafirewall.vllm.gpu	GPU resource type (default nvidia.com/gpu: 1)
models.pii.enabled / .nsfw.enabled / .toxicity.enabled	Toggle individual classifiers

Runtime env vars

• MODEL_HOST_PII_URL — points at model-host-detect-pii.<ns>.svc:8011
• MODEL_HOST_NSFW_URL — points at model-host-nsfw.<ns>.svc:8014
• MODEL_HOST_TOXICITY_URL — points at model-host-toxicity.<ns>.svc:8013
• LLAMAFIREWALL_URL — llamafirewall-server.<ns>.svc:8000 (only if enabled)

Dependencies

Downstream

Target	Protocol	Purpose
model-host-detect-pii	HTTP :8011	PII detection (CPU)
model-host-nsfw	HTTP :8014	NSFW classification (CPU)
model-host-toxicity	HTTP :8013	Toxicity classification (CPU)
llamafirewall-server	HTTP :8000	Deep prompt-injection (optional; forwards to vllm on GPU)

Upstream

• openbox-core-server — primary caller (via internal Service DNS)
• openbox-backend — occasional direct calls for admin scans

Deployment guidelines

Prod

• minReplicas: 2 non-negotiable — hot path for agent actions
• Isolate on guardrails node group (taint workload-line=guardrails) if you follow the prod reference NodePool layout
• Enable llamafirewall only if you have GPU capacity (single g5.xlarge handles most workloads)
• Add PodDisruptionBudget with minAvailable: 1

GPU-free deployments

If your cluster has no GPU nodes:

guardrails-api:
  llamafirewall:
    enabled: false

Consequence: PII/NSFW/toxicity classifiers still work; deep prompt-injection detection is unavailable. Callers should not fail-closed if guardrails-api returns a partial verdict — check response schema.

Local PoC

• autoscaling.minReplicas: 1, maxReplicas: 1
• Disable llamafirewall (laptop has no GPU)
• Model hosts each need ~3.5 Gi RAM — total footprint ~11 Gi for the guardrails plane. Ensure Minikube has enough (--memory=12g).

Troubleshooting

HPA panics under sudden traffic burst → Default HPA uses 2-min stabilization window. Pre-warm before load tests: temporarily set minReplicas: 5 before ramping traffic.

One model host down but scans still returning "ok" → guardrails-api does NOT fail-closed on individual model-host failures. Response includes verdict.partial: true — callers must check. Fix the failing model host (usually OOM — bump resources.limits.memory to 4 Gi).

All requests slow (~2000 ms) — llamafirewall path → vllm cold-start / model reload. First-request-after-restart takes ~10-20 sec while Qwen3-8B loads into GPU. Pre-warm with a curl on pod startup.

503 no available model hosts → All 3 model-host Services have no Ready endpoints. kubectl get pods -l app=model-host-* and check crashloop reasons. Common: OOM (model weights need 3.5 Gi guaranteed).

Guardrails endpoints unreachable from openbox-core → Service DNS misresolution. Verify kubectl get svc guardrails-api -n openbox. If exposed via Istio, check namespace has istio-injection=enabled.