Skip to main content

Resource limits & HPA

When to override chart defaults​

  • Cluster is under-provisioned and default requests would leave pods Pending
  • Cluster is over-provisioned and you want tighter bin-packing
  • Workload pattern differs from OpenBox reference (e.g., mostly-idle dev cluster)

Per-pod resources — copy-paste template​

openbox-backend:
  resources:
    requests: { cpu: 500m, memory: 1Gi }
    limits:   { cpu: 1500m, memory: 2Gi }
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 8
    targetCPUUtilizationPercentage: 70

openbox-core:
  # 5 containers per pod — total pod = sum of all containers
  server:
    resources:
      requests: { cpu: 200m, memory: 500Mi }
      limits:   { cpu: 1, memory: 2Gi }
  # Same shape for governance-worker, attestation-worker, observability-worker, scheduler
  autoscaling:
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 70

guardrails-api:
  resources:
    requests: { cpu: 500m, memory: 1Gi }
    limits:   { cpu: 1, memory: 2Gi }
  autoscaling:
    minReplicas: 2                # ≥ 2 for HA
    maxReplicas: 10
    targetCPUUtilizationPercentage: 60

model-host-detect-pii:
  resources:
    requests: { cpu: 1, memory: 3.5Gi }     # model weights loaded to memory
    limits:   { cpu: 2, memory: 4Gi }
  autoscaling:
    minReplicas: 1
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70

# Same shape for model-host-nsfw and model-host-toxicity

identity-service:
  resources:
    requests: { cpu: 200m, memory: 1700Mi }  # JVM heap
    limits:   { cpu: 500m, memory: 2Gi }
  autoscaling:
    minReplicas: 3                # ≥ 3 for Keycloak HA
    maxReplicas: 5

HPA gotchas​

  • HPA needs metrics-server — see S3 prerequisites
  • If HPA is off (autoscaling.enabled: false), chart uses replicaCount as static
  • Don't set replicas on the Deployment manually when HPA is on — HPA overrides it and creates churn

Node pool taints (advanced — optional)​

Reference production runs uses 5 tainted node pools (cpu, memory, database, llm, guardrails). You do NOT need this — HPA + generic node pool works fine. If you want isolation:

model-host-detect-pii:
  tolerations:
    - key: workload-line
      value: guardrails
      effect: NoSchedule
  nodeSelector:
    workload-line: guardrails

See prod snapshot: openbox-k8s-cluster-prod/nodepool/.

Down-sizing for dev/staging​

# values-dev.yaml
openbox-backend: { autoscaling: { minReplicas: 1, maxReplicas: 2 } }
openbox-core:    { autoscaling: { minReplicas: 1, maxReplicas: 2 } }
guardrails-api:  { autoscaling: { minReplicas: 1, maxReplicas: 2 } }
identity-service: { autoscaling: { minReplicas: 1, maxReplicas: 1 } }
model-host-nsfw:      { enabled: false }
model-host-toxicity:  { enabled: false }
guardrails-api:
  llamafirewall: { enabled: false }        # skip GPU

Result: 6-8 pods on a 3-node dev cluster ($400-600/mo AWS).