Skip to main content

Common Pitfalls

Everything below was hit (and solved) in real deployments. Skim this before the deployment window.

GPU quota is 0 in fresh accounts​

Symptom: every pod is green except vLLM, which sits Pending; the LlamaFirewall server loops on readiness. Cause: "Running On-Demand G and VT instances" quota is 0 by default in many accounts, so Auto Mode cannot launch the g5.xlarge node. Fix: request the quota increase days in advance (see prerequisites §2). Nothing else in the install blocks on it — the GPU workload joins when capacity appears.

DNS delegation missing → certificate stalls​

Symptom: deployment sits at the ACM certificate step for 30+ minutes. Cause: the Route53 zone's NS records were never added at the parent domain, so ACM's DNS validation records aren't publicly resolvable. Check before the window: dig NS <your-zone> must return the four awsdns name servers.

Reaching an internal NLB​

The default ingress is an internal NLB — the service URLs resolve publicly but to private IPs in your VPC. Browsers outside the VPC time out; that's by design, not a fault. Your options, pick one before user testing starts:

  1. Corporate VPN / Client VPN with a route to the VPC (VPC peering + route table entries + VPN route/authorization for the VPC CIDR). Split-tunnel VPN clients must reconnect after routes change — routes are pushed at connect time.
  2. VPC peering from an existing internal network.
  3. ingressPublic: true at deploy time (requires ≥2 public subnets) — then restrict by source ranges.

SES sandbox mode​

Even with SES configured, a sandboxed SES account only delivers to verified recipients. Symptoms look like "invitation emails silently lost". Request SES production access early — approval can take a day. No SES at all → install succeeds, email-dependent flows fail (see prerequisites §7).

reCAPTCHA domain allow-list​

The site key works only on domains listed in the Google reCAPTCHA admin console. The widget renders regardless — the failure appears at verification time on form submit. Add your OpenBox domain (or its parent) to the key before user testing.

First-install database poisoning​

Symptom: after a failed or interrupted first install, every retry crash-loops on Postgres. Cause: a half-initialized Postgres volume persists across reinstalls; the bundled database refuses to re-init a non-empty data directory. Fix (operator runbook): uninstall the release without hooks, delete the Postgres PVC, redeploy. Client action: none — but expect the deployment window to extend ~15 minutes if it happens.

Pinned image tags vs. branch tags​

Production installs must run main-<sha> image tags. Branch-built tags (feat-…) are used only during joint debugging and are pruned by registry retention — never leave an install running on one after the session ends.

Kubernetes version support window​

The CDK supports minors 1.31–1.36 (kubectl tooling is version-locked). Asking for anything else fails config validation immediately — it's a supported-matrix decision, not a flag to flip.

Region availability​

EKS Auto Mode is not in every region. The config validator enforces the allow-list in prerequisites §1; deploying to an unlisted region is a vendor decision (support matrix + testing), not a config override.

Cluster API endpoint mode​

clusterEndpointPublicAccess: false (private-only) means all deploy and debug traffic must originate inside your network — the operator needs VPN/bastion access arranged before the window, or the deployment cannot run at all. The default (public+private) avoids this; the API is still IAM-authenticated.

Executor is disabled by default — and that's correct​

openbox-executor.enabled=true before its governance Secret exists gives you a pod that starts but refuses every turn (fail_closed). The obx_key_ credential it needs is minted in-product after the org is seeded — enablement is inherently two-step (see Deployment Steps). Also note its inbound caller (ElevenLabs cloud) needs ingressPublic: true; on an internal NLB the endpoint is VPN-only.

Preflight rejects the VPC (subnet tags / DNS hostnames)​

Symptom: the Openbox-Preflight stack fails immediately with openbox preflight failed (N issues) naming missing kubernetes.io/role/internal-elb=1 tags or enableDnsHostnames=false. Cause: a hand-built VPC. aws ec2 create-vpc leaves enableDnsHostnames=false, and subnet role tags are never automatic. Fix: apply the commands in prerequisites §3, then re-run the deploy. This is the preflight gate doing its job — it fails in ~2 minutes instead of letting the cluster come up and the ingress silently never get an address 25 minutes later. Note the stack must be deleted before re-running (a failed CREATE leaves ROLLBACK_COMPLETE, which CloudFormation cannot update): npx cdk destroy Openbox-Preflight --force.

The allowlist needs your PUBLIC egress IP, not your internal network​

Asked for "your network CIDR", people reach for their internal range — 10.x, 172.16-31.x, 192.168.x. On an internet-facing load balancer that matches nothing: AWS accepts the rule, it never fires, and everyone is locked out of what looks like a broken deployment. The validator rejects private, CGNAT, loopback and link-local ranges for exactly this reason.

Find the right value from inside your network:

curl -s https://checkip.amazonaws.com

Ask your network team for the full egress range, not just the address you observed. Larger offices egress through several NAT addresses; allowlisting one produces intermittent failures for colleagues — the hardest kind of problem to diagnose remotely.

Locked out of your own deployment​

Removing your IP from allowedCidrs cuts your access. Recovery:

  1. Correct allowedCidrs and redeploy — the proper fix, ~3-5 minutes.
  2. Emergency stopgap: kubectl -n ingress-nginx patch svc ingress-nginx-controller -p '{"spec":{"loadBalancerSourceRanges":["<your-ip>/32"]}}' — effective in seconds, but the next deploy reverts it.

Do NOT edit the security group in the AWS console. That SG is owned by the EKS load-balancer controller (tagged ManagedLBSecurityGroup); it reconciles your change away, so the access appears to work and then silently disappears minutes later. Config is the source of truth, then Kubernetes, then AWS — each layer overwrites the one below.

Changing ingressPublic replaces the load balancer​

Flipping between public and internal does not reconfigure the existing NLB — AWS creates a brand-new one with a different hostname and deletes the old one. Expect:

  • A few minutes of downtime on every openbox hostname while the new NLB provisions and its targets become healthy. Schedule the flip accordingly.
  • A DNS reconverge. The deployment rewrites the CNAMEs to the new load balancer automatically; records carry a 5-minute TTL, so allow that long for resolvers to catch up.
  • A new security group. Any rule added by hand to the old one is gone. This is another reason the allowlist belongs in allowedCidrs and nowhere else.

If hostnames still fail well after the deployment finishes, check what the CNAME actually points at and compare it to the live load balancer:

dig +short openbox-api.<your-domain>
aws elbv2 describe-load-balancers --query 'LoadBalancers[].DNSName' --output text

A CNAME pointing at a load balancer that no longer exists means DNS did not follow the replacement — report it rather than editing Route53 by hand, since the next deployment would overwrite a manual fix.

Public ingress replaces the internal path — VPN users are affected​

There is only ever one load balancer. With ingressPublic: true it is internet-facing, so the internal-only entry point no longer exists, and staff on a VPN no longer reach the application through the tunnel. Their traffic goes out their normal internet connection instead, arriving from whatever public address their office or ISP uses — and that address must be in allowedCidrs or they are blocked like anyone else.

This surprises teams who assume "we're on the VPN" still grants access. Before flipping an existing install to public, collect the public egress address of every site that needs in. The VPN itself keeps working for everything else it does (database access, the Kubernetes API, administrative SSH); it just stops being the way into the application.

Choose ingressPublic: false instead if VPN-only access is what you actually want.

IP allowlists and roaming users​

This is solid for fixed offices and server-to-server callers, and poor for anyone on home broadband, mobile, or hotel wifi — their address changes. For roaming staff, either keep the install internal (ingressPublic: false) and let them in over the VPN, or route them through a gateway with a stable public address that you can allowlist. An allowlist alone does not serve users whose address moves.

Cost expectations while idle​

Even an idle install carries a floor cost: NAT gateway ($35–65/mo), the GPU node ($1/hr while LlamaFirewall is enabled), 2–3 general-purpose nodes, and the EKS control plane. If the environment is for evaluation, agree on a teardown or scale-down date up front.

Last updated on