Skip to main content

Deployment Steps

The operator-side walkthrough, in order. Every step is idempotent — a failed or interrupted run resumes by re-running the same command.

0. Preconditions​

  • Every item in Client Prerequisites confirmed in writing
  • The hand-over summary values in hand
  • Local tooling: Node 20+, npm, AWS CLI v2, kubectl, credentials for the client's deploy role (aws sts get-caller-identity shows the client account)

1. Install and bootstrap​

cd infra-cdk
npm ci
npx cdk bootstrap aws://<CLIENT_ACCOUNT_ID>/<REGION>

Bootstrap runs once per account/region, creates the CDK staging bucket/roles, and respects the client's permissions boundary if the config names one. If it errors about placeholder config values, fill config/client.json first (next step) — the app synthesizes during bootstrap.

2. Fill and validate the config​

Copy the template and fill every field from the hand-over summary — field-by-field meaning in Config Reference:

cp config/client.example.json config/client.json
# edit config/client.json
npm run synth

synth runs the full validation pass and aggregates every problem in one output — placeholder values, malformed ARNs, unsupported region/version, missing license fields, subnet/AZ rules, the reCAPTCHA keys-or-opt-out rule. Fix until it synthesizes clean. Nothing has touched the client account yet.

The template matches the bundle's chart: a Marketplace bundle's example already sets "chartSource": "marketplace", has an admin-email slot and no registry-token field. If synth reports "chartSource is '…' but this bundle ships the … chart", the config and the bundle disagree — set chartSource to the bundle's chart (also printed in the bundle's BUNDLE.txt) or get the other bundle.

3. Deploy the three stacks​

npm run deploy

This deploys Openbox-Preflight → Openbox-Eks → Openbox-Workloads in order with the right flags baked in. Always use the npm scripts, never raw cdk deploy — the scripts pin --require-approval never (a TTY-less raw deploy exits 0 at the IAM approval prompt without deploying) and --no-rollback on the workloads stack (a mid-install rollback would tear down the helm release while a resume needs it in place).

Expected timings:

StackTimeWhat happens
Preflight~2 minValidates client networking against account reality (subnet AZs, SG ownership)
Eks~20 minEKS Auto Mode cluster, node pools, access entries, namespace
Workloads~25 mink8s Secrets from your ASM, ACM cert (DNS-validated), ingress-nginx + internal NLB, OpenBox chart, 5 Route53 records

A green deploy means the release was accepted, not that every pod is ready — convergence is the next step.

4. Watch convergence​

Grab the stack outputs (also printed at the end of the deploy):

aws cloudformation describe-stacks --stack-name Openbox-Workloads \
  --query 'Stacks[0].Outputs' --output table
aws eks update-kubeconfig --name openbox --region <REGION> --role-arn <ADMIN_ROLE_ARN>
kubectl -n openbox get pods -w

Converged state (~10 min after deploy, GPU node last):

  • openbox-backend-service, openbox-core-service + 4 core worker deployments, openbox-fe-service, identity-service, opa-* — all 1/1 Running
  • openbox-postgresql-0, openbox-core-redis-0, Temporal pods — Running
  • guardrails-api + 3 model-host-* on the guardrails pool
  • llamafirewall + vllm — vLLM waits for the g5.xlarge GPU node; if it sits Pending longer than ~15 min, check the GPU quota

5. Post-deploy wiring (only if the CDK created the KMS key)​

The BackendKmsKeyArn output goes into the client's backend env-file secret, then a nonce redeploy re-applies it:

# client updates their backend env secret: KMS_SYMMETRIC_KEY_ID=<arn>, KMS_REGION=<region>
npm run deploy:workloads -- -c rotationNonce=$(date +%s)
kubectl -n openbox rollout restart deploy/openbox-backend-service

The same nonce redeploy is the standing mechanism for any env-file change — the pods re-read /app/.env on restart.

6. Verify the endpoints​

From a machine that can reach the internal NLB (VPN/peering — see Common Pitfalls):

curl -s https://openbox-api.<domain>/health          # {"status":200,...}
open  https://openbox-eks.<domain>                    # frontend loads, register/login works

Seeded org admin (only when seedOrgEmail was configured). The password is not pre-expired — sign in with it, then change it:

kubectl -n openbox get secret openbox-seed-admin -o jsonpath='{.data.password}' | base64 -d

Seeding runs as an init container on the backend, so a failure holds that pod in Init rather than leaving a running install with no organization. If the backend never becomes ready:

kubectl -n openbox logs -l app.kubernetes.io/name=openbox-backend-service -c create-organization

Keycloak admin password (generated in-cluster, never leaves it):

kubectl -n openbox get secret openbox-gen-keycloak \
  -o jsonpath='{.data.KC_BOOTSTRAP_ADMIN_PASSWORD}' | base64 -d

Functional smoke: register an organization on the frontend (exercises reCAPTCHA + backend CORS + Keycloak), confirm the invitation email arrives (exercises SES).

7. Handover​

Send the client: the five URLs, the Keycloak admin retrieval command, the KMS key ARN (if created), and the access-path decision they made (VPN route / peering / public ingress). Passwords for bundled Postgres/Redis/Keycloak stay in-cluster (openbox-gen-* Secrets).

Enabling the executor (optional, two-step)​

The voice-agent executor ships disabled: its governance credentials (an obx_key_… API key plus agent DID/keypair) are minted inside the product, so they cannot exist at install time. After the org exists: create the openbox-executor-governance Secret (command in the subchart values), then --set openbox-executor.enabled=true and redeploy. Until credentials are wired the executor refuses turns (fail_closed) rather than silently allowing them.

Re-running and updating​

  • Config or chart change → npm run deploy (or deploy:workloads for app-only changes). Deploys are idempotent.
  • Secret value rotated in ASM → nonce redeploy (step 5) + restart of the consuming deployment.
  • Image bump → change the tag in values, npm run deploy:workloads.

Teardown​

npx cdk destroy Openbox-Workloads Openbox-Eks Openbox-Preflight --force removes everything the CDK created except the backend KMS key (retained by design — schedule deletion separately, 7-day window). Client-provided resources (VPC, zone, secrets) are never touched. Check afterwards for orphaned EBS volumes from PVCs (aws ec2 describe-volumes --filters Name=status,Values=available).

Last updated on