Skip to main content

Troubleshooting — S2 Fresh AWS

Symptom → cause → fix. Sorted by frequency observed in real deploys.

1. Pod stuck in ImagePullBackOff​

Symptom: kubectl get pods shows ImagePullBackOff.

Cause: Node can't reach the container registry, or IRSA on the node group doesn't include AmazonEC2ContainerRegistryReadOnly (if pulling from private ECR).

Fix:

kubectl describe pod <name> -n openbox | grep -A 5 Events
# Look for 401/403 vs DNS/network errors

# For private ECR:
aws iam attach-role-policy \
  --role-name <node-group-role> \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

# For public registries: check node egress (NAT gateway route, DNS resolution)

2. Pod CrashLoopBackOff — "no permission to decrypt with KMS key"​

Symptom: openbox-backend or openbox-core crashes with AccessDeniedException from KMS.

Cause: IRSA role missing kms:Decrypt on the specific CMK, OR the ServiceAccount annotation doesn't match Terraform output.

Fix:

# Verify SA has the IRSA annotation
kubectl get sa openbox-backend -n openbox -o yaml | grep eks.amazonaws.com/role-arn

# Should equal: terraform output backend_irsa_role_arn
# Fix by editing values-prod.yaml and helm upgrade

3. Aurora / RDS DNS unresolvable from pods​

Symptom: Pods log getaddrinfo ENOTFOUND <rds-endpoint>.

Cause: Pods are in a subnet that can't reach RDS security group / RDS VPC.

Fix: Verify RDS security group ingress allows the EKS node security group on port 5432. If EKS + RDS are in different VPCs, need VPC peering or Transit Gateway (out of Openbox scope — customer decision).

4. Elasticsearch OOMKilled (self-hosted Temporal variant)​

Symptom: elasticsearch-master pods get OOMKilled, Temporal frontend logs "no available nodes".

Cause: Chart default resources: {} doesn't reserve enough JVM heap; ES container evicted under load.

Fix: Pin resources in values-prod.yaml:

temporal:
  elasticsearch:
    replicas: 3
    resources:
      requests: { cpu: 500m, memory: 2Gi }
      limits:   { cpu: 1500m, memory: 2Gi }
    esJavaOpts: "-Xms1g -Xmx1g"

5. HPA reports <unknown> for target metric​

Symptom: kubectl get hpa -n openbox shows <unknown> in TARGETS column.

Cause: metrics-server addon not installed or not running.

Fix:

kubectl get deployment metrics-server -n kube-system
# If missing, install:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

6. cert-manager Certificate stuck in Pending​

Symptom: TLS not working; kubectl describe certificate shows waiting for ACME challenge.

Cause: DNS challenge can't reach the domain, OR HTTP-01 challenge blocked by cluster network policies.

Fix:

kubectl describe challenge -n openbox
# Follow the specific ACME error. Common: A record not yet propagated.

# Verify domain resolves to ingress:
dig +short openbox.your-domain.com
# Should match ingress ALB DNS.

7. Istio sidecar injection not happening​

Symptom: Pods have 1 container instead of 2 (no istio-proxy); mTLS not working.

Cause: Namespace missing istio-injection=enabled label.

Fix:

kubectl label namespace openbox istio-injection=enabled
kubectl rollout restart deployment -n openbox

8. Argo Rollouts CRD missing (chart canary strategy)​

Symptom: helm install fails with no matches for kind "Rollout" in version "argoproj.io/v1alpha1".

Cause: Argo Rollouts CRDs not installed and chart is configured for canary strategy.

Fix (option A — install Argo Rollouts):

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts \
  -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Fix (option B — disable canary in chart):

# values-prod.yaml
openbox-guardrails-service:
  rollout:
    enabled: false

9. helm upgrade --wait times out​

Symptom: Chart install hits the --timeout 15m limit; some pods still Pending.

Cause: Insufficient node capacity for HPA min replicas + guardrail model-hosts memory (each 3.5 Gi).

Fix:

kubectl describe node | grep -A 5 "Allocated resources"
# If close to 100%, either scale up node group OR reduce HPA minReplicas in values.

10. First login fails: "Cannot connect to identity service"​

Symptom: Backend UI redirects to Keycloak but page never loads.

Cause: Keycloak pod is still bootstrapping (JVM warmup + DB migrations, ~2-3 min on cold start) OR ingress not routing /auth/* to Keycloak service.

Fix:

kubectl logs deploy/identity-service -n openbox --tail 50
# Look for "Keycloak … started" line

kubectl get ingress -n openbox openbox -o yaml | grep -A 2 "path:"
# Verify /auth or / route points to identity-service:8080

Still stuck?​