Skip to content

Known Issues & Fixes

Consolidated catalog of bugs hit running Palette VerteX 4.8.40 with EKS as the platform — both the self-hosted management/bootstrap cluster and the workload (spoke) clusters provisioned through it. Every issue below is specific to the airgap + FIPS + commercial AWS combination; they are unlikely to apply to non-airgap or non-FIPS deployments.

The management-cluster install is autonomous (see Air-Gapped Install on EKS). Workload-cluster provisioning is the manual-heavy flow where nearly all of these surface. Most fixes are now codified in the PCG-resident vertex-patches-controller.

Note: Issues surface back-to-back — fix one and the next appears. The full chain must be codified so redeploys and cluster creates work first time. Each entry below uses a Symptom → Root cause → Fix → Recognition signal structure.

Contents

Create-Path Blockers

Delete-Path Blockers

Self-Hosted PCG Gotchas

Patch Coverage

Create-Path Blockers

Everything that prevents a new workload cluster from reaching Running/green.

Pack values don't propagate to the AWSManagedControlPlane CR

Symptom: Values set in the kubernetes-eks pack (managedControlPlane.kubeProxy.disable: false, accessConfig.authenticationMode: API_AND_CONFIG_MAP) are silently ignored — the generated AWSManagedControlPlane CR is born with the CAPA defaults (kubeProxy.disable: true, CONFIG_MAP-only auth).

Root cause: Spectro's kubernetes-eks pack template does not carry these fields from values.yaml onto the CR it generates. Confirmed across multiple test runs — disableAssociateOIDCProvider does propagate, but kubeProxy.disable and accessConfig.authenticationMode do not (2 of 3 broken). Downstream: with kubeProxy.disable: true, CAPA deletes the kube-proxy DaemonSet and Cilium (which ships kube-proxy-replacement: false) can never reach 10.96.0.1; with CONFIG_MAP-only auth, the bastion/SSO roles are locked out post-create.

Fix: The controller patches the live CR as defense-in-depth — ensure_awsmcp_kubeproxy_enabled (sets spec.kubeProxy.disable: false) and ensure_awsmcp_access_mode (flips to API_AND_CONFIG_MAP). These calls are load-bearing, not redundant — the profile-side fix alone does not work.

Recognition signal: Profile shows the correct value but kubectl get awsmanagedcontrolplane -n cluster-<id> <name>-cp -o jsonpath='{.spec.kubeProxy}' returns {"disable":true}; or aws eks list-access-entries fails with authentication mode must be set to one of [API, API_AND_CONFIG_MAP].

disableAssociateOIDCProvider not set, CAPA loops on OIDC discovery

Symptom: After the EKS control plane reaches ACTIVE, CAPA retries every ~2s with failed reconciling OIDC provider ... lookup oidc.eks.us-east-1.amazonaws.com ... no such host; the CR stays ControlPlaneReady=False and no node group is ever created.

Root cause: AWSManagedControlPlane.spec.associateOIDCProvider defaults to true, so CAPA tries to fetch the OIDC discovery document from oidc.eks.us-east-1.amazonaws.com — which has no VPC endpoint and does not resolve in airgap. This is a different field from the UI's "OIDC Identity Provider" toggle (that controls spec.identityProviderConfig, an external user-auth IdP — unrelated to IRSA).

Fix: Uncomment managedControlPlane.disableAssociateOIDCProvider: true in the kubernetes-eks pack values (note the inversion: pack field disableAssociateOIDCProvider: true → CR field associateOIDCProvider: false). This one does propagate. For in-flight clusters: kubectl patch awsmanagedcontrolplane <name>-cp -n cluster-<id> --type=merge -p '{"spec":{"associateOIDCProvider":false}}'.

Recognition signal: CAPA log shows failed reconciling OIDC provider with an oidc.eks.us-east-1.amazonaws.com DNS lookup failure.

CAPA deletes the kube-proxy addon (empty spec.addons[])

Symptom: Even with kubeProxy.disable: false, the EKS-managed kube-proxy addon is deleted within ~2 minutes of creation, repeatedly (observed reinstalled 4× in 14 min). Cilium's init container then hangs on dial tcp 10.96.0.1:443: i/o timeout.

Root cause: CAPA's AWSMCP addon reconciler is a separate code path from kubeProxy.disable. It reads spec.addons[] and deletes any EKS managed addon not in that list — the default empty list means kube-proxy gets wiped every reconcile.

Fix: Two parts. (1) ensure_kube_proxy_addon reinstalls the addon out-of-band (aws eks create-addon --addon-name kube-proxy --resolve-conflicts OVERWRITE). (2) The real fix — ensure_awsmcp_kubeproxy_in_addons_list patches spec.addons to [{"name":"kube-proxy","version":"<matching>","conflictResolution":"overwrite"}] so CAPA owns the addon and stops deleting it. The version field is mandatory — an empty version makes CAPA call UpdateAddon with a blank addonVersion and AWS rejects it (Addon version specified is not supported); look it up via aws eks describe-addon-versions. With the addons-list patch, whack-a-mole drops to zero.

Recognition signal: aws eks describe-addon --cluster-name <name> --addon-name kube-proxy flips ACTIVE → gone → ACTIVE on a ~60-150s cycle; CAPA logs Kube-proxy has been removed from the cluster.

palette-controller-manager CloudFormation hang (missing non-FIPS CFN VPC endpoint)

Symptom: Cluster create stalls at aws_presetup.go:137 "Attempting to reconcile CloudFormation stack" for 6+ min with zero pod CPU (blocked, not retrying); or errors Post "https://cloudformation.us-east-1.amazonaws.com/": dial tcp ...: i/o timeout.

Root cause: Spectro's clusterawsadm/palette-controller-manager uses AWS SDK Go v1, which builds the non-FIPS CloudFormation endpoint by default. The airgap VPC only had the cloudformation-fips VPC endpoint, so the non-FIPS hostname resolved to an unroutable public IP. AWS_ENDPOINT_URL_CLOUDFORMATION is not honored by SDK v1 — only v2 reads per-service URL overrides.

Fix: Two options, both in play. (1) Create the non-FIPS CloudFormation Interface VPC endpoint (com.amazonaws.us-east-1.cloudformation, private_dns_enabled = true) in terraform — the only create-path fix requiring terraform. (2) Interim env-var route: AWS_USE_FIPS_ENDPOINT=true (respected by both SDK versions) sends CFN to the cfn-fips endpoint; safe because the spectro-drive binary patch rewrites fips.eks→eks and EC2 hostnames at runtime so EKS/EC2 still land correctly. Controller handler: patch_palette_cm_envs.

Recognition signal: palette-controller-manager logs stop at Attempting to reconcile CloudFormation stack for >2 min with no error.

Missing non-FIPS Auto Scaling VPC endpoint, MachinePool stuck Failed

Symptom: Cluster is functionally healthy (kubectl works, pods Running) but the VerteX UI shows Worker nodes - LaunchWorkerNode forever. AwsManagedMachinePool.status.replicas is empty; condition shows Post "https://autoscaling.us-east-1.amazonaws.com/": dial tcp ...: i/o timeout.

Root cause: CAPA calls DescribeAutoScalingGroups against the non-FIPS regional autoscaling endpoint to populate status.replicas. Only the autoscaling-fips VPC endpoint existed, so the non-FIPS hostname timed out. Unlike EKS, CAPA's autoscaling client honors no AWS_ENDPOINT_URL_* override (AWS_ENDPOINT_URL_AUTOSCALING and AWS_ENDPOINT_URL_AUTO_SCALING both ignored).

Fix: Add the non-FIPS Auto Scaling Interface VPC endpoint (com.amazonaws.us-east-1.autoscaling, private_dns_enabled = true) in terraform. CAPA's default call then resolves in-VPC and status.replicas populates.

Recognition signal: functional cluster, UI stuck at LaunchWorkerNode, describe awsmanagedmachinepool shows an autoscaling i/o timeout condition.

CAPA FIPS-endpoint failures (iam-fips timeout, fips.eks x509 mismatch)

Symptom: AWSManagedControlPlane conditions IAMControlPlaneRolesReady=False (Post "https://iam-fips.amazonaws.com/": i/o timeout) and EKSControlPlaneReady=False (fips.eks.us-east-1.amazonaws.com: x509: certificate is valid for ... eks.us-east-1.amazonaws.com, NOT fips.eks.us-east-1.amazonaws.com).

Root cause: Spectro's FIPS-mode SDK builds call iam-fips/fips.eks by name. iam-fips has no route from private subnets; and even after CoreDNS points fips.eks.us-east-1.amazonaws.com at the EKS VPC endpoint, that endpoint's TLS cert covers only eks.us-east-1.amazonaws.com (and *.eks.us-east-1.vpce.amazonaws.com) — not the FIPS hostname. Post-pivot CAPA is a fresh deployment on the workload cluster, so bastion-side patchers never touched it.

Fix: Set on both pre-pivot (PCG) and post-pivot (workload) CAPA: AWS_USE_FIPS_ENDPOINT=false + AWS_ENDPOINT_URL_EKS=https://eks.us-east-1.amazonaws.com (+ AWS_STS_REGIONAL_ENDPOINTS=regional). The non-FIPS EKS hostname resolves via VPC-endpoint Private DNS and the cert matches — no security downgrade, KMS/encryption is identical. Controller handlers: patch_capa_envs + ensure_workload_capa_envs. Also requires workload-cluster CoreDNS FIPS rewrites (ensure_workload_coredns).

Recognition signal: AWSMCP conditions reference iam-fips or fips.eks...x509 in their messages.

PCG palette-controller-manager iterates all AWS regions (cross-region DNS hang)

Symptom: palette-controller-manager on PCG hangs at the CloudFormation reconcile even after the non-FIPS CFN endpoint exists; /proc/net/tcp shows outbound SYN_SENT to rotating public AWS IPs across many regions.

Root cause: CAPA's aws_presetup.go iterates all AWS regions for bootstrap-stack discovery, issuing DNS queries for cloudformation.<region>.amazonaws.com (saw ca-central-1, sa-east-1, eu-west-3, …). VPC endpoints only cover us-east-1; foreign-region hostnames resolve to public IPs that SYN-and-hang in airgap. No flag controls this — it is hardcoded.

Fix: Patch PCG CoreDNS to return NXDOMAIN for any non-us-east-1 regional hostname, with a more-specific us-east-1.amazonaws.com server block allowlisting the home region (longest-suffix match wins). The template plugin block must include fallthrough — without it, global hostnames like iam.amazonaws.com return SERVFAIL instead of forwarding. Codified as ensure_pcg_coredns.

Recognition signal: palette-cm blocked at CFN reconcile; tcpdump on UDP/53 shows queries for multiple non-us-east-1 regions.

cert-manager excluded from imageswap NotIn scope

Symptom: Workload cluster stuck at "Provisioning / Preparation"; palette-controller-manager silent after a startup burst; cert-manager namespace pods in ImagePullBackOff pulling us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/....

Root cause: palette-controller-manager runs ensure-cert-manager and creates cert-manager Deployments using upstream public-registry paths. The imageswap mutating webhook would rewrite them, but the bootstrap codification had explicitly added cert-manager to the webhook's namespaceSelector NotIn exclusion list — so those pods were never rewritten and palette blocked forever waiting for the rollout.

Fix: Remove cert-manager from the NotIn list (it was unnecessary defensive code — imageswap maps us-docker.pkg.dev → ECR to the same path bootstrap pods already use). Temp: kubectl patch mutatingwebhookconfiguration imageswap-webhook to replace the NotIn values with [kube-system, kube-public, kube-node-lease, imageswap-system], then delete the stuck cert-manager pods. On workload clusters where cert-manager isn't a profile pack, palette falls back to the built-in vendorcrd install (unmirrored paths) — the durable fix is to add cert-manager as an explicit profile pack with ECR-rewritten registries and recreate the cluster.

Recognition signal: palette-cm Running but no capa-controller-manager after 5 min, plus cert-manager pods in ImagePullBackOff from us-docker.pkg.dev.

imageswap webhook namespaceSelector misses cluster-XXX namespaces

Symptom: Pods in cluster-<id> namespaces on PCG stuck in ImagePullBackOff from us-docker.pkg.dev even with the imageswap webhook deployed.

Root cause: The default MWC skipped cluster-* namespaces. A NotIn selector excluding only system namespaces forces cluster-wide coverage.

Fix: patch_imageswap_selector sets namespaceSelector.matchExpressions to key: kubernetes.io/metadata.name, operator: NotIn, values: [kube-system, kube-public, kube-node-lease, imageswap-system] (idempotent). Note a custom imageswap-webhook addon pack does not work — palette hardcodes downloadpriority.go:73 "skipping addon pack in pilot environment" and ignores any pack by that name; the built-in imageSwap.imageChange mechanism on the kubernetes-eks pack is the supported path.

Recognition signal: pods in cluster-* namespaces stuck pulling us-docker.pkg.dev.

Cilium: ipam: kubernetes incompatible with EKS

Symptom: cilium-agent Running but stuck 0/1 with "required IPv4 PodCIDR not available" repeating every ~30s.

Root cause: Spectro's cni-cilium-fips pack defaults to ipam.mode=kubernetes, which expects kube-controller-manager --allocate-node-cidrs to set each node's spec.podCIDR (kubeadm style). EKS does not set podCIDR — the CNI must allocate its own IPs.

Fix: Set ipam.mode: cluster-pool with clusterPoolIPv4PodCIDRList: [10.244.0.0/16] and clusterPoolIPv4MaskSize: 24 (must not overlap the VPC 10.1.0.0/16 or service CIDR 10.96.0.0/12). Restart the DS and the operator so it picks up the new mode.

Recognition signal: cilium-agent 0/1, logs show required IPv4 PodCIDR not available.

Cilium operator can't schedule on single-node clusters

Symptom: cilium-operator-* pod Pending with didn't have free ports for the requested pod ports.

Root cause: The default chart runs the operator with replicas: 2, RollingUpdate, and a hostPort. On a 1-node cluster (typical early in provisioning) the two anti-affine pods both want the same hostPort and can't co-locate — rollout never completes; an old-config pod may keep running, so kubectl rollout restart does nothing.

Fix: Set operator.replicas: 1 (preferred for single-node), or keep replicas: 2 with updateStrategy.type: Recreate.

Recognition signal: cilium-operator Pending with didn't have free ports.

metrics-server APIService "Address is not allowed" (needs hostNetwork)

Symptom: VerteX UI shows 0 / 0 Cores / 0 / 0 Gb; kubectl top nodes returns "Metrics API not available"; v1beta1.metrics.k8s.io APIService Available=False with Address is not allowed against a 10.244.x.x pod IP.

Root cause: metrics-server isn't in any Spectro pack, so workload clusters lack it. When installed with a default pod-network deployment, the EKS-managed apiserver can't dial the Cilium PodCIDR (10.244.0.0/16) — no route is advertised back to the EKS-managed VPC, so the dial is rejected at L3.

Fix: ensure_metrics_server installs it with hostNetwork: true (pod takes the routable node IP 10.1.x.x) and --secure-port 4443 to avoid colliding with kubelet's 10250.

Recognition signal: metrics-server APIService Available=False with Address is not allowed, pod IP in the Cilium PodCIDR.

vertex.recrocog.com resolves to public NLB IPs (same-VPC hairpin)

Symptom: ally/jet pods (on workload and on PCG itself) fail to register with the mgmt hubble API: Post "https://vertex.recrocog.com/v1/auth/certs": dial tcp 3.x.x.x:443: connect: connection refused / context deadline exceeded.

Root cause: AWS NLB does not allow same-VPC hairpin — a pod that resolves vertex.recrocog.com to the NLB's public IPs and connects from inside the same VPC gets dropped.

Fix: Patch CoreDNS with a vertex.recrocog.com hosts override pointing at the NLB's private-subnet ENI IPs (discover via aws ec2 describe-network-interfaces --filters "Name=description,Values=ELB *" Name=vpc-id,Values=<vpc>). Codified in ensure_workload_coredns (workload) and ensure_pcg_coredns (PCG). The controller re-derives the IPs each reconcile so ENI rotation self-heals.

Recognition signal: ally/jet show connection-refused to vertex.recrocog.com; nslookup from a pod returns public IPs.

Note: The CoreDNS health plugin must use the multi-line block form (health {\n lameduck 5s\n}). The inline form health { lameduck 5s } crashes CoreDNS on EKS v1.31+ (Wrong argument count or unexpected line ending), CrashLooping all CoreDNS pods and blocking nodes from going Ready. Confirmed on both PCG and workload clusters.

EKS API extra Security Group has zero ingress

Symptom: AWSMCP SecondaryCidrsReady=False — deleting resource kube-system/aws-node: ... Get "https://<eks-api>:443/api?timeout=1m0s": dial tcp <api-eni-ip>:443: i/o timeout.

Root cause: The workload cluster's additional cluster SG was created with zero ingress rules, so workload-node pods (and CAPA's aws-node deletion step) couldn't reach the cluster's own EKS API.

Fix: ensure_eks_api_sg_ingress adds 443/tcp ingress from the PCG node SG, the workload primary cluster SG, and the workload node-group additional SG. Critical: discover source SGs via AWSManagedControlPlane.status.networkStatus.securityGroups only — enumerating all node SGs pulls in the EKS-managed eks-remoteAccess-* SG and creates a delete-time deadlock (see Delete-Path Blockers).

Recognition signal: AWSMCP SecondaryCidrsReady=False with an i/o timeout to the EKS API endpoint IP.

EBS CSI IAM, IMDS chain (inline policy, hop=2, hostNetwork)

Symptom: ebs-csi-controller CrashLoopBackOff — first 403 UnauthorizedOperation on ec2:DescribeAvailabilityZones, then (after policy attach) no EC2 IMDS role found, context deadline exceeded.

Root cause: three stacked problems. (1) The default node IAM role lacks AmazonEBSCSIDriverPolicy; and attaching it as a managed policy gets detached within ~30s because CAPA reconciles attached policies to match the (empty) roleAdditionalPolicies. (2) EC2 default HttpPutResponseHopLimit=1 blocks pods from reaching IMDS. (3) Cilium with ipam: cluster-pool doesn't masquerade pod→169.254.169.254, which only answers on the host network namespace.

Fix: (1) Use an inline role policy (aws iam put-role-policy) — CAPA only manages attached managed policies, so inline survives; ensure_ebs_csi_node_role_policy. (2) ensure_imds_hop_2 sets --http-put-response-hop-limit 2 --http-tokens required on each worker. (3) ensure_ebs_csi_hostnetwork patches the deployment to hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet, then force-deletes crashlooping pods for fresh STS creds. Setting roleAdditionalPolicies in the profile can propagate here (unlike kubeProxy) but the controller keeps all three as defense-in-depth.

Recognition signal: ebs-csi errors cycle through the three messages above.

Pack runtime images not in ECR mirror

Symptom: Workload node NotReady; cilium/csi pods in ImagePullBackOff from us-docker.pkg.dev/palette-images-fips/packs/...; after ~30 min the node group hits CREATE_FAILED: Unhealthy nodes.

Root cause: vertex-mirror-pull.sh syncs chart bundles + Spectro's own runtime images but not every pack's runtime images (cilium was missing). Workload clusters have no imageswap webhook and can't reach us-docker.pkg.dev.

Fix (preferred): rewrite repository:/registry: fields in each pack's values.yaml to ECR paths (us-docker.pkg.dev/palette-images-fips → <acct>.dkr.ecr.us-east-1.amazonaws.com/vertex-bootstrap/palette-images-fips; quay.io/gcr.io/docker.io → .../vertex-bootstrap) — palette renders ECR paths directly, no runtime kubectl set image. The cni-cilium-fips pack has 13 repository: fields (watch preflight.envoy.image at quay.io/cilium/cilium-envoy and the spire initImage under third-party/alpine); csi-aws-ebs has 8. The images must still exist in ECR (one-time mirror per pack version).

Recognition signal: worker NotReady, kube-system pods pulling us-docker.pkg.dev.

Delete-Path Blockers

Everything that makes a workload-cluster delete stall indefinitely.

eks-remoteAccess SG circular dependency

Symptom: UI stuck at "Worker nodes being deleted" for 20+ min. EC2 + ASG already gone, but the EKS nodegroup stays DELETING with health.issues: [{code: Ec2SecurityGroupDeletionFailure, message: DependencyViolation}] on an eks-remoteAccess-* SG.

Root cause: When a nodegroup spec includes remoteAccess (SSH key), EKS creates a remote-access SG and deletes it during teardown. But ensure_eks_api_sg_ingress (the create-path SG fix) had widened its source-SG discovery to every SG on the nodes — pulling in the EKS-managed remote-access SG as a 443/tcp source on the cluster SG. EKS then can't delete its own SG because the cluster SG still references it → deadlock.

Fix: Discover source SGs from AWSManagedControlPlane.status.networkStatus.securityGroups only (never enumerate arbitrary node SGs) — the remote-access SG is not in networkStatus, so it's naturally excluded. Validated — no manual SG revoke needed to finalize. Manual unblock for a stuck cluster: aws ec2 revoke-security-group-ingress the offending 443/tcp pair referencing the eks-remoteAccess-* SG.

Recognition signal: an eks-remoteAccess-* SG appears as a UserIdGroupPair.GroupId in a rule described vertex-patches-controller: workload nodes to EKS API.

CAPI reverse-pivot drops the kubeconfig Secret OwnerRef

Symptom: After clicking Delete, CAPA refuses to proceed: failed reconciling kubeconfig: updating kubeconfig secret: EKS kubeconfig cluster-<id>/<name>-kubeconfig missing expected AWSManagedControlPlane ownership. DeleteNodegroup never fires; nodegroup stays ACTIVE.

Root cause: The <name>-kubeconfig Secret's ownerReferences points at kind: Cluster instead of kind: AWSManagedControlPlane — either CAPI's reverse-pivot (clusterctl move) drops the ref, or the SA-token rebuild during an earlier recovery didn't set it. CAPA won't update a Secret it doesn't own, so it wedges on the delete cascade.

Fix: ensure_kubeconfig_secret_ownerref patches the Secret's ownerReferences to apiVersion: controlplane.cluster.x-k8s.io/v1beta2, kind: AWSManagedControlPlane, name: <name>-cp, uid: <AWSMCP-uid>, controller: true, blockOwnerDeletion: true. Must run during phase=Deleting (see next entry) and restores the ref every tick if pivot drops it.

Recognition signal: CAPA delete-time logs spam the "missing expected AWSManagedControlPlane ownership" error while the nodegroup stays ACTIVE.

Over-broad phase=Deleting skip stalls teardown

Symptom: A cluster created and deleted within one 60s reconcile tick never gets its CAPA env patches; during delete, CAPA errors fips.eks.us-east-1.amazonaws.com: no such host / iam-fips.amazonaws.com: i/o timeout and DeleteNodegroup is never issued.

Root cause: the controller blanket-skipped any phase=Deleting cluster ("don't fight CAPI's deletion reconciles"). But without the FIPS-disable env vars, CAPA can't reach any AWS API — so it can't even DescribeNodegroup, let alone delete. A bootstrap-vs-teardown race.

Fix: split patches into lifecycle-safe (always apply, before the guard) vs active-management (skip on Deleting). Lifecycle-safe: CAPA/palette-cm env vars, PCG CoreDNS rewrites, and the kubeconfig-Secret OwnerRef patch. Active (skipped on Deleting): workload CoreDNS, metrics-server, palette cert DN, Cluster.spec.paused=false, AWSMCP annotation kicks, SG ingress. Validated: FIPS env applied first → DeleteNodegroup fires cleanly during teardown.

Recognition signal: controller logs show only phase=Deleting (no [applied] lines) for the cluster, while workload CAPA env still shows AWS_USE_FIPS_ENDPOINT=true.

Self-Hosted PCG Gotchas

Issues with the Self-Hosted PCG cluster and the Spectro provisioning agents that run on it (jet, ally, cluster-management-agent).

PCG agent deletes workload clusters mid-provision

Symptom: After a mongo restore + redeploy, workload cluster creates succeed to worker-Ready then get auto-deleted ~10 min later; cluster-management-agent-lite on PCG logs Spectro cluster AWS is not found every minute.

Root cause: The PCG's own spectrocluster CR (AWS) is missing from mgmt's Kubernetes after the mongo restore (mongo has the record; the K8s resource does not). The agent can't find it, classifies new workload namespaces as orphans, and deletes them. On a fresh bastion this didn't happen because a bootstrap step created the CR; a destroy+reapply+restore didn't recreate it.

Fix: Recreate the PCG spectrocluster CR from the mongo doc (name AWS, annotations HubbleUid/tenantUid/scope=tenant, spec.cloudType: eks) after restore, idempotently. Fallback: scale cluster-management-agent-lite to 0 in vertex-precluster-reset.sh.

Recognition signal: Spectro cluster AWS is not found in the PCG agent log, and workload namespaces vanishing shortly after the workers go Ready.

Auth-mode locks out bastion/SSO post-pivot

Symptom: After pivot, aws eks update-kubeconfig --name <cluster> returns Unauthorized for bastion and SSO roles; aws eks list-access-entries fails demanding auth mode [API, API_AND_CONFIG_MAP].

Root cause: CAPA creates the workload cluster with bootstrapClusterCreatorAdminPermissions=false and CONFIG_MAP-only auth (the profile's accessConfig doesn't propagate — see Create-Path). The only admin principal is the static CAPA bootstrap IAM user. If the controller then breaks workload CoreDNS, the operator can't easily reach the cluster to fix it.

Fix: ensure_awsmcp_access_mode flips auth to API_AND_CONFIG_MAP, then the controller creates an access entry for an admin/SSO role. Recovery path: aws eks update-cluster-config --access-config authenticationMode=API_AND_CONFIG_MAP (via the CAPA principal) → create-access-entry → associate-access-policy.

Recognition signal: Unauthorized from update-kubeconfig and the list-access-entries auth-mode error.

Note: After pivot (~25-30 min in), the cluster-<id> namespace and its <cluster>-kubeconfig Secret move off PCG onto the workload cluster. The controller must fetch a fresh kubeconfig via aws eks get-token rather than relying on the Secret (which disappears with the move) — otherwise post-pivot patch functions stop firing. This build_workload_kubeconfig refactor is validated in production.

IMDSv2 hop limit must be 2

Symptom: Pods on PCG and workload nodes can't reach IMDS (169.254.169.254) through the Cilium veth, breaking any component that falls back to the node IAM role.

Root cause: EC2 default HttpPutResponseHopLimit=1 blocks the extra hop from the pod network namespace to IMDS.

Fix: Set metadata_options { http_put_response_hop_limit = 2, http_tokens = "required" } on node-group launch templates in terraform, and ensure_imds_hop_2 bumps live workers (aws ec2 modify-instance-metadata-options).

Recognition signal: IMDS timeouts from pods that fall back to node-role credentials (e.g., ebs-csi no EC2 IMDS role found).

Patch Coverage

Status of what the PCG-resident vertex-patches-controller auto-fixes vs. what remained manual or terraform-only (as of the 2026-05-13 cold reapply and the 2026-05-14 validation runs).

Auto-patched by the controller

Issue Controller handler Notes
Cluster paused after pivot ensure_cluster_unpaused waits for AWSMCP Ready=True before un-pausing
palette-webhook empty-DN cert ensure_palette_cert_dn forces cert-manager re-issue
NLB hairpin + FIPS CoreDNS (workload) ensure_workload_coredns NLB private IPs + FIPS rewrites
NLB hairpin + cross-region NXDOMAIN (PCG) ensure_pcg_coredns multi-line health{}; fallthrough required
CAPA FIPS env (pre + post pivot) patch_capa_envs + ensure_workload_capa_envs AWS_USE_FIPS_ENDPOINT=false
palette-cm FIPS/CFN env patch_palette_cm_envs + ensure_workload_palette_cm_envs plus non-FIPS CFN VPC endpoint (terraform)
kube-proxy addon delete loop ensure_kube_proxy_addon + ensure_awsmcp_kubeproxy_in_addons_list addons-list patch (with version) is the real fix
AWSMCP kubeProxy.disable gap ensure_awsmcp_kubeproxy_enabled profile value doesn't propagate
AWSMCP auth-mode gap ensure_awsmcp_access_mode gated on EKS cluster ACTIVE
EKS API SG zero-ingress ensure_eks_api_sg_ingress SG discovery via networkStatus only
metrics-server + APIService ensure_metrics_server hostNetwork: true, port 4443
imageswap selector patch_imageswap_selector NotIn system namespaces
IMDS hop=2 ensure_imds_hop_2
ebs-csi hostNetwork ensure_ebs_csi_hostnetwork
ebs-csi inline node policy ensure_ebs_csi_node_role_policy inline (survives CAPA reconcile)
kubeconfig Secret OwnerRef (delete) ensure_kubeconfig_secret_ownerref runs before the phase=Deleting guard

Terraform-only (not in-cluster)

Fix Location
Non-FIPS CloudFormation VPC endpoint terraform/modules/vertex-bootstrap
Non-FIPS Auto Scaling VPC endpoint terraform/modules/vertex-bootstrap
Node-group IMDSv2 hop=2 launch template terraform/modules/vertex-pcg/eks.tf

Remaining manual / defense-in-depth

Issue Status Mitigation
AWSMCP SecondaryCidrsReady=False stuck after SG fix cosmetic — cluster still reaches Running kubectl annotate awsmanagedcontrolplane <name> capi-refresh=$(date +%s) --overwrite
jet/ally/cma/crony images from us-docker.pkg.dev race — only bites if PCG agent pods land before the webhook is patched delete the pods to re-admit, or kubectl set image to ECR paths
WorkerNodeAdditionDone stuck (MachinePool has no per-node Machine CRs) resolved upstream in 4.8.40 — cma reads MachinePool.status.nodeRefs shim function ensure_machine_crs_for_machinepool deleted as dead code
CAPI OwnerRefs missing post-pivot resolved upstream in 4.8.40 — CRs get native OwnerRefs ensure_capi_ownerrefs deleted (never fired)
CAPA revokes controller SG rules each reconcile noisy, not blocking rules re-added every 60s; codify-later: SG-rule marker CAPA respects
mgmt mgmt_monitor_services.go:56 packsync panic recovered panic, watcher loop dies periodic mgmt restart; upstream Spectro bug
Pack registry sync gotchas (creds cleared on UI edit; baseContentPath; 5-day backoff) documented re-enter secret key; use baseContentPath=vertex-bootstrap; delete+re-add registry to clear backoff

Note: The controller keeps ensure_kube_proxy_addon, ensure_ebs_csi_node_role_policy, ensure_awsmcp_kubeproxy_enabled, and ensure_awsmcp_access_mode as defense-in-depth even where a profile edit should handle them — because Spectro's pack → AWSMCP propagation has confirmed systemic gaps. Each is a no-op when the field is already correct.

See Also