VerteX Air-Gapped Install on EKS¶

Self-hosted Palette VerteX deployment on AWS EKS in the Recro internal account. Air-gapped: private-only EKS API, no NAT, all images pulled from ECR via VPC endpoints.

The bastion is fully autonomous — apply Terraform, wait (ETA ~40 min, varies), VerteX is running. No manual intervention.

Note: This page covers only the management / bootstrap cluster install, which is autonomous end-to-end. Provisioning workload (spoke) clusters through the VerteX UI is a separate, manual-heavy flow with its own set of FIPS/CAPA/kube-proxy fixes — see Known Issues & Fixes. The custom AWS OS pack, backups and restore, and full teardown are covered below.

Contents¶

Background

Architecture
Why a Self-Hosted PCG?
Repo Layout (recro-aws-iac)

Install

First-Time vs Rebuild
Prerequisites (one-time)
First-Time Setup

Operate

Destroy and Rebuild
Backups and Restore
Full Teardown (Decommission)
Monitoring and Troubleshooting

Reference

Quick Reference
CI Tag Convention
Scripts Reference
Custom AWS OS Pack

Architecture¶

S3 Bundle Bucket (vertex-ecr module)
  ├── charts.zip, palette-vertex-appliance-4.8.40.tar.zst
  ├── index.json, oci-layout (OCI metadata)
  ├── Scripts (vertex-mirror-pull, ecr-mirror, cluster-setup, full-setup, dns-update)
  └── Helm values (cert-manager, image-swap, vertex airgap configs)

ECR (vertex-ecr module)
  └── 189 repos under vertex-bootstrap/* (the mirrored container images)
        (only 2 are Terraform-managed: spectro-packs, spectro-images;
         the other ~187 are created out-of-band by the mirror push)

vertex-bootstrap module
  ├── VPC 10.1.0.0/16 (private + public subnets, no NAT)
  ├── VPC Endpoints (~24 interface + S3 gateway; ECR/S3/EKS/STS/EC2/SSM/logs
  │                  + FIPS variants + cloudformation/autoscaling/iam — subset shown)
  ├── EKS cluster (K8s 1.32, private-only, 3x m5.2xlarge)  ← management/mgmt plane
  ├── Client VPN (mutual TLS, split tunnel, 10.250.0.0/22)
  └── Bastion (t3.large, public subnet, SSM access)
        On boot:
        1. Install tools (kubectl, helm, oras, docker)
        2. Download scripts + values from S3
        3. Pull bundle from S3 + extract (17 GB)
        4. Wait for EKS cluster ACTIVE
        5. Wait for nodes Ready + create gp3 StorageClass
        6. Check ECR → mirror if empty (~20 min first time, skip on rebuild)
        7. Helm install cert-manager, image-swap, hubble
        8. Update DNS (vertex.recrocog.com → NLB)
        9. Export VPN .ovpn config to S3
        10. Init PCG (patches-controller) + mgmt CoreDNS CronJob
        11. VerteX running

vertex-pcg module          ← ALSO provisioned by apply-vertex
  └── EKS cluster (K8s 1.31, private-only, 1x t3.xlarge) in the SAME VPC
        Self-Hosted PCG: jet + ally + CAPA/CAPI agents that provision AWS
        workload clusters (the built-in System PCG can't register AWS accounts).
        See Architecture for why this must be a separate cluster.

Note: apply-vertex (and the <date>-vertex tag) stands up two EKS clusters, not one — the vertex-bootstrap mgmt cluster and the vertex-pcg Self-Hosted PCG cluster (vertex-pcg-aws, K8s 1.31, 1× t3.xlarge, same VPC). This page focuses on the mgmt-plane install; the Why a Self-Hosted PCG? section below explains the PCG's role.

Why a Self-Hosted PCG?¶

VerteX ships a built-in System PCG (Private Cloud Gateway) in the management plane, but it only supports private-cloud providers (MAAS/CloudStack/vSphere) and cannot register an AWS cloud account. Provisioning EKS/AWS workload clusters must instead route through a Self-Hosted PCG — the jet + ally agents plus the CAPA/CAPI controllers that drive cluster bring-up.

Spectro also forbids running the Self-Hosted PCG on a Palette-managed cluster, and the mgmt plane is itself Palette-managed — so the PCG gets its own dedicated EKS cluster (vertex_pcg), placed in the same VPC as vertex_bootstrap (reusing its private subnets) so the bastion and mgmt plane reach its Kubernetes API without peering. Net: airgapped VerteX-on-EKS needs two EKS clusters — mgmt (vertex_bootstrap) + PCG (vertex_pcg) — in one shared VPC. The rationale is captured in the module.vertex_pcg comment in terraform/main.tf.

Repo Layout (recro-aws-iac)¶

All VerteX infra lives in recro-aws-iac. Three sibling Terraform modules apply in a strict order — vertex_ecr → vertex_bootstrap → vertex_pcg — wired in terraform/main.tf (~lines 98–201) and configured in terraform/vertex.tfvars.

recro-aws-iac/terraform/
├── main.tf                  # wires the 3 vertex modules (~lines 98–201)
├── vertex.tfvars            # config blocks: vertex_ecr_config, vertex_config, vertex_pcg_config
├── ci.tf                    # ci-vertex-kms policy (CI uploads to the bundle bucket)
├── Makefile                 # make plan-vertex / apply-vertex  (+ plan-core)
└── modules/
    ├── vertex-ecr/          # durable artifact layer — survives cluster rebuilds
    │   ├── ecr.tf           #   ECR repos + alias/vertex-bootstrap-ecr KMS CMK
    │   ├── s3.tf            #   891377028731-vertex-bootstrap-bundle (versioned)
    │   ├── iam_reader.tf    #   vertex-bootstrap-ecr-reader (UI Pack Registry creds)
    │   ├── iam_backup.tf    #   vertex-bootstrap-backup-writer (mongo backup → S3)
    │   ├── ssm_params.tf    #   /vertex/*-key SecureString params
    │   ├── scripts.tf       #   uploads the 26 bastion scripts/values to S3
    │   ├── scripts/         #   the bastion scripts (see Scripts Reference)
    │   └── custom-packs/    #   build-amazon-linux-eks.sh (see Custom AWS OS Pack)
    ├── vertex-bootstrap/    # airgap VPC + management EKS + bastion + VPN
    │   ├── vpc.tf           #   VPC 10.1.0.0/16, no NAT (+ destroy-time ELB cleanup)
    │   ├── vpc_endpoints.tf #   ~24 interface + S3 gateway endpoints (only egress)
    │   ├── eks.tf           #   management EKS (K8s 1.32)
    │   ├── bastion.tf       #   bastion — runs vertex-full-setup.sh
    │   ├── vpn.tf           #   AWS Client VPN (mutual TLS)
    │   ├── iam.tf           #   cluster/node/mgmt IAM + OIDC (IRSA)
    │   └── templates/       #   bastion-user-data.sh.tftpl
    └── vertex-pcg/          # Self-Hosted PCG EKS (2nd cluster, shared VPC)
        ├── eks.tf          #   PCG EKS (K8s 1.31), 443 ingress from bastion
        ├── iam.tf          #   cluster/node IAM + OIDC (patches-controller/agents)
        └── main.tf         #   SSO-group → EKS-access map

Dependency chain — outputs threaded between modules in main.tf (enforced by depends_on):

vertex_ecr ──► vertex_bootstrap ──► vertex_pcg

From → To	Outputs passed
`vertex_ecr` → `vertex_bootstrap`	`repository_arns`, `kms_key_arn`, `content_bundle_bucket_name` / `_arn`
`vertex_ecr` → `vertex_pcg`	`repository_arns`, `kms_key_arn` (PCG nodes pull airgapped images)
`vertex_bootstrap` → `vertex_pcg`	`vpc_id`, `private_subnet_ids`, bastion role/SG, cluster SG

All three modules also receive the architect / engineer / audit SSO role ARNs, so each cluster grants those groups EKS Access Entries.

Make targets (Makefile):

Target	Scope
`make plan-vertex` / `apply-vertex`	All three vertex modules (`vertex_ecr` + `vertex_bootstrap` + `vertex_pcg`)
`make plan-core` / `apply-core`	Everything except the vertex clusters

First-Time vs Rebuild¶

Two distinct paths share the same autonomous bastion bootstrap:

	First-time install	Rebuild (redeploy-after-destroy)
ECR	Empty → bastion mirrors the container images (ETA ~20-25 min)	Already populated → mirror skipped
S3 bundle	Must be uploaded (17 GB, 60-90 min one-time)	Persists across rebuilds
Mongo state	Fresh (empty tenant)	Restored from `s3://…/backups/latest.tgz` (tenant, cluster profiles, PCG, Cloud Account)
Wall time (ETA, varies)	~40 min after apply	~25 min after apply

First-time follows the full 6-phase deployment plan: AWS foundation (VPC, endpoints, ECR, EKS) → mirror content bundle → install VerteX (Helm) → build cluster profiles → deploy spokes → (future) Spectro-native migration. Only Phase 1-3 are automated by the bastion; Phase 4-5 (profiles, workload clusters) are UI/manual.

Rebuild is hands-off: make apply-vertex (or a <date>-vertex tag) brings back VPC/EKS/bastion, and the bastion's vertex-full-setup.sh re-runs every codified fix (VPC endpoints, CoreDNS sinkhole/hairpin, hubble binary patch, imageswap NotIn webhook, Mongo restore, and vertex-pcg-reapply.sh to flip the restored PCG back to Running). Plan for ~1.5 hours end-to-end if you also recreate a workload cluster. Codified fixes apply automatically; the per-workload-cluster FIPS/kube-proxy/CAPA patches remain manual — see Known Issues & Fixes.

Warning: Before destroying vertex_bootstrap, force a fresh Mongo backup so latest.tgz reflects current state — a stale backup surfaces stale tenant/PCG records on redeploy. See Backups and Restore.

Prerequisites (one-time)¶

1. Spectro Cloud credentials¶

Download VerteX installer files from Artifact Studio.

Username: spectro
Password: mV715z##spPSJC

Shared credentials. Do not commit.

2. Download from Artifact Studio¶

Artifact Studio portal → Palette VerteX → Releases → 4.8.40

File	Size	Purpose
`charts.zip`	~390 KB	Helm charts (cert-manager, image-swap, VerteX mgmt plane)
`palette-vertex-appliance-4.8.40.tar.zst`	~17 GB	Container images + Spectro packs

3. Extract OCI metadata from the bundle¶

The tar.zst contains index.json, index.json.lock, and oci-layout that the mirror script needs. Extract them separately:

tar --use-compress-program=unzstd -xf palette-vertex-appliance-4.8.40.tar.zst \
  -C /tmp index.json index.json.lock oci-layout

4. AWS access¶

Member of the recro GitHub org with access to recro-aws-iac.

aws sso login --profile <your-profile>
git clone https://github.com/recro/recro-aws-iac.git
cd recro-aws-iac

5. Local tools¶

git, terraform 1.10+, aws CLI v2, aws-session-manager-plugin

First-Time Setup¶

Step 1: Apply vertex_ecr (S3 bucket + ECR repos)¶

cd terraform && make init
terraform apply \
  -var="region=us-east-1" \
  -var-file=sso.tfvars -var-file=dns.tfvars -var-file=eks.tfvars \
  -var-file=cognito.tfvars -var-file=resource-cleanup.tfvars \
  -var-file=resource-startup.tfvars -var-file=manual-cleanup-web-app.tfvars \
  -var-file=vertex.tfvars \
  -target='module.vertex_ecr[0]'

Step 2: Upload bundle files to S3¶

aws s3 cp charts.zip s3://891377028731-vertex-bootstrap-bundle/
aws s3 cp palette-vertex-appliance-4.8.40.tar.zst s3://891377028731-vertex-bootstrap-bundle/

# Upload OCI metadata (extracted in prerequisite step 3)
aws s3 cp /tmp/index.json s3://891377028731-vertex-bootstrap-bundle/
aws s3 cp /tmp/index.json.lock s3://891377028731-vertex-bootstrap-bundle/
aws s3 cp /tmp/oci-layout s3://891377028731-vertex-bootstrap-bundle/

The 17 GB upload takes 60-90 min on residential internet. This is a one-time step — the files persist in S3 across rebuilds.

Step 3: Apply vertex_bootstrap¶

Via CI (recommended):

git tag <date>-vertex -m "Deploy vertex_bootstrap"
git push origin <date>-vertex

Tag must contain "vertex" to trigger the vertex-scoped CI apply. ~30-40 min total.

Or locally:

make plan-vertex && make apply-vertex

Step 4: Wait¶

The bastion does everything automatically:

Installs tools (~3 min)
Downloads scripts + values from S3 (~10 sec)
Pulls 17 GB bundle from S3 + extracts (~10 min)
Waits for EKS cluster ACTIVE (~12 min, overlaps with bastion boot)
Creates gp3 StorageClass (~5 sec)
Mirrors the container images to ECR (ETA ~20 min, first time only)
Helm installs cert-manager, image-swap, hubble (~10 min)
Updates DNS vertex.recrocog.com → NLB

Total first-time ETA: ~40 min. Rebuilds (ECR already populated): ~25 min. These are rough estimates — actual time varies with bundle download speed and AWS API latency.

Step 5: Access VerteX console via VPN¶

The bastion auto-exports a .ovpn config file to S3 after each rebuild. Download it and connect.

First time — install AWS VPN Client:

Download from aws.amazon.com/vpn/client-vpn-download

Download .ovpn config (re-download after each rebuild):

aws s3 cp s3://891377028731-vertex-bootstrap-bundle/vertex-vpn.ovpn .

Connect:

Open AWS VPN Client
File → Manage Profiles → Add Profile → select vertex-vpn.ovpn
Click Connect
Browse https://vertex.recrocog.com/system

Login: admin / set your password (14+ chars, upper, lower, digit, special)

kubectl also works directly while connected:

aws eks update-kubeconfig --region us-east-1 --name vertex-bootstrap
kubectl get nodes

Fallback — SSM tunnel (if VPN is unavailable):

BASTION=$(aws ec2 describe-instances --region us-east-1 \
  --filters "Name=tag:Name,Values=vertex-bootstrap-bastion" \
            "Name=instance-state-name,Values=running" \
  --query 'Reservations[0].Instances[0].InstanceId' --output text)

aws ssm start-session --region us-east-1 --target $BASTION \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["<NLB-hostname>"],"portNumber":["443"],"localPortNumber":["8443"]}'

Browse: https://localhost:8443/system

Destroy and Rebuild¶

Destroy (save costs)¶

Destroy is manual / admin-only — a <date>-vertex tag triggers an apply, not a destroy. Run locally, targeting both compute modules so the PCG isn't left orphaned in the VPC (Terraform destroys vertex_pcg before vertex_bootstrap automatically, since PCG depends on the bootstrap VPC/subnets):

terraform destroy \
  -target='module.vertex_pcg[0]' \
  -target='module.vertex_bootstrap[0]' \
  -auto-approve ...   # (same -var-file set as apply-vertex)

Warning: Do not destroy only module.vertex_bootstrap[0] — that leaves the vertex-pcg cluster running (and billing) in a VPC Terraform is trying to tear down, and breaks the dependency graph on the next apply.

Destroys: VPC, both EKS clusters (mgmt + PCG), bastion, all K8s workloads (~$1,100/mo → $4/mo idle) Keeps: ECR repos (189 repos), S3 bucket (bundle + scripts), KMS keys

Note: ECR lives in the separate vertex_ecr module, so it survives a vertex_bootstrap destroy. Only 2 of the 189 repos (spectro-packs, spectro-images) are Terraform-managed; the other ~187 are created out-of-band by the mirror push. To remove ECR permanently, see Full Teardown (Decommission) below.

Rebuild¶

git tag <date>-vertex -m "Rebuild"
git push origin <date>-vertex

Bastion boots, skips ECR mirror (images already there), installs Helm, VerteX up in ~25 min. DNS auto-updates. New .ovpn uploaded to S3 — re-download and reconnect.

Backups and Restore¶

VerteX management-plane state lives in MongoDB (hubbledb + hubble_timeseriesdb) in the hubble-system namespace. Each backup is a single bundle capturing both the database and the K8s Secrets needed to decrypt it — both halves are required for a working restore.

What's in each backup (backups/latest.tgz, written to the S3 bundle bucket by the vertex-backup-writer IAM user):

dump.archive   # mongodump --gzip of hubbledb + hubble_timeseriesdb  (data / ciphers)
secrets.json   # the hubble-system Secrets (all except the replica-set / SA keys, ~20+);
               #   critical: configserversecret (holds the master encryption key), spectromongosecret, mongo-tls

Why both halves: mongo stores tenant/user/activation values as {cipher}… encrypted with a master key that lives only in a K8s Secret (etcd), never in mongo. Restore the database alone and helm mints a new key on install → every cipher becomes permanently unreadable. Backing up and restoring the Secret is what keeps the data decryptable. (The bundle deliberately excludes the mongo replica-set / ServiceAccount keys, which must regenerate fresh.)

Schedule & rotation — an in-cluster CronJob (vertex-mongo-backup) copies each run into rotating slots:

S3 path	Cadence	Slots
`backups/latest.tgz`	every run	overwritten
`backups/daily/<Mon-Sun>.tgz`	daily	7
`backups/weekly/week-<0-3>.tgz`	Sundays	4
`backups/monthly/month-<0-2>.tgz`	1st of month	3

A second weekly CronJob (vertex-backup-verify) downloads latest.tgz, confirms both halves are present and non-empty, and runs mongorestore --dryRun to validate the archive — writing the outcome to backups/last-verify.json so a stale or corrupt backup is caught before it's needed.

How restore works — orchestrated by vertex-full-setup.sh on a fresh / rebuild install:

Secrets first (before helm) — vertex-secrets-restore.sh re-applies secrets.json, so helm's lookup finds the existing encryption key and reuses it instead of generating a new one.
Helm installs the mgmt plane with the preserved key.
Mongo after — vertex-restore-backup.sh runs mongorestore --drop; the restored ciphers decrypt cleanly against the reused key.

Scripts (terraform/modules/vertex-ecr/scripts/):

Script	Runs on	Purpose
`vertex-backup.sh`	in-cluster CronJob	Dump mongo + curated K8s Secrets → S3
`vertex-backup-verify.sh`	in-cluster CronJob	Weekly integrity check → `backups/last-verify.json`
`vertex-secrets-restore.sh`	bastion	Pre-helm Secrets restore (preserves the encryption key)
`vertex-restore-backup.sh`	bastion	Post-helm `mongorestore --drop`

Full Teardown (Decommission)¶

Permanent removal of the whole stack — as opposed to Destroy and Rebuild above, which keeps ECR/S3/KMS for a later rebuild. The compute tier (vertex_bootstrap + vertex_pcg) is already gone; this removes the persistence tier plus the resources Terraform doesn't manage.

Warning: Destructive, and not yet run against the account. Terraform manages only module.vertex_ecr (49 resources: the 2 named ECR repos, the S3 bucket, KMS CMK, SSM params, 2 IAM users). Everything else below is out-of-band and Terraform will NOT delete it — the ~187 other vertex-bootstrap/* ECR repos, the IAM users vertex-airgap-cloud-svc / vertex-pack-reader / vertex-pack-registry-reader, the two ACM VPN certs, the ci-vertex-kms policy, and the stale vertex.recrocog.com DNS record. Verify identity with /aws-whoami first.

⚠️ Landmine — tfvars will rebuild the stack: terraform/vertex.tfvars still declares vertex_config and vertex_pcg_config. A plain terraform apply — or any *-vertex date tag hitting CI — will recreate the entire bootstrap VPC + mgmt EKS + PCG from scratch. Do not push a <date>-vertex tag while these blocks remain; strip them (step 7) before or during teardown.

Run in order — ECR/S3 must be emptied before Terraform can destroy their containers, and the KMS key must outlive everything it encrypts.

1. Purge ECR repos (force-deletes all vertex-bootstrap/* repos + images; the 2 TF-managed ones are reconciled in step 3):

aws ecr describe-repositories --region us-east-1 \
  --query "repositories[?starts_with(repositoryName,'vertex-bootstrap/')].repositoryName" \
  --output text | tr '\t' '\n' | while read -r repo; do
    aws ecr delete-repository --region us-east-1 --repository-name "$repo" --force
  done

2. Empty the S3 bundle bucket — force_destroy = false + versioning enabled, so delete all objects, versions, and delete markers first (repeat until list-object-versions is empty — it's paginated):

BUCKET=891377028731-vertex-bootstrap-bundle
aws s3 rm "s3://${BUCKET}" --recursive
aws s3api list-object-versions --bucket "$BUCKET" \
  --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --output json > /tmp/v.json
aws s3api delete-objects --bucket "$BUCKET" --delete file:///tmp/v.json
aws s3api list-object-versions --bucket "$BUCKET" \
  --query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}' --output json > /tmp/m.json
aws s3api delete-objects --bucket "$BUCKET" --delete file:///tmp/m.json

3. Destroy module.vertex_ecr (removes the S3 bucket, the 2 TF ECR repos, KMS CMK/alias, SSM params /vertex/*, and the 2 TF-managed IAM users):

cd terraform && make init
terraform destroy -var="region=us-east-1" \
  -var-file=sso.tfvars -var-file=dns.tfvars -var-file=eks.tfvars \
  -var-file=cognito.tfvars -var-file=resource-cleanup.tfvars \
  -var-file=resource-startup.tfvars -var-file=manual-cleanup-web-app.tfvars \
  -var-file=vertex.tfvars -target='module.vertex_ecr[0]'

4. Schedule KMS key deletion — 30-day window; skip if step 3 already removed it:

KEY_ID=$(aws kms describe-key --region us-east-1 --key-id alias/vertex-bootstrap-ecr \
  --query 'KeyMetadata.KeyId' --output text)
aws kms schedule-key-deletion --region us-east-1 --key-id "$KEY_ID" --pending-window-in-days 30

5. Remove the out-of-band resources Terraform won't touch:

IAM users vertex-airgap-cloud-svc, vertex-pack-reader, vertex-pack-registry-reader — delete their access keys + inline/attached policies first, then aws iam delete-user --user-name <u>.
ACM VPN certs — the vpn_server_cert_arn and vpn_ca_cert_arn from vertex.tfvars (both issued for vertex.recrocog.com); imported out-of-band, so aws acm delete-certificate --certificate-arn <arn> each.
CI policy ci-vertex-kms + its attachment (terraform/ci.tf) — remove the blocks and apply, or terraform destroy -target='aws_iam_user_policy_attachment.ci_vertex_kms' -target='aws_iam_policy.ci_vertex_kms'.

6. Delete the stale DNS record — vertex.recrocog.com in the recrocog.com hosted zone (was managed dynamically by the deleted bastion). Confirm the record set, then submit a change batch with Action=DELETE.

7. Strip the vertex blocks from the IaC so no future apply/tag can rebuild anything:

terraform/vertex.tfvars — remove vertex_ecr_config, vertex_config, vertex_pcg_config.
terraform/main.tf — remove the three module.vertex_* blocks (and the vertex_config-gated SSO-role data sources/locals if unused elsewhere).
Makefile — drop the vertex -var-file/-target lines from plan-core, and remove the plan-vertex / apply-vertex targets.
terraform/ci.tf — remove ci_vertex_kms (from step 5).

Open a PR, merge, and confirm a full make plan shows no VerteX resources.

Monitoring and Troubleshooting¶

Check bastion logs¶

aws ssm start-session --region us-east-1 --target <bastion-id>
sudo -i

cat /var/log/vertex-full-setup.log     # Full orchestrator
cat /var/log/vertex-ecr-mirror.log     # ECR mirror
cat /var/log/vertex-cluster-setup.log  # Node wait + StorageClass
cat /var/log/vertex-dns-update.log     # DNS update
cat /var/log/vertex-bastion-bootstrap.log  # Tool versions

Re-run orchestrator¶

sudo /usr/local/bin/vertex-full-setup.sh

Idempotent — skips steps already completed.

Common issues¶

Issue	Symptom	Fix
Scripts missing from S3	`[boot] ERROR: Failed to download`	`terraform apply -target='module.vertex_ecr[0]'`
Cluster not ready	`Cluster status is CREATING`	Wait — orchestrator retries every 30s for 20 min
Image pull failures	Pods `ImagePullBackOff`	Check ECR mirror log, re-run `sudo /usr/local/bin/vertex-ecr-mirror.sh`
MongoDB PVCs pending	PVCs stuck `Pending`	Check `kubectl get sc` — gp3 should be default
ELB blocks destroy	`DependencyViolation` on subnet	Destroy provisioner handles this automatically (ELB + ENI + SG cleanup)
DNS stale	vertex.recrocog.com wrong NLB	`sudo /usr/local/bin/vertex-dns-update.sh`

Quick Reference¶

Thing	Value
Region	`us-east-1`
EKS cluster	`vertex-bootstrap`
EKS endpoint	Private only
VPC CIDR	`10.1.0.0/16`
ECR registry	`891377028731.dkr.ecr.us-east-1.amazonaws.com`
ECR repo prefix	`vertex-bootstrap/`
S3 bundle bucket	`s3://891377028731-vertex-bootstrap-bundle/`
Root domain	`vertex.recrocog.com`
VerteX version	4.8.40
Node sizing	3x m5.2xlarge (8 vCPU, 32 GB, 110 GB)
Spectro Artifact Studio	`spectro` / `mV715z##spPSJC`

CI Tag Convention¶

Tag	Scope
`<date>` (e.g., `2026-04-17`)	Core infra only (SSO, DNS, recro-eks) — no vertex
`<date>-vertex` (e.g., `2026-04-17-vertex`)	Vertex only (`vertex_ecr` + `vertex_bootstrap` + `vertex_pcg`)

Adding a new module to main.tf: add a -target line to plan-core in the Makefile or non-vertex CI applies will skip it.

Scripts Reference¶

All scripts and values files live in terraform/modules/vertex-ecr/scripts/ and are uploaded to the S3 bundle bucket as Terraform-managed objects (scripts.tf, 26 files). The bastion's user_data pulls them on first boot via aws s3 sync. vertex-full-setup.sh is the orchestrator that runs Steps 0–13 and invokes the rest.

Note: The config file /usr/local/bin/vertex-mirror-config.sh is not in S3 — it is rendered by Terraform user_data (bundle bucket, cluster name, Route53 zone, SSO admin role ARN) and sourced by every script.

Boot orchestration — run on the bastion by vertex-full-setup.sh:

Script	Step	Purpose	Auto on boot?
`vertex-full-setup.sh`	—	Orchestrator; runs Steps 0–13 in order (idempotent)	Yes
`vertex-mirror-pull.sh`	pre	Pull bundle + values from S3, extract (~17 GB)	Yes
`vertex-cluster-setup.sh`	2	Wait for nodes Ready, create gp3 StorageClass	Yes
`vertex-ecr-mirror.sh`	3	Mirror OCI images to ECR	Yes (if ECR empty)
`vertex-pack-push.sh`	4	Push Spectro packs to ECR `spectro-packs`	Yes
`vertex-cluster-packs-push.sh`	4b	Push cluster-provisioning packs (incl. custom AWS OS pack)	Yes
`vertex-binary-patch.sh` (+ `binary-patch.py`)	7d	Binary-patch `spectro-drive` (`fips.eks`→`eks` hostname rewrite)	Yes
`vertex-coredns-refresh.sh`	7c	CoreDNS overrides for legacy AWS hostnames / NLB hairpin	Yes
`vertex-dns-update.sh`	8	Point `vertex.recrocog.com` → NLB	Yes (after Helm)
`vertex-vpn-export.sh`	9	Export the `.ovpn` client config to S3	Yes
`vertex-pcg-reapply.sh`	11	Re-apply PCG jet/ally manifests after restore (no-op if no PCG in mongo)	Yes
`vertex-pcg-init.sh`	12	Build + apply the PCG-resident `vertex-patches-controller`	Yes
`vertex-mgmt-init.sh`	13	Install the mgmt-cluster CoreDNS-refresh CronJob	Yes

Backup / restore (Mongo VerteX state → S3 backups/, via the vertex-backup-writer IAM user):

Script	Purpose	Auto on boot?
`vertex-backup.sh`	`mongodump` → S3; installed as an in-cluster CronJob (Step 7d)	Yes (CronJob)
`vertex-backup-verify.sh`	Weekly restore dry-run integrity check (Step 7e)	Yes (CronJob)
`vertex-restore-backup.sh`	Restore Mongo from `s3://…/backups/latest.tgz`	On fresh install only (Step 7-restore)
`vertex-secrets-restore.sh`	Restore K8s Secrets (incl. master encryption key) alongside the Mongo dump	On fresh install only

See Backups and Restore for the backup model and the encryption-key gap.

In-cluster payloads (applied to a cluster, not run on the bastion):

File	Applied by	Runs on
`vertex-patches-controller.yaml` + `.Dockerfile`	`vertex-pcg-init.sh` (Step 12)	PCG cluster — auto-patches workload clusters (Known Issues & Fixes)
`vertex-mgmt-coredns-refresh.sh` + `-cronjob.yaml`	`vertex-mgmt-init.sh` (Step 13)	Mgmt cluster — CoreDNS refresh CronJob (replaces the bastion systemd timer)

Manual / per-cluster (not run on boot):

Script	Purpose
`vertex-capa-patcher.sh`	Break-glass CAPA FIPS patch — no longer needed in normal operation (automated by the PCG patches-controller)

Note: vertex-capa-patcher.sh is not required anymore — the same fix is now automated by the PCG-resident vertex-patches-controller (patch_capa_envs / ensure_workload_capa_envs), so a normal install never runs it. It is kept as a break-glass tool for the window before the controller is up, or if the controller isn't patching.

What it does: when a workload cluster is created, Spectro's spectro-drive spawns a capa-controller-manager in the cluster's cluster-XXX namespace with AWS_USE_FIPS_ENDPOINT=true hardcoded. The FIPS EKS endpoint (fips.eks.<region>.amazonaws.com) isn't in the VPC-endpoint cert SAN list, so TLS fails and CAPA hangs. The script overrides those deployments to AWS_USE_FIPS_ENDPOINT=false + AWS_ENDPOINT_URL_EKS=https://eks.<region>.amazonaws.com (idempotent). Run it on the bastion only after a cluster-XXX namespace with a capa-controller-manager exists:
sudo KUBECONFIG=/root/.kube/config AWS_DEFAULT_REGION=us-east-1 \
  /usr/local/bin/vertex-capa-patcher.sh
See Known Issues & Fixes for the underlying FIPS-endpoint bug.

Helm values (airgap overrides, consumed during Steps 5–7): cert-manager-values-airgap.yaml, image-swap-values-airgap.yaml, vertex-values-airgap.yaml.

Custom AWS OS Pack¶

A VerteX AWS EKS cluster profile needs an OS layer plus a kubernetes-eks layer. In the airgapped 4.8.40 install the OS layer for EKS — amazon-linux-eks — is served by the SaaS Palette Registry but absent from Artifact Studio's airgap bundle (spectro-cluster-packs.tar). Without it the OS-layer dropdown has no valid AWS EKS option, so the profile can't be completed. It was reverse-engineered from the ubuntu-aws-22.04 OS pack and mirrored into the airgap ECR Pack Registry.

Note: This is the OS layer only. The kubernetes-eks layer also needs value edits (OIDC, kube-proxy, access config, ECR registry rewrites) before an EKS cluster comes up cleanly — see Known Issues & Fixes.

What the pack is — a metadata-only OS pack (no AMI, no scripts). AMI selection happens at cluster-create via EKS managed node groups against AWS EKS-optimized AMIs, keyed off K8s version + region. Key pack.json fields: name: amazon-linux-eks, version: 1.0.0, layer: os, cloudTypes: ["eks"], osName: amzn, skipK8sInstall: true, sshUsername: ec2-user; depends on kubernetes-eks >= 1.27.0. The build output is a zstd-compressed OCI layout whose index.json carries two manifests — an archive manifest (→ ECR vertex-bootstrap/spectro-packs/archive/amazon-linux-eks:1.0.0, the pack the registry serves) and a bundle-definition manifest (→ …/spectro-packs/bundle-definitions/…, catalog metadata).

Build (needs python3, tar, gzip, zstd, sha256sum):

cd terraform/modules/vertex-ecr/custom-packs
./build-amazon-linux-eks.sh          # → amazon-linux-eks-1.0.0.zst

Deploy (airgap) — the custom pack ships via the custom-packs/ channel of the bundle bucket, separate from the Artifact Studio bundle:

# 1. upload the .zst
aws s3 cp amazon-linux-eks-1.0.0.zst s3://891377028731-vertex-bootstrap-bundle/custom-packs/

# 2. push into ECR from the bastion (idempotent; creates repos on demand)
sudo /usr/local/bin/vertex-cluster-packs-push.sh
tail -f /var/log/vertex-cluster-packs-push.log

vertex-cluster-packs-push.sh mirrors two channels — the Artifact Studio bundle (spectro-cluster-packs.tar) and the custom-packs/*.zst (where amazon-linux-eks lives) — and re-tags …/manifest → …/spectro-manifests/manifest so Specman's catalog resolves. After the push, amazon-linux-eks appears as an OS-layer option for AWS EKS profiles.

Note: If Cluster Profile OS dropdowns stay empty after a push, check that the re-tag step ran (see the log).