k

OpenTofu source for Talos/Kubernetes clusters on the homelab Proxmox, bootstrapped with Argo CD instead of Flux.

Layout

infra/
  main.tf              # shared root module: backend + provider config + call into ../talos-faktory
  variables.tf           # cluster_name, proxmox_nodes, nameservers, kubernetes_version, ...
  proxmox-ccm.tf          # writes proxmox-ccm's credential into OpenBao
  argo-bootstrap.tf        # renders each cluster's own argo baseline (skipped if bootstrap_argocd = false)
  argocd-cluster.tf         # every cluster writes its own Argo cluster-registration credential to OpenBao
  hub-bootstrap.tf           # hub only: the ApplicationSet + per-spoke cluster-registration secrets
  hub-essentials.tf           # hub only: remote external-secrets/proxmox-ccm/bao-secrets for bare_spokes
  <cluster>.tfvars              # per-cluster inputs -- usually just cluster_name
  <cluster>/                     # generated argo baseline + real per-cluster workloads (empty if bare)
    bootstrap/
      root.yaml                    # Argo Application: syncs <cluster>/manifests
    manifests/                    # everything Argo CD deploys on this cluster
      external-secrets.yaml
      proxmox-ccm.yaml
      bao-secrets.yaml              # wraps bao-secrets/ (see "CRD-installing components" below)
      bao-secrets/
      appset.yaml                    # hub only -- see "Hub and spoke"
      spokes/<spoke>/                 # hub only, bare spokes -- see "Bare spokes"
templates/argo/          # .tftpl sources for the generated files above
argocd/
  bootstrap/             # shared, cluster-agnostic
    namespace.yaml
    install.yaml          # patched upstream manifest: one-time, gets a minimal argo cd running
    project.yaml           # AppProject "homelab"
    argo-cd.yaml            # argo cd's *real*, ongoing desired state (see below)
  components/<name>/      # shared workloads, fanned out by the hub -- see "Hub and spoke" below
    manifests/               # the actual k8s manifests
    clusters/<cluster>.yaml    # opt-in marker + per-cluster override, hand-authored

One infra/ root module, one workspace per cluster (same pattern as talos-faktory itself). Every cluster (hub or spoke) is fully self-sufficient: its own standalone Argo CD, own bootstrap, own OpenBao auth -- nothing depends on the hub to function. Hub-spoke is an additional, optional layer on top (see below), not a replacement for that self-sufficiency.

argo-bootstrap.tf's local_file resources are the only ones that generate content instead of provisioning anything — they write <cluster>/bootstrap/root.yaml and every file under <cluster>/manifests/ from templates/argo/*.tftpl, parameterized by cluster_name. There's nothing hand-authored left under <cluster>/ at all now — real per-cluster workloads live in argocd/components/<name>/ instead (see "Hub and spoke").

How a cluster boots

1. ../talos-faktory (vendored locally, ours to modify) provisions the VMs and Talos/Kubernetes.
2. Its extra_manifests input is used (instead of enable_flux) to hand Talos five raw-URL manifests, applied once during cluster bootstrap, in order:
   • namespace.yaml — the argocd namespace
   • install.yaml — a patched upstream Argo CD manifest (every namespaced object explicitly carries namespace: argocd, since Talos applies these without -n). Just enough to get a minimal Argo CD running.
   • project.yaml — the shared homelab AppProject
   • argo-cd.yaml — an Argo Application describing Argo CD's real desired state (the official Helm chart, our tolerations, config). Once Argo CD is running it adopts and continuously reconciles itself against this object, so from here on, changing Argo CD's own config/version is a git commit away — install.yaml never needs a live kubectl apply again.
   • <cluster>/bootstrap/root.yaml — the cluster's root Application
3. root points at <cluster>/manifests in this repo (recursive directory sync) — drop manifests there and Argo picks them up, no further bootstrap needed.

Because Talos fetches those URLs live over HTTP during node bring-up, any bootstrap file change must be committed and pushed before running make apply for that cluster. root.yaml and project.yaml's own spec (not the things they point at) are the one part of this that's still genuinely one-time: changing either after a cluster already exists needs a live kubectl apply against that cluster (or a recreate) — same as upstream Argo CD's own bootstrap docs describe.

Adding a cluster

Just variables: write infra/<name>.tfvars (usually just cluster_name = "<name>" — see variables.tf for what else can be overridden), then:

make render  cluster=<name>   # generates <name>/bootstrap + <name>/manifests, nothing else
git add infra/<name> && git commit && git push   # talos fetches these over http during boot
make apply   cluster=<name>   # the real thing: provisions the cluster

make selects (creating if needed) the workspace matching cluster=, so state stays isolated per cluster in the same way tofu workspace select $(ws) does for talos-faktory itself. The render step exists because of the ordering constraint below — skipping straight to apply on a new cluster will fail partway through, since Talos will try to fetch <name>/bootstrap/root.yaml before it exists upstream.

make destroy also deletes the generated <name>/bootstrap + <name>/manifests files from disk (they're tracked as local_file resources in the same state) — git checkout -- infra/<name> or another make render brings them back before the next apply.

State & secrets

State is remote (S3-compatible, same bucket as talos-faktory, workspace-per-cluster under key k/terraform.tfstate). Provider credentials come from ~/.env (Proxmox, Unifi, AWS); the Makefile sources it and exports the TF_VAR_* terraform needs. tofu apply writes <cluster>.kubeconfig / <cluster>.talosconfig into infra/ — both gitignored, never commit them. This repo is public: no secrets in manifests either. Where an Argo-deployed component needs a credential (e.g. proxmox-ccm's Proxmox API token), external-secrets pulls it from OpenBao (<cluster>/manifests/bao-secrets/, <cluster>/manifests/external-secrets.yaml) via the kubernetes-auth backend talos-faktory's bao.tf already provisions per cluster at path <cluster_name> with read/list on pvekv/*. Terraform's only job is writing the credential itself into OpenBao (infra/proxmox-ccm.tf, via vault_generic_secret) — to an isolated pvekv/<cluster_name>/... key, never the shared production entries other clusters read.

Adding a CRD-installing component

If a component you're adding installs its own CRDs (like external-secrets) and something else needs an instance of one of those CRDs (like the ClusterSecretStore external-secrets consumes), don't put the CRD-consuming objects in root's own directly-synced manifests/ tree alongside the Application that installs the CRDs. root's directory sync validates every resource's kind against the API server before applying anything, and aborts the entire sync the instant one kind isn't discoverable yet — including the Application that would've installed it, since that's bundled in the same all-or-nothing batch. Sync-wave ordering doesn't help: it only gates on wave completion within one Application's own sync, not on another Application's independent, asynchronous reconciliation.

The fix (see <cluster>/manifests/bao-secrets/ + <cluster>/manifests/bao-secrets.yaml): put the CRD-consuming objects in their own subdirectory, wrap that subdirectory in its own Application, and exclude the subdirectory from root's own recursion (<cluster>/bootstrap/root.yaml's directory.exclude). That Application then retries on its own independent schedule until the CRD exists, instead of deadlocking everything else root manages.

Hub and spoke

Optional, additional layer on top of every cluster's own self-sufficient setup above. One cluster (by convention, and in this repo literally, named hub) runs an ApplicationSet (templates/argo/appset.yaml.tftpl, generated only when a cluster's tfvars set role = "hub") that fans a shared component library out to whichever clusters opt in.

Registering a spoke (infra/hub.tfvars's spokes list): every cluster, regardless of role, writes its own Argo CD cluster-registration credential into OpenBao when provisioned (infra/argocd-cluster.tf -- derives the specific fields Argo's cluster Secret format needs, tlsClientConfig.{caData,certData,keyData}, from module.cluster.kubeconfig via yamldecode/jsonencode; never the raw kubeconfig, never committed anywhere). For each name in spokes, the hub generates an ExternalSecret (infra/hub-bootstrap.tf, placed in manifests/bao-secrets/ since it hits the exact same CRD-ordering problem external-secrets' own objects do -- see "Adding a CRD-installing component") that reads that credential back out and labels it argocd.argoproj.io/secret-type: cluster, which is how Argo CD recognizes a Secret as a registered remote destination. No manual kubectl cluster-add step, ever -- adding a spoke is spokes = [..., "newcluster"] plus make render/apply on the hub.

Enabling a component on a cluster: drop argocd/components/<name>/clusters/<cluster>.yaml (just two fields: component:, cluster: -- the file's existence is the opt-in, its content is what the ApplicationSet's git generator reads to know what to template, so there's no path-parsing to keep in sync if the layout ever changes). That's it -- no tofu apply, no Argo CD config change, the same way adding a Kustomization to a Flux cluster is just a git commit.

Why this doesn't lock you in: a spoke's own bootstrap (Argo CD, external-secrets, proxmox-ccm) never depends on the hub existing or being reachable. If the hub goes away, every spoke keeps running exactly what it already had; you only lose the ability to add or change shared components until it's back. Going the other way -- a pure standalone cluster becoming a spoke -- is also just config: it already wrote its own credential to OpenBao the moment it was created, so registering it later needs nothing new from that cluster, only an addition to the hub's spokes list.

Helm-chart shared components

Real services (migrated so far from isengard's Flux setup: prometheus, alloy -- k8s/infrastructure/base/{prometheus,alloy}/helm.yaml there) use a second, separate hub-only ApplicationSet (templates/argo/appset-helm.yaml.tftpl, helm-components) rather than the marker-file one: that one suits simple manifest components with no per-cluster variation and no fan-out (a cluster opts itself in via one marker file's mere existence); this one is for real Helm services that want base+per-cluster-override values and can target several clusters from a single descriptor -- same git+matrix+elementsYaml pattern as isengard's own (not-yet-live) argo/appsets/helm.yaml: a git generator emits one element per argocd/components/helm/<name>.yaml descriptor, and a nested list generator's elementsYaml dynamically builds its fan-out list from that same descriptor's own clusters: field, so each (component, cluster) pair becomes one Application.

argocd/components/
  helm/<name>.yaml                       # descriptor: chart/version/namespace/clusters
  values/<name>.yaml                      # base helm values, shared by every cluster
  clusters/<cluster>/values/<name>.yaml     # optional per-cluster override

Enabling a component on another cluster is - cluster: <name> added to its descriptor's clusters: list -- no tofu apply, no Argo CD config change, the same way adding a Kustomization to a Flux cluster is just a git commit.

Two real gotchas hit deploying prometheus/alloy to clusterc, worth knowing before adding the next component:

• The git generator's file glob (argocd/components/helm/*.yaml) matches recursively into subdirectories despite the bare * -- confirmed it was also feeding values/<name>.yaml's own content into the same pipeline (which has no clusters field, so the fan-out step failed with "map has no entry for key clusters"). values/ and clusters/ live as siblings of helm/, not nested under it, so the descriptor glob can't reach them.
• A component's namespaceLabels (e.g. prometheus's node-exporter DaemonSet uses hostNetwork/hostPID/hostPath/hostPort, which needs pod-security.kubernetes.io/enforce: privileged or the "baseline" PSA default rejects every pod at admission) can't be set via {{ .namespaceLabels | toJson }} as a field value -- ApplicationSet substitutes Go-template expressions inside already-typed struct fields, it can't inject a whole dynamic map from one expression. templatePatch (a separate field, rendered as text and merged over the template) with a range $k, $v := .namespaceLabels does work, matching isengard's own proven pattern.

Bare spokes (no local Argo CD at all)

A cluster can skip self-bootstrapping Argo CD entirely: bootstrap_argocd = false in its own tfvars empties out extra_manifests (no namespace/install/project/ argo-cd/root) and skips every argo-bootstrap.tf file -- the cluster comes up as a plain Talos/Kubernetes cluster with nothing running beyond what talos-faktory itself provisions (its own OpenBao kubernetes-auth backend, its own cluster-registration credential -- both unconditional, independent of role or bootstrap_argocd). It has no way to deploy anything to itself until some hub registers it.

For each name in a hub's bare_spokes (a subset of spokes -- registration is still required for the destination lookup to resolve), hub-essentials.tf generates the same external-secrets/proxmox-ccm/bao-secrets Applications a self-bootstrapping cluster would generate for itself, just targeting that spoke as a remote destination. The external-secrets/proxmox-ccm Application objects live under this hub's own manifests/spokes/<spoke>/; their raw ClusterSecretStore/ExternalSecret manifests nest under manifests/bao-secrets/<spoke>/ instead (alongside the hub's own copy) -- root's exclude is a single, plain bao-secrets/** anchored at that top-level name, and neither a comma-separated list of two patterns nor a leading **/ wildcard (to reach a nested spokes/<spoke>/bao-secrets/ path) actually worked when tried, so every per-cluster bundle nests inside the one folder that's already proven to be excluded rather than asking exclude to reach further down the tree. This is deliberately separate from the argocd/components/ + ApplicationSet mechanism above: these need per-cluster templated values (the OpenBao auth mountPath/role) and their own guaranteed-independent Applications (same CRD-ordering reasoning as "Adding a CRD-installing component"), where the marker-file system is built for uniform workloads with no per-cluster variation.

Cloud-provider taint

Every node boots with kubelet --cloud-provider=external (set by talos-faktory), which taints it node.cloudprovider.kubernetes.io/uninitialized:NoSchedule until proxmox-ccm's node-lifecycle controller clears it. Argo CD's own workloads (argocd/bootstrap/install.yaml for the initial bootstrap window, argo-cd.yaml for its ongoing self-managed state) and proxmox-ccm itself (<cluster>/manifests/proxmox-ccm.yaml) both carry an explicit toleration for this so they can schedule and break the chicken-and-egg — the upstream proxmox-ccm chart doesn't enable this toleration by default. Ordinary application workloads intentionally do not get it, so they wait for the cluster's cloud-provider integration to actually be ready before landing on a node.

Networking

Each cluster takes one subnet_octet, which derives its 10.77.X.0/24 subnet, gateway/DNS address (10.77.X.1), and VLAN ID (X). Nodes use the reserved first quarter, UniFi DHCP uses .64-.127, and MetalLB uses .128-.254.

MetalLB + DNS-SD for cluster Prometheus targets

MetalLB is configured automatically from the cluster's subnet_octet. Setting that variable does three things:

• Feeds talos-faktory's own unifi_static_route (network.tf), routing that block to the cluster.
• Enables metallb-<cluster> (argocd/components/helm/metallb.yaml), the chart itself, through the same helm-components mechanism as prometheus/alloy. The IPAddressPool/L2Advertisement CRs (need the actual per-cluster address range, which the shared-component layer has no way to parameterize) are instead generated by Terraform (infra/metallb.tf, templates/argo/metallb-pool.yaml.tftpl) straight into the cluster's own bao-secrets/ directory — reusing that directory's already-proven CRD-ordering-safe Application rather than inventing a new one. For a bare spoke, the hub generates it remotely instead (bare_spoke_metallb_cidrs, keyed by cluster name, must be kept in sync with that spoke's own metallb_cidr — no cross-workspace state sharing in this repo, so it's plain duplication instead).
• Pins that cluster's prometheus to a specific LoadBalancer IP — the first address in its own metallb range (cidrhost(var.metallb_cidr, 1)) — via a Terraform-generated per-cluster values override (argocd/components/clusters/<cluster>/values/prometheus.yaml). Pinning it (rather than letting MetalLB auto-assign) means the DNS records below can be written at plan time, no live service-status read needed.

A hub deploying a shared component to itself (see hub.tfvars) is the one exception to "the values-override folder name matches cluster_name": a hub's own ApplicationSet reaches its own cluster through Argo CD's built-in in-cluster destination name, never through the name it's registered under for other clusters (that name only exists so other hubs/spokes can reach it). component_destination_name in infra/metallb.tf picks the right one; the component descriptors' own clusters: list needs - cluster: in-cluster for the hub's own entry, not - cluster: hub.

Terraform then writes two unifi_dns_record resources (infra/metallb.tf): an A record prometheus-<cluster>.r.ss pointing at that IP, and an SRV record _prometheus._tcp.r.ss (port 9090) pointing at that A name — reusing the exact SRV name isengard's own node_exporter_dns_sd job already queries for node-level metrics (deliberate call: simpler naming beats keeping the two scrape targets cleanly separated, since dns_sd_configs preserves each record's own port regardless of which job queries the name). The practical effect: the central Prometheus server's existing dns_sd_configs job picks up every cluster's own Prometheus automatically the moment a new cluster sets metallb_cidr and applies — no change needed on the central-server side at all.

On-demand nodes (experimental — spike/capi-lab-worker-split, unmerged)

There's no official cluster-autoscaler cloud provider for bare Proxmox. The real path is Cluster API (CAPI): a MachineDeployment that cluster-autoscaler's generic clusterapi backend can scale, backed by CAPMOX (the Proxmox infrastructure provider) and Sidero's CABPT (the Talos bootstrap provider). Adopting CAPI wholesale — control plane and workers both provisioned by it, the way every public example does it — would be a full architecture swap away from this repo's Tofu-native model. This branch instead implements a hybrid split, gated behind a single on_demand_nodes boolean tfvar (default false): control plane stays exactly as-is (talos-faktory, Tofu, unchanged), and only a worker MachineDeployment is handed to CAPI. Validated end to end on lab more than once — a CAPI/CAPMOX-provisioned VM booted, ran the same machine config talos-faktory itself would have generated, and joined lab's existing, Tofu-created control plane as a real Ready node, including across both Proxmox nodes in the same run.

flowchart TB
    subgraph tofu["OpenTofu"]
        TF["talos-faktory module<br/>(lab workspace)"]
        ODN["infra/on-demand-nodes.tf<br/>(var.on_demand_nodes)"]
        HODN["infra/hub-on-demand-nodes.tf<br/>(hub, for bare spokes)"]
    end

    subgraph proxmox["Proxmox (pve0/pve1, baelor = shared NFS)"]
        CP["talos-lab-cp-0<br/>control plane VM"]
        W0["talos-lab-0/1<br/>Tofu-provisioned workers"]
        TMPL["talos-lab-capi-template<br/>VM (template=true, disk on baelor)"]
        NEWVM["CAPMOX-cloned worker VM<br/>(any allowed node)"]
    end

    subgraph argo["Argo CD (hub, targeting lab remotely)"]
        HELM["cert-manager + capi-operator<br/>Helm components"]
        PROV["providers layer:<br/>CoreProvider/BootstrapProvider/<br/>InfrastructureProvider/IPAMProvider<br/>+ CAPMOX credential ExternalSecret"]
        WORK["workload layer:<br/>Cluster/ProxmoxCluster (no controlPlaneRef)<br/>ProxmoxMachineTemplate/MachineDeployment"]
    end

    TF -->|creates + bootstraps| CP
    TF -->|creates + bootstraps| W0
    ODN -->|builds template, reserves IPs,<br/>writes CAPMOX credential to OpenBao| TMPL
    ODN -->|"local-exec: applies TalosConfigTemplate<br/>directly (never committed to git)"| WORK
    HODN -->|generates same manifests remotely<br/>for bare spokes like lab| PROV
    HODN --> WORK

    HELM -->|installs| PROV
    PROV -->|installs real controllers,<br/>reads credential via ExternalSecret| WORK
    WORK -->|"IPAM assigns IP,<br/>CAPMOX clones template"| TMPL
    TMPL -.clone.-> NEWVM
    WORK -->|passes through pre-rendered<br/>worker config as boot data| NEWVM
    NEWVM -->|joins via existing<br/>cluster secrets, no CABPT-owned PKI| CP

    style tofu fill:#1a3a1a,stroke:#4a9,color:#eee
    style proxmox fill:#3a2a1a,stroke:#a84,color:#eee
    style argo fill:#1a2a3a,stroke:#48a,color:#eee

Why this shape, not vendored manifests

The CAPI controller stack (cert-manager, CAPI core, CABPT, CAPMOX, the in-cluster IPAM provider) isn't distributed as Helm charts upstream — clusterctl works by fetching and envsubst-ing large static manifest bundles. Vendoring those directly into this repo was seriously considered and measured: 43,506 lines, versus 28,352 lines for the entire rest of this repo combined, plus a three-layer CRD-ordering chain (cert-manager → {core/CABPT/CAPMOX/IPAM} → the actual Cluster/MachineDeployment instances) since all four CAPI bundles create their own Issuer/Certificate objects for webhook certs.

Instead: cert-manager and kubernetes-sigs/cluster-api-operator both have real, current, first-party Helm charts, added as two new shared components (argocd/components/helm/{cert-manager,capi-operator}.yaml, opted into lab the same manual way every other shared component works). cluster-api-operator manages the actual CAPI providers declaratively: you create small CoreProvider/BootstrapProvider/InfrastructureProvider/ IPAMProvider CRs (~10 lines each, version pin + for CAPMOX a configSecret reference) and it fetches/installs the real controllers itself, at runtime, inside the cluster — the same way Argo already fetches Helm charts live rather than vendoring them. Net result: ~150 lines of new Tofu-templated YAML instead of 43.5K vendored ones.

This still needs the "CRD-ordering" treatment documented above (a resource referencing a CRD that isn't discoverable yet aborts an entire sync batch, confirmed to not resolve via sync-waves across separate Applications) — now three shallow layers instead of vendored bundles:

1. Helm components (cert-manager, capi-operator) — each already its own independent, self-retrying Application via the existing helm-components ApplicationSet, so this layer needs no special handling.
2. Providers layer (templates/argo/on-demand-nodes-providers*.tftpl) — the CR instances above. Needs layer 1's CRDs.
3. Workload layer (templates/argo/on-demand-nodes-workload*.tftpl) — Cluster/ProxmoxCluster/ProxmoxMachineTemplate/MachineDeployment. Needs layer 2's CRs to have actually finished installing CAPI-core/CAPMOX's own CRDs, one link further down the same chain.

Both layer 2 and 3 nest under the existing bao-secrets/ folder (e.g. <hub>/manifests/bao-secrets/lab/on-demand-nodes/{providers,workload}/) — root's own exclude only reliably works as that one top-level glob (see "Adding a CRD-installing component" above), so anything needing the independently-retrying-Application treatment reuses it rather than getting a path of its own, however unrelated the name.

Bare spokes (lab's actual situation)

lab is a bare spoke (bootstrap_argocd = false, no local Argo CD), so the Argo manifests above can't be generated into a local <cluster>/manifests/ tree the way a self-bootstrapping cluster's on_demand_nodes = true would — they have to come from a hub instead, exactly mirroring hub-essentials.tf's existing pattern for bare spokes. infra/hub-on-demand-nodes.tf generates the same providers/workload manifests remotely for each entry in a new on_demand_node_spokes map (hub.tfvars), targeting that spoke as the Argo destination.

The Proxmox-level and Vault-level resources (template VM, reserved IPs, CAPMOX's OpenBao credential) still belong to lab's own Tofu workspace (infra/on-demand-nodes.tf, gated by on_demand_nodes = true on lab.tfvars) — they're tied to that specific cluster regardless of who generates its manifests. Since there's no cross-workspace state sharing in this repo, the concrete values (on_demand_template_vmid, worker_ips, control_plane_nodes[0]) are hand-copied from lab's own tofu output into hub.tfvars' on_demand_node_spokes entry — same duplication tradeoff already accepted for bare_spoke_metallb_cidrs.

The one thing that can't be GitOps-managed

TalosConfigTemplate.spec.data (CABPT's generateType: none passthrough, same mechanism as before) is a plain inline string in the CRD schema — no secret-reference field exists (confirmed by reading CABPT's actual Go types), so there's no way to keep the real worker join config out of git and have Argo apply this one object. infra/on-demand-nodes.tf applies it directly instead, via a local-exec provisioner that waits for CABPT's CRD to exist (polls up to ~30 min, since the whole chain in front of it is async GitOps reconciliation) then runs infra/scripts/apply-on-demand-talosconfig.py — a small standalone script (no secrets baked into the file itself, takes everything via environment variables) that splices lab's real worker_machine_configuration into the manifest and kubectl applys it. Never committed, same spirit as this repo's "no secrets in manifests" rule (that rule is about what's committed; this is the one object that structurally can't be).

CAPMOX's memory scheduler check (fixed)

Its default schedulerHints.memoryAdjustment (100) sums every non-template VM's configured memory on a node — running or stopped — to decide what's "reservable" (internal/service/scheduler/vmscheduler.go, GetReservableMemoryBytes). A node with enough large stopped scratch VMs can have that sum permanently exceed its physical total, pinning reservable memory at exactly 0 regardless of what's actually running or how much real free RAM exists. schedulerHints.memoryAdjustment: 0 on the ProxmoxCluster (set in the workload template) disables the check entirely — CAPMOX's own documented escape hatch, and the actual fix used here.

Known upstream bug: CAPMOX skips provisioning steps when reconciling fast (worked around)

CAPMOX's ProxmoxMachine reconciler (ReconcileVM in internal/service/vmservice/vm.go) is a single, straight-line sequential chain per reconcile call: clone → configure → disks → IP allocation → bootstrap data injection → power on → cloud-init check. Each step's own guard checks whether the preceding step left the condition in exactly the reason it expects before doing real work — but a guard mismatch returns (false, nil), which the caller can't distinguish from "nothing to do, carry on." When the whole provider stack is already warm (every scale-up after the initial install — the first spike test, with providers freshly installing via clusterctl init, had enough natural delay between states that this never surfaced), something early in the chain doesn't leave its expected reason in time, and the rest of the guards silently no-op in cascade, straight through to a false terminal "ready" condition in one pass — skipping IP allocation, bootstrap data injection, and power-on entirely. Confirmed via status.bootstrapDataProvided staying unset, zero IPAddressClaim objects, and zero qmstart Proxmox tasks; reproduced identically on a from-scratch Machine recreation, so it's deterministic once the providers are warm, not a one-off race. Corroborated by other users hitting the identical symptom upstream. Checked both an older pre-refactor CAPMOX version (predates this condition-gated state machine entirely, introduced ~March 2026, but crosses a v1alpha1→v1alpha2 API break) and the newest unreleased commit (a real fix, but for a different scenario — Velero-restore status reconstruction, not this) — neither was a viable fix.

Worked around with a reconciliation watchdog (templates/argo/on-demand-nodes-workload.yaml.tftpl's CronJob, every 2 minutes): detects a ProxmoxMachine whose VirtualMachineProvisioned condition reports a terminal reason while status.bootstrapDataProvided was never set (the telltale signature, distinct from any legitimate in-progress WaitingForX state — a 180s grace period avoids false positives on the normal happy path), then Server-Side Apply-patches the condition's reason back to WaitingForStaticIPAllocation with --force-conflicts (needed — CAPMOX's own controller owns that field via SSA, so the override has to be explicit; CAPMOX's CRD declares x-kubernetes-list-type: map on status.conditions keyed by type, so this merges just the one condition without touching the others or needing to know its array index). This makes CAPMOX's own next reconcile actually execute the skipped steps using its own real logic, rather than reimplementing them externally. Validated live: deliberately reproduced the bug, manually triggered the watchdog CronJob as a one-off Job, confirmed bootstrapDataProvided flipped to true, the VM picked up its correct IPAM-assigned static IP (not a DHCP fallback), and the node joined lab as Ready.

cluster-autoscaler

kubernetes/autoscaler's own chart, a third new shared Helm component (argocd/components/helm/cluster-autoscaler.yaml), self-hosted in the same topology as everything else here: clusterAPIMode: incluster-incluster (the chart's own default) since the CAPI management objects and the actual workload nodes both live on lab — no separate kubeconfig/Secret needed for either side, both use the in-cluster service account. autoDiscovery.namespace: on-demand-nodes scopes it to --node-group-auto-discovery=clusterapi:namespace=on-demand-nodes.

Pinned to 9.46.6 (appVersion 1.32.0), not chart latest (9.59.0, appVersion 1.35.0) — this matters, not just hygiene. Confirmed live: 1.35.0 hardcodes DRA (Dynamic Resource Allocation) handling on (--enable-dynamic-resource-allocation is documented as "locked to true" in that version's --help), which made the controller block forever waiting for DeviceClass/ResourceClaim/ResourceSlice informer caches that never sync since lab's Kubernetes v1.32.3 doesn't serve those APIs — zero scan-loop iterations, zero scale-up decisions, and critically no visible error (it just silently never got past its first log line). 9.46.6 matches lab's own Kubernetes minor version, which is upstream's own stated compatibility guidance for this project, and has DRA genuinely optional. Worth remembering if this cluster's Kubernetes version ever moves and the chart version doesn't move with it.

MachineDeployment annotations (templates/argo/on-demand-nodes-workload.yaml.tftpl): cluster.x-k8s.io/cluster-api-autoscaler-node-group-{min,max}-size (min 0, max = the reserved IP pool size, computed in hub-on-demand-nodes.tf as length(worker_ips) — that pool is the hard ceiling regardless of what the autoscaler wants). CAPMOX has no opt-in "scale from zero" support of its own, so the capacity.cluster-autoscaler.kubernetes.io/{cpu,memory,ephemeral-disk} annotations are load-bearing, not optional — without them the autoscaler has no way to know what a node from an empty group would look like, and these have to be kept in sync with the ProxmoxMachineTemplate spec by hand (both hardcoded in the same file). The rbac.additionalRules chart value adds get/list/watch on proxmoxmachinetemplates per the scale-from-zero RBAC prerequisite the clusterapi provider docs call out.

Argo/autoscaler ownership conflict, same shape as the watchdog's selfHeal fight: since cluster-autoscaler now owns MachineDeployment.spec.replicas, on-demand-nodes-workload-app.yaml.tftpl needed ignoreDifferences on that one field — without it, Argo's selfHeal reverts every scale-up/down back to the committed replicas: 0 baseline within one sync cycle (confirmed live, same failure mode hit earlier during the watchdog's own validation).

Validated fully live, with zero manual intervention at any step: deployed a Deployment with pod resource requests exceeding lab's existing free capacity → cluster-autoscaler detected the Pending pods, discovered the MachineDeployment as a node group (min: 0, max: 2, replicas: 0), decided to scale up, and patched machinedeployments/lab-workers/scale from 0→1 entirely on its own → CAPI created the Machine → CAPMOX hit the exact warm-stack bug described above (reproduced identically a third time) → the watchdog CronJob fired on its own 2-minute schedule (not manually triggered) and recovered it → the node joined lab as Ready with its correct IPAM-assigned static IP → the pending pod scheduled onto it. The full loop, start to finish, with nothing hand-driven except creating the demand in the first place.

Status: on_demand_nodes end to end — Helm components, providers layer, workload layer, the CAPMOX warm-stack watchdog, and cluster-autoscaler — is complete and validated live, including a real, autoscaler-triggered scale-up recovering automatically from the known CAPMOX bug with no manual intervention. lab.tfvars has on_demand_nodes = true and its Tofu-managed resources (template VM, reserved IPs, OpenBao credential) exist right now; the Argo-delivered pieces were torn down after testing (this branch is intentionally unmerged, so master's hub stays on its own, unaffected sync — the on-demand-nodes and cluster-autoscaler Applications only existed while root's targetRevision was temporarily, manually pointed at this branch for the test, since neither the wrapper Applications nor the ApplicationSet's own ref: values git source can otherwise see files that only exist on an unmerged branch). Nothing left to build for the core feature — what's left is deciding when (or whether) to merge, tuning autoscaler behavior (scale-down timing, --balance-similar-node-groups if a second node group is ever added), and optionally pursuing a real fix for the CAPMOX bug upstream instead of leaning on the watchdog indefinitely.