RA-06. Kubernetes Service - HA cluster in a VPC tier

The management path

The operator uses kubectl and SSH against public IP A. The Kubernetes API is published on 6443 across the control plane, so a control node can drop out without breaking kubectl. SSH is published as port forwards from 2222 up, one per node, all landing on port 22 inside the tier - log in as user cloud with the SSH key pair named at cluster creation.

Once the cluster is provisioned, the kubeconfig for this API endpoint is available for download on the cluster's detail page in the portal - see accessing the cluster - and from the getKubernetesClusterConfig API, as the CloudMonkey section shows. That file is the credential for the management path: distribute it as deliberately as you would an SSH key.

The traffic path

End users reach public IP B. Load balancer rules send 80 and 443 to NodePorts 30080 and 30443 on the workers, where the ingress controller terminates the request and routes it by hostname to the right Service. The VPC router applies the tier's ingress ACL on the way in.

In-tier traffic

The whole cluster lives in the one k8s-cluster tier, so scheduling, etcd replication and kubelet traffic never leave the tier - and, as RA-02 explains, traffic between VMs in the same tier never reaches the VPC router. The ACL governs what enters and leaves the cluster, not what the cluster says to itself. Isolation between workloads inside the cluster is Kubernetes' job, via NetworkPolicy.

Every node uses the same compute offering deliberately, so the portal build and the scripted builds below produce the same cluster. CloudStack 4.22 can size node types differently via the nodeofferings map - see the note in the CloudMonkey section - but a reference architecture should produce the same result every time it is built, whichever tool built it.

Check these against your account's resource limits before you start.

One ACL, k8s-cluster-acl. Rules are numbered in bands so there is room to insert without renumbering, and egress uses a separate band from ingress because rule numbers are unique per ACL list, not per direction. Anything not matched is denied.

Rule 140 is mostly precautionary: VM-to-VM traffic inside the tier never reaches the router, so the rule matters only for tier-addressed traffic that originates at the router, such as load balancer health checks.

Why a VPC tier and not an isolated network

CKS runs on an isolated network without any of this structure, and will provision API access for you there. The reason this architecture uses a VPC tier is the same reason RA-03 exists: the ACL at the virtual router, and the ability to add tiers later without re-platforming. When the cluster eventually needs a database VM that should not be a pod, it lands in an adjacent k8s-data tier with its own ACL, exactly as RA-03 places db01.

Why three control nodes

etcd needs a quorum, and a quorum of one is a single point of failure that also goes down for every upgrade. Two is worse than one for etcd - two nodes cannot form a majority if either fails. Three is the minimum that tolerates one node being down, whether from failure or planned maintenance. This matches the standing recommendation on the Kubernetes Service page: dev/test clusters can run one control node, production runs at least three.

Why an explicit load balancer to fixed NodePorts, not Service type LoadBalancer

The CloudStack cloud provider inside the cluster can create load balancer rules on your behalf when a Service of type LoadBalancer is declared, but the behaviour depends on the cluster's network type and release. This architecture instead pins the ingress controller to NodePorts 30080 and 30443 and declares the load balancer rules in the plan, because a reference architecture should produce the same result every time it is built. If you later verify the in-cluster provider works against your VPC tier, adopting it is an additive change, not a rebuild.

Stateful workloads

This architecture makes no assumption about persistent volume provisioning. Treat the cluster as stateless: databases belong on a VM in an adjacent tier (the RA-03 pattern), and object storage belongs in the S3 service. If you need in-cluster PersistentVolumes backed by cloud volumes, that is a storage driver decision to make deliberately, not a default - CKS exposes it as the enablecsi option at cluster creation.

1. Register your SSH key. Compute, SSH Key Pairs, Create SSH key pair. Name it ops-key and paste your public key. CKS installs it on every node it builds, for user cloud.
2. Create the VPC. Network, VPC, Add VPC. Name k8s, CIDR 10.40.0.0/16, offering vpc-std-v1, zone au-mel01.
3. Create the ACL before the tier. Open the VPC, Network ACL Lists, Add ACL List. Name k8s-cluster-acl, then add the six rules from the plan above.
   Create the ACL first and attach it when you create the tier. A tier created without one gets no usable ACL: through the API the tier is left with no ACL list at all (ingress blocked; egress open only because the tier offerings set egressdefaultpolicy=true), and the portal steers you to the built-in default_deny. CKS refuses to build on a default_deny tier, and the refusal surfaces as a generic HTTP 530 Internal error - the real message appears only in the management server log.
4. Create the tier. Within the VPC, add network tier k8s-cluster, gateway 10.40.10.1, netmask 255.255.255.0, ACL k8s-cluster-acl.
5. Create the cluster. Containers, Kubernetes, Create Cluster. Name prod01, Kubernetes version 1.35.4, control node count 3 and worker count 2, both with offering s-medium-gen2, node root disk 32 GB, SSH key pair ops-key, network: the k8s-cluster tier. Creation takes a few minutes while the VMs provision and the cluster bootstraps.
6. Download the kubeconfig from the cluster detail page and confirm kubectl get nodes shows five nodes Ready. If kubectl times out, do step 7 first - the API is only reachable once the 6443 rule exists.
7. Check what CKS published. CKS publishes the API and SSH rules on the VPC's source NAT address - that address is Public IP A. Reconcile it against the plan: 6443 to the control plane (a load balancer rule across the three control nodes where the release creates one, otherwise a port forward) and SSH 2222 and up, one per node. Add anything missing rather than duplicating.
8. Acquire Public IP B on the k8s-cluster tier and create two load balancer rules: 80 → 30080 and 443 → 30443, algorithm round robin, assigned to the two worker VMs.
9. Deploy an ingress controller pinned to those NodePorts, for example ingress-nginx with service.type=NodePort, nodePorts.http=30080, nodePorts.https=30443.

Same requirements as RA-01. The script builds everything up to and including the cluster, then the application load balancer on Public IP B. Public IP A is deliberately not scripted - CKS publishes those rules on the source NAT address itself; reconcile them per step 7 of the walkthrough.

#!/usr/bin/env bash
# ra06-kubernetes-vpc.sh - deploy RA-06
set -euo pipefail

ADMIN=203.0.113.4/32   # your admin CIDR
K8S_VERSION=1.35.4

# 1. Register your SSH key
cmk register sshkeypair name=ops-key publickey="$(cat ~/.ssh/id_ed25519.pub)"

# 2. Create the VPC
ZONE=$(cmk list zones name=au-mel01 filter=id | jq -r '.zone[0].id')
VPCOFF=$(cmk list vpcofferings name=vpc-std-v1 filter=id | jq -r '.vpcoffering[0].id')
VPC=$(cmk create vpc name=k8s displaytext=k8s zoneid=$ZONE \
  cidr=10.40.0.0/16 vpcofferingid=$VPCOFF | jq -r '.vpc.id')

# 3. Create the ACL and its rules. CKS refuses to build on a tier using the
# default_deny ACL (as a generic HTTP 530), so the custom ACL is created
# first and attached to the tier at creation.
ACL=$(cmk create networkacllist name=k8s-cluster-acl \
  description="CKS cluster tier ACL" vpcid=$VPC | jq -r '.networkacllist.id')

rule () {  # number ingress|egress proto startport endport cidr
  cmk create networkacl aclid=$ACL number=$1 action=allow traffictype=$2 \
    protocol=$3 startport=$4 endport=$5 cidrlist=$6
}

rule 100 ingress tcp 6443  6443  $ADMIN       # Kubernetes API via public IP A
rule 110 ingress tcp 22    22    $ADMIN       # node SSH, arrives via forwards on 2222+
rule 120 ingress tcp 30080 30080 0.0.0.0/0    # LB to the ingress NodePort, HTTP
rule 130 ingress tcp 30443 30443 0.0.0.0/0    # LB to the ingress NodePort, HTTPS
cmk create networkacl aclid=$ACL number=140 action=allow traffictype=ingress \
  protocol=all cidrlist=10.40.10.0/24         # intra-cluster via the router
cmk create networkacl aclid=$ACL number=200 action=allow traffictype=egress \
  protocol=all cidrlist=0.0.0.0/0             # image pulls, CKS bootstrap

# 4. Create the tier, with the ACL attached at creation
TIEROFF=$(cmk list networkofferings name=net-vpctier-std-v1 filter=id | jq -r '.networkoffering[0].id')
TIER=$(cmk create network name=k8s-cluster displaytext=k8s-cluster zoneid=$ZONE \
  networkofferingid=$TIEROFF vpcid=$VPC gateway=10.40.10.1 netmask=255.255.255.0 \
  aclid=$ACL | jq -r '.network.id')

# 5. Create the cluster
K8SVER=$(cmk list kubernetessupportedversions | jq -r --arg v "$K8S_VERSION" \
  '.kubernetessupportedversion[] | select(.semanticversion==$v) | .id')
SVCOFF=$(cmk list serviceofferings name=s-medium-gen2 filter=id | jq -r '.serviceoffering[0].id')

CLUSTER=$(cmk create kubernetescluster name=prod01 description="Production CKS" \
  zoneid=$ZONE kubernetesversionid=$K8SVER serviceofferingid=$SVCOFF \
  controlnodes=3 size=2 noderootdisksize=32 keypair=ops-key \
  networkid=$TIER | jq -r '.kubernetescluster.id')

# 6. The kubeconfig - the same file the portal offers on the cluster detail
# page. configdata is the plain-text kubeconfig. The call fails with "config
# not available at this moment" while the cluster is still Starting.
cmk get kubernetesclusterconfig id=$CLUSTER \
  | jq -r '.clusterconfig.configdata' > prod01.kubeconfig

# 8. Public IP B and the application load balancer
IPB=$(cmk associate ipaddress vpcid=$VPC networkid=$TIER | jq -r '.ipaddress.id')

lb () {  # publicport nodeport name
  cmk create loadbalancerrule name=$3 description=$3 \
    publicipid=$IPB networkid=$TIER algorithm=roundrobin protocol=tcp \
    publicport=$1 privateport=$2 | jq -r '.loadbalancer.id'
}
LBHTTP=$(lb 80  30080 ingress-http)
LBHTTPS=$(lb 443 30443 ingress-https)

# CKS names the node VMs itself. List them, then assign the two workers.
cmk list virtualmachines networkid=$TIER filter=id,name
cmk assign toloadbalancerrule id=$LBHTTP  virtualmachineids=<worker01-id>,<worker02-id>
cmk assign toloadbalancerrule id=$LBHTTPS virtualmachineids=<worker01-id>,<worker02-id>

Same requirements as RA-01.

ansible-galaxy collection install ngine_io.cloudstack --upgrade
pip install 'cs>=3.4.0' sshpubkeys

export CLOUDSTACK_ENDPOINT=https://cloud.lightspeedhosting.com.au/client/api
export CLOUDSTACK_KEY=<api key>
export CLOUDSTACK_SECRET=<secret key>

The collection covers the SSH key, VPC, ACL and tier, but has no module for CKS clusters, so the cluster step shells out to cmk behind a guard on the error output - re-runs tolerate an existing cluster, but there is no true state comparison.

---
# ra06-kubernetes-vpc.yml - deploy RA-06
- name: RA-06 Kubernetes Service in a VPC tier
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    zone: au-mel01
    admin_cidr: 203.0.113.4/32      # your admin CIDR
    tier_cidr: 10.40.10.0/24

    acl_rules:
      # Ingress - the 100 band
      - {n: 100, t: ingress, p: tcp, s: 6443,  e: 6443,  cidr: "{{ admin_cidr }}"}
      - {n: 110, t: ingress, p: tcp, s: 22,    e: 22,    cidr: "{{ admin_cidr }}"}
      - {n: 120, t: ingress, p: tcp, s: 30080, e: 30080, cidr: 0.0.0.0/0}
      - {n: 130, t: ingress, p: tcp, s: 30443, e: 30443, cidr: 0.0.0.0/0}
      - {n: 140, t: ingress, p: all,                     cidr: "{{ tier_cidr }}"}
      # Egress - the 200 band, because numbers are unique per ACL LIST, not
      # per direction, and an egress rule reusing an ingress number silently
      # rewrites it.
      - {n: 200, t: egress,  p: all,                     cidr: 0.0.0.0/0}

  tasks:
    - name: Register SSH key
      ngine_io.cloudstack.ssh_key:
        name: ops-key
        public_key: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"

    - name: VPC
      ngine_io.cloudstack.vpc:
        name: k8s
        cidr: 10.40.0.0/16
        vpc_offering: vpc-std-v1
        zone: "{{ zone }}"

    # The ACL list first - CKS refuses to build on a tier using default_deny,
    # which is where a tier created without an ACL can end up.
    - name: Network ACL list
      ngine_io.cloudstack.network_acl:
        name: k8s-cluster-acl
        vpc: k8s
        zone: "{{ zone }}"

    - name: Network ACL rules
      ngine_io.cloudstack.network_acl_rule:
        network_acl: k8s-cluster-acl
        vpc: k8s
        zone: "{{ zone }}"
        rule_position: "{{ item.n }}"
        traffic_type: "{{ item.t }}"
        protocol: "{{ item.p }}"
        start_port: "{{ item.s | default(omit) }}"
        end_port: "{{ item.e | default(omit) }}"
        cidrs: ["{{ item.cidr }}"]
        action_policy: allow
      loop: "{{ acl_rules }}"

    - name: Tier
      ngine_io.cloudstack.network:
        name: k8s-cluster
        vpc: k8s
        network_offering: net-vpctier-std-v1
        zone: "{{ zone }}"
        gateway: 10.40.10.1
        netmask: 255.255.255.0
        acl: k8s-cluster-acl
      register: tier

    # No collection module exists for CKS clusters, so this shells out.
    # zone_id, k8s_version_id and offering_id are UUIDs - resolve them with
    # the cmk list lookups from the CloudMonkey section and pass them in
    # with -e, or register them from api_request lookups.
    - name: CKS cluster (no collection module - via cmk)
      ansible.builtin.command: >
        cmk create kubernetescluster name=prod01 description="Production CKS"
        zoneid={{ zone_id }} kubernetesversionid={{ k8s_version_id }}
        serviceofferingid={{ offering_id }} controlnodes=3 size=2
        noderootdisksize=32 keypair=ops-key
        networkid={{ tier.id }}
      register: cks
      changed_when: cks.rc == 0
      failed_when: cks.rc != 0 and 'already exists' not in (cks.stderr + cks.stdout)

The play stops at the cluster deliberately. Public IP B and its load balancer rules need the worker VM names, and CKS generates those at creation time - assign them from the portal or CloudMonkey (walkthrough step 8) once the cluster reports Running.

Recent versions of the CloudStack Terraform provider include a Kubernetes cluster resource, so unlike Ansible the whole build to the cluster is declarative.

# ra06-kubernetes-vpc.tf - deploy RA-06
terraform {
  required_providers {
    cloudstack = {
      source  = "cloudstack/cloudstack"
      version = "~> 0.6.0"
    }
  }
}

variable "api_url"    { default = "https://cloud.lightspeedhosting.com.au/client/api" }
variable "api_key"    { sensitive = true }
variable "secret_key" { sensitive = true }
variable "admin_cidr" { default = "203.0.113.4/32" }

provider "cloudstack" {
  api_url    = var.api_url
  api_key    = var.api_key
  secret_key = var.secret_key
}

locals {
  zone      = "au-mel01"
  tier_cidr = "10.40.10.0/24"
}

resource "cloudstack_ssh_keypair" "ops" {
  name       = "ops-key"
  public_key = file("~/.ssh/id_ed25519.pub")
}

resource "cloudstack_vpc" "k8s" {
  name         = "k8s"
  display_text = "k8s"
  cidr         = "10.40.0.0/16"
  vpc_offering = "vpc-std-v1"
  zone         = local.zone
}

# The ACL is declared before the tier that references it. CKS refuses to
# build on a tier using the default_deny ACL - see the portal walkthrough.
resource "cloudstack_network_acl" "cluster" {
  name        = "k8s-cluster-acl"
  description = "CKS cluster tier"
  vpc_id      = cloudstack_vpc.k8s.id
}

# One rule block per port, mirroring the numbered ACL plan exactly.
# 0.6.0 removed the plural `ports` list; it remains in the schema so validate
# and plan pass, and apply fails. Use singular `port`, and set rule_number
# explicitly because rules are created concurrently.
resource "cloudstack_network_acl_rule" "cluster" {
  acl_id = cloudstack_network_acl.cluster.id

  rule {
    rule_number  = 100
    action       = "allow"
    cidr_list    = [var.admin_cidr]
    protocol     = "tcp"
    port         = "6443"
    traffic_type = "ingress"
  }

  rule {
    rule_number  = 110
    action       = "allow"
    cidr_list    = [var.admin_cidr]
    protocol     = "tcp"
    port         = "22"
    traffic_type = "ingress"
  }

  rule {
    rule_number  = 120
    action       = "allow"
    cidr_list    = ["0.0.0.0/0"]
    protocol     = "tcp"
    port         = "30080"
    traffic_type = "ingress"
  }

  rule {
    rule_number  = 130
    action       = "allow"
    cidr_list    = ["0.0.0.0/0"]
    protocol     = "tcp"
    port         = "30443"
    traffic_type = "ingress"
  }

  rule {
    rule_number  = 140
    action       = "allow"
    cidr_list    = [local.tier_cidr]
    protocol     = "all"
    traffic_type = "ingress"
  }

  rule {
    rule_number  = 200
    action       = "allow"
    cidr_list    = ["0.0.0.0/0"]
    protocol     = "all"
    traffic_type = "egress"
  }
}

resource "cloudstack_network" "cluster_tier" {
  name             = "k8s-cluster"
  display_text     = "k8s-cluster"
  cidr             = local.tier_cidr
  gateway          = "10.40.10.1"
  network_offering = "net-vpctier-std-v1"
  vpc_id           = cloudstack_vpc.k8s.id
  acl_id           = cloudstack_network_acl.cluster.id
  zone             = local.zone
}

resource "cloudstack_kubernetes_cluster" "prod01" {
  name               = "prod01"
  description        = "Production CKS"
  zone               = local.zone
  kubernetes_version = "1.35.4"
  service_offering   = "s-medium-gen2"
  control_nodes_size = 3
  size               = 2
  noderootdisksize   = 32
  keypair            = cloudstack_ssh_keypair.ops.name
  network_id         = cloudstack_network.cluster_tier.id
}

terraform init
terraform validate
terraform apply -var api_key=... -var secret_key=...

As in the Ansible version, Public IP B and its load balancer rules are left outside the configuration: cloudstack_loadbalancer_rule takes member_ids, and the worker VMs are created and named by CKS rather than by Terraform. Assign them from the portal or CloudMonkey (walkthrough step 8) once the cluster is Running.

Check the two paths independently - the failure argument only holds if each one is known to work on its own.

The management path

# Five nodes, all Ready, versions matching 1.35.4
kubectl get nodes

# CNI, kube-proxy and core components running on every node
kubectl get pods -n kube-system

# SSH to the first node - user is cloud, key is the ops-key pair
ssh -i ~/.ssh/id_ed25519 -p 2222 cloud@<public-ip-a>

Control node failure

Stop one control node VM from the portal, confirm kubectl get nodes still answers, start it again. This demonstrates the three-node quorum working, and it is worth observing once before relying on it during an upgrade. While you are in the portal, confirm the three control node VMs report different hypervisor hosts - see control plane placement under Day 2.

The traffic path

# Deploy the ingress controller, then a test workload with an Ingress
curl -H "Host: test.example.com" http://<public-ip-b>/

The split

From an address outside your admin range, confirm 6443 on Public IP A times out and 443 on Public IP B answers. That pair of results is the whole design working: management closed to the world, applications open to it.

• Scaling workers. Change the worker count on the cluster detail page - see scaling a cluster. New workers join the cluster automatically, but they do not join your load balancer rules: assign the new VM to both rules on Public IP B, and remove it before scaling down. This is the one place the explicit-LB decision costs you a manual step. CKS can also autoscale the worker pool (scaleKubernetesCluster with autoscalingenabled=true, applied after the cluster is Running - 4.22 has no creation-time autoscaling parameters); leave it off in this design, because autoscaled workers join the cluster but never join the load balancer rules.
• Control plane placement. CKS cannot apply anti-affinity groups at creation, so nothing guarantees the three control nodes land on different hypervisor hosts - and three on one host is a quorum lost to a single host failure. Check placement after creation, and if a pair shares a host, apply a host anti-affinity group by hand: updateVMAffinityGroup requires each VM to be stopped and restarted, so roll it one control node at a time and the API stays up throughout. Nodes added later by scaling do not inherit the group.
• Upgrades. One minor version at a time (1.34.x → 1.35.x), rolled node by node from the portal - see upgrading Kubernetes versions. With three control nodes the API stays up throughout; schedule it anyway for anything customer-facing, because workloads reschedule as each worker drains.
• Node sizing. Moving nodes to a larger offering is supported and one-directional. To go smaller, scale the worker count down and redeploy workers at the smaller offering instead.
• Stop and start. A cluster you are not using can be stopped from the portal; state is preserved, workloads are unreachable while stopped. See stopping and starting a cluster.
• Backups. The control plane's etcd state is managed by the service, but your manifests are your responsibility - keep them in git and treat the cluster as rebuildable. Anything stateful lives outside the cluster by design in this architecture, so it is backed up wherever it actually lives.
• Monitoring. Watch 6443 on Public IP A from your admin range and 443 on Public IP B from anywhere. Those two checks cover the failure argument this architecture is built on: the first tells you the control plane is healthy, the second tells you your users are being served, and they are allowed to fail independently.