RA-04. Highly Available Multi-Tier VPC - Public and Internal Load Balancers

Worth being precise about, because "highly available" gets used loosely and the gap between HA and disaster recovery is where outages become incidents.

This is high availability, not disaster recovery

Every resource on this page sits in au-mel01. The architecture removes the server as a single point of failure; it does not remove the zone. A datacentre-level event takes the whole deployment down regardless of how many nodes each tier has.

Disaster recovery is a separate design with its own decisions: a second zone or region, asynchronous replication of the database, a plan for how much data you can afford to lose and how long you can afford to be down, DNS with a low TTL, and a documented failover procedure that somebody has actually rehearsed. Building HA first is the right order - it's cheaper, it handles the failures you'll actually see most often, and it doesn't preclude DR later. Just don't mistake one for the other.

Traffic flow

Users reach the public load balancer VIP. The VPC router applies the prod-web ingress ACL and distributes connections across web01 and web02. The web servers call the app tier's internal load balancer at 10.1.2.10:8080 rather than any individual app server; that crosses the router and is checked by both tiers' ACLs, exactly as in RA-03. The app servers query the database through the db tier's internal load balancer at 10.1.3.10:3306.

The addressing matters: nothing ever names a specific server. Every hop targets a load balancer address, which is what lets you replace a node without touching configuration anywhere else.

Public and internal load balancers are different things

The public load balancer is a rule on the VPC router applied to a public IP. It costs nothing extra and is created with the same API as any other LB rule.

An internal load balancer is a separate appliance VM that the platform deploys into the tier and gives a guest IP from that tier's subnet. It's a different API, it consumes a small amount of tier address space and resource, and it's why the app and db tiers use net-vpctier-intlb-v1 rather than the standard tier offering.

Check these against your account's resource limits - this architecture roughly doubles the instance and volume count of RA-03.

Why the load balancer needs its own public IP

VPC tier offerings run with conserve mode off, which means a public IP carries one service. The VPC's source NAT IP is already doing source NAT, so the load balancer rule needs a second, dedicated public IP. You don't get to stack them.

Sizing under failure

Two nodes per tier only helps if one node can carry the load while the other is gone. If each web server runs at 70% at peak, losing one doesn't give you a degraded service, it gives you an outage with extra steps. Size each node so that a single node handles peak traffic, and treat the second as headroom rather than capacity you've already spent.

Putting a load balancer in front of two web servers is uncomplicated: the servers are stateless and interchangeable. Putting one in front of two database servers is not, and a reference architecture that glosses over this does you no favours.

The problem

With classic MySQL or MariaDB asynchronous replication, one server is the primary and accepts writes; the other is a replica and is read-only. If the load balancer round-robins connections across both, half your writes land on a read-only server and fail, and reads may return data the primary wrote moments ago but the replica hasn't applied yet.

A load balancer does not make a database highly available. It distributes connections. Whether that's safe depends entirely on the replication topology behind it.

Three ways to do this correctly

This page uses the first option, because it's the one that behaves predictably and is honest about where the manual step is. The internal LB gives the application a single stable address for the database, which is worth having on its own - it means promoting db02 is a load balancer change rather than a configuration change on every app server.

Health checks

A TCP health check on 3306 tells you the port is open, not that the database is healthy. A replica that has fallen hours behind, or a primary that has filled its disk, both pass a TCP check. Where the load balancer supports it, use an HTTP check against a small endpoint that queries the database and reports replication status. Where it doesn't, monitor replication lag separately and alert on it.

The rules are the same as RA-03, because the tier boundaries haven't changed - only the number of servers behind them. Everything is expressed as tier CIDRs, so adding the second server in each tier needs no rule changes at all.

Two additions specific to this architecture:

• Database replication needs no ACL rule. db01 and db02 are in the same subnet, so their replication traffic never reaches the router. Convenient here, but remember it also means the ACL isn't protecting them from each other.
• Internal load balancer traffic is ordinary tier traffic. The app tier's LB VIP is an address in 10.1.2.0/24, so the existing "TCP 8080 from 10.1.1.0/24" ingress rule already covers connections to it. No special rule is required for the appliance.

The full numbered rule tables are in RA-03, and the same "checked by two ACLs" behaviour applies to every cross-tier connection here. If you haven't read How ACLs are evaluated, read it before you start debugging anything.

1. SSH key and anti-affinity groups

Compute → SSH key pairs → Create SSH key pair. Name it ops-key.

Compute → Affinity groups → Add affinity group, three times: web-ag, app-ag, db-ag, each of type host anti-affinity.

2. VPC, ACLs and tiers

As RA-03 steps 2 to 4, with one change: the app and db tiers use offering net-vpctier-intlb-v1 instead of net-vpctier-std-v1.

The tier offering is fixed once the tier exists. Choosing net-vpctier-std-v1 for the app tier and discovering later that you need an internal LB means building a new tier and moving the VMs.

3. Deploy the six VMs

Compute → Instances → Add instance, six times. Template almalinux-9, SSH key ops-key, root disk 40 GB, and set the affinity group in the advanced section of the dialog:

4. Data volumes

Storage → Volumes → Create volume twice: db01-data and db02-data, 100 GB each, attached to their respective servers.

5. The public load balancer

Network → VPC → prod → Public IP addresses → Acquire new IP. On the new IP open the Load balancing tab and add a rule:

• Name web-http, public port 80, private port 80, algorithm roundrobin, tier prod-web
• Add web01 and web02 as members

Repeat for HTTPS on 443 if you terminate TLS on the web servers.

6. The internal load balancers

Network → VPC → prod → Internal LB → Add internal LB, twice:

db-lb has a single member by design - see The database tier. Promoting db02 after a failure means removing db01 from this load balancer and adding db02, after you've promoted it at the database level.

7. Point everything at the load balancers

Configure the web servers to call http://10.1.2.10:8080 rather than an app server directly, and the app servers to use 10.1.3.10 as their database host. If any configuration file names app01 or db01, the load balancers aren't doing anything for you.

The VPC, ACL and tier commands are as RA-03, with net-vpctier-intlb-v1 for the app and db tiers. What follows is what's new in this architecture.

Anti-affinity groups and the six VMs

for g in web-ag app-ag db-ag; do
  cmk create affinitygroup name=$g type="host anti-affinity"
done

TMPL=$(cmk list templates templatefilter=executable keyword="almalinux-9" filter=id | jq -r '.template[0].id')

deploy_vm () {  # name offering tierid ip affinitygroup
  OFF=$(cmk list serviceofferings name=$2 filter=id | jq -r '.serviceoffering[0].id')
  AG=$(cmk list affinitygroups name=$5 filter=id | jq -r '.affinitygroup[0].id')
  cmk deploy virtualmachine name=$1 zoneid=$ZONE templateid=$TMPL \
    serviceofferingid=$OFF networkids=$3 ipaddress=$4 affinitygroupids=$AG \
    keypair=ops-key rootdisksize=40
}

deploy_vm web01 s-small-gen2  $WEBTIER 10.1.1.11 web-ag
deploy_vm web02 s-small-gen2  $WEBTIER 10.1.1.12 web-ag
deploy_vm app01 s-medium-gen2 $APPTIER 10.1.2.11 app-ag
deploy_vm app02 s-medium-gen2 $APPTIER 10.1.2.12 app-ag
deploy_vm db01  m-small-gen2  $DBTIER  10.1.3.11 db-ag
deploy_vm db02  m-small-gen2  $DBTIER  10.1.3.12 db-ag

The public load balancer

WEB01=$(cmk list virtualmachines name=web01 filter=id | jq -r '.virtualmachine[0].id')
WEB02=$(cmk list virtualmachines name=web02 filter=id | jq -r '.virtualmachine[0].id')

LBIP=$(cmk associate ipaddress vpcid=$VPC | jq -r '.ipaddress.id')

LBRULE=$(cmk create loadbalancerrule name=web-http \
  publicipid=$LBIP networkid=$WEBTIER algorithm=roundrobin \
  publicport=80 privateport=80 | jq -r '.loadbalancer.id')

cmk assign toloadbalancerrule id=$LBRULE virtualmachineids=$WEB01,$WEB02

The internal load balancers

Internal load balancers use a different API from public ones - createLoadBalancer rather than createLoadBalancerRule. The two families are separate all the way through: listLoadBalancerRules filters to public-scheme rules only and will not show you an internal LB.

APP01=$(cmk list virtualmachines name=app01 filter=id | jq -r '.virtualmachine[0].id')
APP02=$(cmk list virtualmachines name=app02 filter=id | jq -r '.virtualmachine[0].id')
DB01=$(cmk list virtualmachines name=db01 filter=id | jq -r '.virtualmachine[0].id')

# App tier: balance 8080 across both application servers
APPLB=$(cmk create loadbalancer name=app-lb \
  description="Internal LB for the app tier" \
  networkid=$APPTIER sourceipaddressnetworkid=$APPTIER \
  sourceipaddress=10.1.2.10 scheme=internal \
  sourceport=8080 instanceport=8080 algorithm=roundrobin | jq -r '.loadbalancer.id')

cmk assign toloadbalancerrule id=$APPLB virtualmachineids=$APP01,$APP02

# Database tier: a stable address in front of the current primary only
DBLB=$(cmk create loadbalancer name=db-lb \
  description="Internal LB for the db tier" \
  networkid=$DBTIER sourceipaddressnetworkid=$DBTIER \
  sourceipaddress=10.1.3.10 scheme=internal \
  sourceport=3306 instanceport=3306 algorithm=roundrobin | jq -r '.loadbalancer.id')

cmk assign toloadbalancerrule id=$DBLB virtualmachineids=$DB01

Creating an internal load balancer deploys an appliance VM into the tier. Give it a moment before the VIP answers, and remember it occupies an address in the tier's subnet - which is why the VM addressing starts at .11 and leaves .10 free.

ansible-galaxy collection install ngine_io.cloudstack --upgrade
pip install 'cs>=3.4.0' sshpubkeys

export CLOUDSTACK_ENDPOINT=https://cloud.lightspeedhosting.com.au/client/api
export CLOUDSTACK_KEY=<api key>
export CLOUDSTACK_SECRET=<secret key>

---
# ra04-ha-multi-tier-lb.yml - deploy RA-04 HA multi-tier VPC
- name: RA-04 highly available multi-tier VPC
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    zone: au-mel01
    admin_cidr: 203.0.113.4/32
    template: almalinux-9
    vpc_cidr: 10.1.0.0/16
    web_cidr: 10.1.1.0/24
    app_cidr: 10.1.2.0/24
    db_cidr: 10.1.3.0/24

    tiers:
      - {name: prod-web, acl: prod-web-acl, gateway: 10.1.1.1, offering: net-vpctier-std-v1}
      - {name: prod-app, acl: prod-app-acl, gateway: 10.1.2.1, offering: net-vpctier-intlb-v1}
      - {name: prod-db,  acl: prod-db-acl,  gateway: 10.1.3.1, offering: net-vpctier-intlb-v1}

    instances:
      - {name: web01, offering: s-small-gen2,  tier: prod-web, ip: 10.1.1.11, ag: web-ag}
      - {name: web02, offering: s-small-gen2,  tier: prod-web, ip: 10.1.1.12, ag: web-ag}
      - {name: app01, offering: s-medium-gen2, tier: prod-app, ip: 10.1.2.11, ag: app-ag}
      - {name: app02, offering: s-medium-gen2, tier: prod-app, ip: 10.1.2.12, ag: app-ag}
      - {name: db01,  offering: m-small-gen2,  tier: prod-db,  ip: 10.1.3.11, ag: db-ag}
      - {name: db02,  offering: m-small-gen2,  tier: prod-db,  ip: 10.1.3.12, ag: db-ag}

    affinity_groups: [web-ag, app-ag, db-ag]

    acl_rules:
      # prod-web-acl
      - {acl: prod-web-acl, n: 100, t: ingress, p: tcp, s: 80,   e: 80,   cidr: 0.0.0.0/0}
      - {acl: prod-web-acl, n: 110, t: ingress, p: tcp, s: 443,  e: 443,  cidr: 0.0.0.0/0}
      - {acl: prod-web-acl, n: 120, t: ingress, p: tcp, s: 22,   e: 22,   cidr: "{{ admin_cidr }}"}
      - {acl: prod-web-acl, n: 200, t: egress,  p: tcp, s: 8080, e: 8080, cidr: "{{ app_cidr }}"}
      - {acl: prod-web-acl, n: 210, t: egress,  p: tcp, s: 22,   e: 22,   cidr: "{{ app_cidr }}"}
      - {acl: prod-web-acl, n: 220, t: egress,  p: tcp, s: 22,   e: 22,   cidr: "{{ db_cidr }}"}
      - {acl: prod-web-acl, n: 230, t: egress,  p: tcp, s: 80,   e: 80,   cidr: 0.0.0.0/0}
      - {acl: prod-web-acl, n: 240, t: egress,  p: tcp, s: 443,  e: 443,  cidr: 0.0.0.0/0}
      - {acl: prod-web-acl, n: 250, t: egress,  p: udp, s: 53,   e: 53,   cidr: 0.0.0.0/0}
      - {acl: prod-web-acl, n: 260, t: egress,  p: tcp, s: 53,   e: 53,   cidr: 0.0.0.0/0}
      - {acl: prod-web-acl, n: 270, t: egress,  p: udp, s: 123,  e: 123,  cidr: 0.0.0.0/0}

      # prod-app-acl
      - {acl: prod-app-acl, n: 100, t: ingress, p: tcp, s: 8080, e: 8080, cidr: "{{ web_cidr }}"}
      - {acl: prod-app-acl, n: 110, t: ingress, p: tcp, s: 22,   e: 22,   cidr: "{{ web_cidr }}"}
      - {acl: prod-app-acl, n: 200, t: egress,  p: tcp, s: 3306, e: 3306, cidr: "{{ db_cidr }}"}
      - {acl: prod-app-acl, n: 210, t: egress,  p: tcp, s: 80,   e: 80,   cidr: 0.0.0.0/0}
      - {acl: prod-app-acl, n: 220, t: egress,  p: tcp, s: 443,  e: 443,  cidr: 0.0.0.0/0}
      - {acl: prod-app-acl, n: 230, t: egress,  p: udp, s: 53,   e: 53,   cidr: 0.0.0.0/0}
      - {acl: prod-app-acl, n: 240, t: egress,  p: tcp, s: 53,   e: 53,   cidr: 0.0.0.0/0}
      - {acl: prod-app-acl, n: 250, t: egress,  p: udp, s: 123,  e: 123,  cidr: 0.0.0.0/0}

      # prod-db-acl
      - {acl: prod-db-acl, n: 100, t: ingress, p: tcp, s: 3306, e: 3306, cidr: "{{ app_cidr }}"}
      - {acl: prod-db-acl, n: 110, t: ingress, p: tcp, s: 22,   e: 22,   cidr: "{{ web_cidr }}"}
      - {acl: prod-db-acl, n: 200, t: egress,  p: tcp, s: 80,   e: 80,   cidr: 0.0.0.0/0}
      - {acl: prod-db-acl, n: 210, t: egress,  p: tcp, s: 443,  e: 443,  cidr: 0.0.0.0/0}
      - {acl: prod-db-acl, n: 220, t: egress,  p: udp, s: 53,   e: 53,   cidr: 0.0.0.0/0}
      - {acl: prod-db-acl, n: 230, t: egress,  p: tcp, s: 53,   e: 53,   cidr: 0.0.0.0/0}
      - {acl: prod-db-acl, n: 240, t: egress,  p: udp, s: 123,  e: 123,  cidr: 0.0.0.0/0}

  tasks:
    - name: Register SSH key
      ngine_io.cloudstack.ssh_key:
        name: ops-key
        public_key: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"

    - name: Create host anti-affinity groups
      ngine_io.cloudstack.affinity_group:
        name: "{{ item }}"
        affinity_type: host anti-affinity
      loop: "{{ affinity_groups }}"

    - name: Create VPC
      ngine_io.cloudstack.vpc:
        name: prod
        zone: "{{ zone }}"
        cidr: "{{ vpc_cidr }}"
        vpc_offering: vpc-std-v1

    - name: Create the three network ACLs
      ngine_io.cloudstack.network_acl:
        name: "{{ item.acl }}"
        description: "ACL for the {{ item.name }} tier"
        vpc: prod
        zone: "{{ zone }}"
      loop: "{{ tiers }}"

    - name: Create all ACL rules
      ngine_io.cloudstack.network_acl_rule:
        network_acl: "{{ item.acl }}"
        vpc: prod
        zone: "{{ zone }}"
        rule_position: "{{ item.n }}"
        action_policy: allow
        traffic_type: "{{ item.t }}"
        protocol: "{{ item.p }}"
        start_port: "{{ item.s }}"
        end_port: "{{ item.e }}"
        cidrs: ["{{ item.cidr }}"]
      loop: "{{ acl_rules }}"
      loop_control:
        label: "{{ item.acl }} {{ item.t }} {{ item.n }}"

    - name: Create the three tiers
      ngine_io.cloudstack.network:
        name: "{{ item.name }}"
        zone: "{{ zone }}"
        vpc: prod
        acl: "{{ item.acl }}"
        network_offering: "{{ item.offering }}"
        gateway: "{{ item.gateway }}"
        netmask: 255.255.255.0
      loop: "{{ tiers }}"

    - name: Deploy instances
      ngine_io.cloudstack.instance:
        name: "{{ item.name }}"
        zone: "{{ zone }}"
        template: "{{ template }}"
        service_offering: "{{ item.offering }}"
        networks: ["{{ item.tier }}"]
        ip_address: "{{ item.ip }}"
        affinity_groups: ["{{ item.ag }}"]
        ssh_keys: [ops-key]
        root_disk_size: 40
      loop: "{{ instances }}"

    - name: Create and attach the database volumes
      ngine_io.cloudstack.volume:
        name: "{{ item }}-data"
        zone: "{{ zone }}"
        disk_offering: Custom
        size: 100
        vm: "{{ item }}"
        state: attached
      loop: [db01, db02]

    - name: Acquire the public IP for the load balancer
      ngine_io.cloudstack.ip_address:
        zone: "{{ zone }}"
        vpc: prod
        tags:
          - key: purpose
            value: web-public-lb
      register: lbip

    - name: Public load balancer rule for HTTP
      ngine_io.cloudstack.lb_rule:
        name: web-http
        ip_address: "{{ lbip.ip_address }}"
        zone: "{{ zone }}"
        vpc: prod
        network: prod-web
        algorithm: roundrobin
        public_port: 80
        private_port: 80

    - name: Add the web servers to the load balancer
      ngine_io.cloudstack.lb_rule_member:
        name: web-http
        ip_address: "{{ lbip.ip_address }}"
        zone: "{{ zone }}"
        vms: [web01, web02]

    # Internal load balancers - a different API family from lb_rule above.
    # source_ip is optional; omit it and the platform allocates one from the
    # tier. It is pinned here because the ACLs and app config reference it.
    - name: Internal load balancer for the app tier
      ngine_io.cloudstack.lb_internal:
        name: app-lb
        description: Internal LB for the app tier
        network: prod-app
        vpc: prod
        zone: "{{ zone }}"
        source_ip: 10.1.2.10
        source_port: 8080
        instance_port: 8080
        algorithm: roundrobin

    - name: Both app servers behind the app tier load balancer
      ngine_io.cloudstack.lb_internal_member:
        name: app-lb
        network: prod-app
        vpc: prod
        zone: "{{ zone }}"
        vms: [app01, app02]

    - name: Internal load balancer for the db tier
      ngine_io.cloudstack.lb_internal:
        name: db-lb
        description: Internal LB for the db tier
        network: prod-db
        vpc: prod
        zone: "{{ zone }}"
        source_ip: 10.1.3.10
        source_port: 3306
        instance_port: 3306
        algorithm: roundrobin

    # One member on purpose - the current primary. See "The database tier".
    - name: Current database primary behind the db tier load balancer
      ngine_io.cloudstack.lb_internal_member:
        name: db-lb
        network: prod-db
        vpc: prod
        zone: "{{ zone }}"
        vms: [db01]

    - name: Show the load balancer IP
      ansible.builtin.debug:
        msg: "The application is reachable at {{ lbip.ip_address }}"

Affinity groups have to exist before the instances that reference them, which is why that task comes second. Applying a group to a running instance isn't possible - it has to be stopped first.

The VPC, ACL, tier and instance resources follow RA-03, with the app and db tiers changed to net-vpctier-intlb-v1 and the instance map extended to six entries. The additions specific to this architecture:

locals {
  zone = "au-mel01"

  instances = {
    web01 = { offering = "s-small-gen2",  tier = "web", ip = "10.1.1.11", ag = "web" }
    web02 = { offering = "s-small-gen2",  tier = "web", ip = "10.1.1.12", ag = "web" }
    app01 = { offering = "s-medium-gen2", tier = "app", ip = "10.1.2.11", ag = "app" }
    app02 = { offering = "s-medium-gen2", tier = "app", ip = "10.1.2.12", ag = "app" }
    db01  = { offering = "m-small-gen2",  tier = "db",  ip = "10.1.3.11", ag = "db" }
    db02  = { offering = "m-small-gen2",  tier = "db",  ip = "10.1.3.12", ag = "db" }
  }
}

resource "cloudstack_affinity_group" "ag" {
  for_each = toset(["web", "app", "db"])

  name = "${each.key}-ag"
  type = "host anti-affinity"
}

resource "cloudstack_instance" "vm" {
  for_each = local.instances

  name               = each.key
  zone               = local.zone
  template           = "almalinux-9"
  service_offering   = each.value.offering
  network_id         = local.tier_ids[each.value.tier]
  ip_address         = each.value.ip
  affinity_group_ids = [cloudstack_affinity_group.ag[each.value.ag].id]
  keypair            = cloudstack_ssh_keypair.ops.name
  root_disk_size     = 40
  expunge            = true
}

resource "cloudstack_disk" "db_data" {
  for_each = toset(["db01", "db02"])

  name               = "${each.key}-data"
  zone               = local.zone
  disk_offering      = "Custom"
  size               = 100
  attach             = true
  virtual_machine_id = cloudstack_instance.vm[each.key].id
}

resource "cloudstack_ipaddress" "lb" {
  zone   = local.zone
  vpc_id = cloudstack_vpc.prod.id
}

resource "cloudstack_loadbalancer_rule" "web_http" {
  name          = "web-http"
  ip_address_id = cloudstack_ipaddress.lb.id
  network_id    = cloudstack_network.web.id
  algorithm     = "roundrobin"
  public_port   = 80
  private_port  = 80

  member_ids = [
    cloudstack_instance.vm["web01"].id,
    cloudstack_instance.vm["web02"].id,
  ]
}

output "load_balancer_ip" {
  value = cloudstack_ipaddress.lb.ip_address
}

local.tier_ids is the same map from tier name to network id used in RA-03, defined alongside the three cloudstack_network resources:

locals {
  tier_ids = {
    web = cloudstack_network.web.id
    app = cloudstack_network.app.id
    db  = cloudstack_network.db.id
  }
}

network_id is not optional here in practice. The provider documents it as "required when public IP address is not associated with any network yet (VPC case)", which is exactly this case - the IP belongs to the VPC, not to a tier. Omit it and rule creation fails.

Internal load balancers are the one part of this architecture where the tooling isn't uniform. Worth explaining properly, because the reason determines whether you work around it or pick a different tool.

The platform supports it fully

Internal load balancers are created by the createLoadBalancer API, added in CloudStack 4.2. It's a user-level API - the command class lives in the ...api.command.user.loadbalancer package, not the admin package - so you can call it with your own API keys, on your own account, with no elevated privileges. listLoadBalancers, updateLoadBalancer, deleteLoadBalancer and assignToLoadBalancerRule are all available alongside it.

So this is not a restriction on the Lightspeed Cloud platform, and not something you need us to enable.

Where each tool stands

The two APIs are separate families, not one API with a mode switch. createLoadBalancerRule routes to a code path that hard-codes the public scheme, and listLoadBalancerRules filters its results to public-scheme rules - so a public-LB resource can never see or manage an internal one, however it's configured. That's why each tool needs a distinct implementation rather than an extra argument.

The Terraform gap

The Terraform provider's resource map has no internal LB entry. The underlying Go SDK the provider depends on does expose the API, so this is a missing resource rather than a missing capability - a contribution waiting to happen rather than a blocker in the client library.

Until it exists, the options are:

• Recommended: create the internal load balancers with the Ansible tasks above, or once with CloudMonkey or the portal, and manage everything else in Terraform. Internal LBs are long-lived - you create them when the tier is built and rarely touch them again - so the split costs less than it sounds.
• A null_resource with local-exec calling CloudMonkey will create one, but Terraform won't track it in state and won't remove it on destroy. Treat it as a bootstrap step rather than managed infrastructure.

The segmentation tests from RA-03 all still apply. These are the ones specific to high availability, and every one of them should be run at least once - an untested failover is a hypothesis.

1. curl -I http://<lb-ip> returns a response, and repeated requests are served by both web servers. Check the access logs on each.
2. Anti-affinity is real. In the portal, confirm web01 and web02 report different hosts. Same for the app and db pairs. If a pair shares a host, the group wasn't applied at deploy time.
3. Stop web01. The site stays up. Traffic moves to web02 within the load balancer's health check interval.
4. Stop app01. The site stays up, served through app02 via the internal load balancer.
5. From web01, nc -vz 10.1.2.10 8080 reaches the app tier's internal load balancer VIP.
6. From app01, nc -vz 10.1.3.10 3306 reaches the database VIP.
7. Count the rules in each ACL, as in RA-03 - a short count means a number was reused across directions.
8. Grep your configuration for individual hostnames. Nothing on the web servers should name app01, and nothing on the app servers should name db01. If they do, the failover you just tested will not work in production.
9. Rehearse the database promotion. Promote db02, swap the db-lb member, confirm the application recovers, then swap back. Write down how long it took and who has to be awake to do it.

• Rolling deployments. Remove one node from its load balancer, deploy, verify, add it back, then do the other. This is the day-to-day payoff of the architecture and is worth scripting.
• Scaling out. A third web server is a new VM in prod-web added to the load balancer. No ACL change, because the rules use tier CIDRs.
• Same-tier traffic is still unfiltered. web01 and web02 can reach each other on any port, as can each other pair. If that matters for your compliance posture, add host firewall rules within each tier.
• Watch capacity, not just health. Two nodes at 70% is a single node at 140%, which is an outage. Alert on per-node utilisation with the failure case in mind.
• Monitor replication lag on db02 independently of the load balancer health check. A TCP check on 3306 will not tell you the replica is hours behind.
• Keep backups off-platform. Volume snapshots protect against volume loss, not against a bad migration or a dropped table. Logical dumps to object storage, tested by restoring them somewhere.
• When you outgrow this. Automatic database failover means a quorum-based cluster - three nodes, not two. Surviving the loss of the zone means a second site and a rehearsed DR plan, which is a different piece of work entirely.