Swapping AWS NAT Gateways for self-healing EC2 instances without ever changing a whitelisted IP address our partners depended on

AWS NAT Gateway is one of those services you configure once during VPC setup and never look at again. It bills $0.045 per gigabyte in both directions. That's the trap AWS charges the same rate for the responses coming back to you as for the requests you send out.

When we pulled our baseline, 87% of our forward-proxy's NAT bytes were ingress API responses flowing back from suppliers. We were paying AWS to "process" traffic that, from the internet's side, was already free. A plain EC2 instance doing the identical job, IP masquerading, costs the price of the box. No per-byte toll.

Going back to how it used to be done decade ago

This isn't a new idea. it's an old one and revisited. Before AWS introduced NAT Gateway roughly a decade ago, this was the standard pattern a NAT configured EC2 instance in a public subnet, handling address translation itself. AWS's own documentation still lays out exactly what a team taking this on has to solve:

Three gaps highlighted in the chart no longer need hand-rolled solutions. An Auto Scaling Group replaces a dead instance without a bespoke failover script. SSM Parameter Store hands a fresh instance its configuration. IaC (Terraform) makes the whole thing repeatable and reviewable. We weren't rediscovering NAT instances so much as deliberately rebuilding the three things NAT Gateway was invented to solve
1. availability,
2. bandwidth,
3. maintenance
with primitives that weren't commodity tools when that trade was first made. And we kept only the part we no longer needed to pay for the flat per-byte fee.

Two NAT Gateways became two Linux boxes

We migrated two production NAT layers, months apart, with the same pattern each time one c5n.xlarge EC2 instance per Availability Zone, running ordinary iptables MASQUERADE rules in place of the managed NAT Gateway.

First pass: the forward-proxy NAT pair, cut over roughly 4.4 TB/day.
Second pass: three weeks later the larger production pair, roughly 9.5 TB/day. Same architecture, same rollout playbook, twice.

You can't just take an EIP off a NAT Gateway

The forward-proxy's outbound addresses were whitelisted at several partner APIs, so silently swapping them for new IPs wasn't an option. We assumed reallocating the Elastic IP onto the new instance would be a sub-second operation. AWS disagreed:

InvalidParameter: Elastic IP address ... is primary EIP of NAT, disassociating primary EIP is not supported.

The only way to free a NAT Gateway's EIP is to delete the NAT Gateway itself after which the address takes roughly 163 seconds to become available again.

Here's what that actually looked like. Before touching anything in production, both NAT instances were already built and running each sitting on a disposable & non-whitelisted EIP, with traffic forwarding proven end-to-end through a parallel test subnet that had zero connection to real traffic. Only once that was validated did the real cutover start, run once per Availability Zone:

1. Reroute that AZ's traffic to a neighboring AZ's NAT by changing one route table entry a temporary cross-AZ hop. For AZ-2 (cut over first), that meant pointing its private route table at AZ-4's still-live NAT Gateway. AZ-2's workloads keep working the entire time, just exiting via AZ-4's IP for a few minutes.
2. Delete that AZ's own NAT Gateway. Its whitelisted EIP enters the ~163-second "held" state the same wait we'd already measured and budgeted for in the earlier throwaway-EIP test, so no surprise.
3. The moment the hold clears, attach the freed EIP to that AZ's NAT instance the same instance that had been sitting on a throwaway address, already proven to forward traffic correctly.
4. Update the SSM parameter holding that AZ's EIP allocation ID, so if the ASG ever replaces this instance later, the replacement reattaches the whitelisted address automatically, not the throwaway one.
5. Flip the route table one more time 0.0.0.0/0 now points at that AZ's own NAT instance, carrying its original whitelisted IP. Done.

Then the whole five-step sequence runs again for the second AZ except this time the "neighboring AZ" doing the buffering is AZ-2's NAT instance, which had just finished cutting over. Each pass took about three minutes, start to finish, with a one-to-three-minute window of retried connections while in-flight TCP sessions rode out the cross-AZ hop which settled on its own well before the post-cutover soak period ended. At no point was a partner-facing address ever unreachable; traffic was always flowing somewhere, just briefly through a neighboring AZ's gateway instead of its own. From outside, no partner ever saw an address change hands because, from their side, it never did.

What's actually running is self-healing, not autoscaling

Every NAT instance sits in its own Auto Scaling Group configured as min = max = desired = 1. That is not elastic scale-out nothing responds to load. The ASG's entire job is blunter, if the one instance disappears, launch exactly one replacement, immediately.

We sized it that way deliberately. NAT translation for our traffic isn't CPU-bound a single c5n.xlarge sits at single-digit percent utilization. The problem this design solves isn't throughput. It's availability what happens the instant one box, holding a whitelisted IP that partners depend on, dies under production traffic.

How a replacement instance re-adopts the same IP

This is what turns a dead instance from a 3am page into a non-event. On boot, every NAT instance's user-data script runs five steps, in order:

1. Detect its own primary network interface, so one AMI works regardless of which ENI AWS assigns.
2. Disable source/dest check required for any instance to forward traffic not addressed to itself.
3. Read the Elastic IP allocation ID for its AZ from SSM Parameter Store.
4. Attach that EIP to itself with --allow-reassociation the flag that lets a new instance take the address from whatever is holding it, dead or not.
5. Rewrite the route table's default route (0.0.0.0/0) to its own interface gated behind an SSM arm/disarm flag.

No human, no runbook, no on-call page. The new instance makes itself the NAT for its zone the moment it finishes booting.

62 seconds, timed, not estimated

We didn't want to ship this on faith. So we ran the actual failure: manually terminated the live production NAT instance in one AZ and timed the recovery, end to end.

We plan around a 60–90 second blip for whichever AZ loses an instance mid-traffic Squid's retry logic absorbs that without anything user-visible. That's a deliberate, well-understood trade against a saving this large. Worth saying plainly the 62-second figure is one validated drill on the first migration, not an average across incidents.

The bill, before and after

Verified months later against real AWS Cost Explorer billing not the projection, the invoice the NatGateway-Bytes line item:

What we'd tell anyone doing this

• Check your ingress-vs-egress split first. The saving is largest when most of your NAT traffic is inbound responses exactly what a NAT Gateway bills for and a NAT instance doesn't.
• If any NAT addresses are whitelisted anywhere, plan around the "can't move a primary EIP without deleting the gateway" behavior it changes the order everything else happens in.
• Don't trust the design doc's recovery-time estimate. Terminate a live instance on purpose and time the real thing.
• Decommission the old NAT Gateway promptly. We didn't, on the second migration, and it sat there billing hourly and EIP fees for weeks doing nothing.