← Back to profile

Built It Before the Button

ECS on Fargate built by hand in the console, broken on purpose to prove rollbacks, then compared against ECS Express Mode.

Archived · $0.24/month


I wanted to learn how containers run on AWS, so I built an ECS on Fargate service by hand in the console, one piece at a time: the roles, the log group, the cluster, the task definition, the security group and the service. Then I broke it on purpose to watch it recover, wrote a deploy workflow that can tell a real deploy from a rollback, and only then pressed the button: ECS Express Mode, which builds most of it for you. Comparing the two showed me what the button actually does. Everything has since been torn down, so there's no live demo, only the write-up.

2 rollbacks tested, site stayed up · 4 min 11 s deploys · $0.24 total

Context and goals

I had already built a CI/CD pipeline as Terraform in an earlier project. This time I deliberately went the other way: no infrastructure as code until I understood every field by hand. Writing Terraform for something I hadn't built yet would have meant copying defaults I didn't understand, and as it turned out, several of the console's defaults were wrong.

What I was aiming for:

Out of scope: EKS (it has its own project), infrastructure as code, and the later stages I planned and then dropped (more on that under What's missing).

The app is a small FastAPI service I wrote for the lab. It shows which task served the page, has a /healthz endpoint, logs JSON to stdout, shuts down cleanly on SIGTERM, and can be told to crash on startup so I could test failures.

Architecture

There are two architecture diagrams here, because I ran the same app two ways. The first is the service I built by hand: one public task, no load balancer, my own deploy workflow and alerts. The second is what ECS Express Mode built from the same image with one click: a load balancer, HTTPS and a rollback alarm in front of the same task. The gap between the two is what the button does for you, which is the point of the project.

Architecture of the hand-built service: GitHub Actions pushes images to ECR and assumes AWS roles through IAM OIDC; one Fargate task in a public subnet pulls the image, takes HTTP on port 8000 through the internet gateway, and sends logs to CloudWatch; deployment failed events go through EventBridge and SNS to an email.
The hand-built service: build, deploy, run and alert

Architecture of the Express Mode deployment: users reach an Application Load Balancer over HTTPS, the load balancer forwards to the Fargate task on port 8000 through a task security group, the task pulls from ECR, and logs and load balancer metrics go to CloudWatch alarms.
What Express Mode built: a load balancer in front of the same task

Build. Every pull request runs the tests and a Docker build. A merge to main does the same, then pushes the image to a private ECR repository, tagged with the full commit SHA. The tags are immutable, so nothing can replace an image once it's pushed.

The private ECR repository with one image, tagged with the full 40-character commit SHA 4a1e0b7…, pushed by CI.
The first image CI pushed to ECR

Deploy. A separate workflow, started by hand with a commit SHA. It reads the service's current task definition, swaps only the image, registers a new revision, updates the service and then waits until the rollout is finished on that new revision.

Run. One Fargate task (0.25 vCPU, 0.5 GB) in the default VPC with a public IP. The security group allows port 8000 from my IP only. Logs go to CloudWatch Logs with one week of retention.

The ecs-lab-svc service in the ECS console: active, 1 of 1 tasks running on task definition ecs-lab:1, deployment status Success.
The first deploy, built by hand

The lab app's page showing it was served by task 9cd3b156 in ap-southeast-1b on FARGATE, cluster ecs-lab, version 4a1e0b7, with notes kept in memory.
The app reports which task served it and which version it runs

The /healthz endpoint returning {"status":"ok"}.
The health endpoint

CloudWatch log events: uvicorn startup lines, a JSON startup event with version 4a1e0b7 and the task ID, and JSON request logs for /, /notes and /healthz.
JSON logs in CloudWatch, client IPs blacked out

Watch. Two EventBridge rules send ECS deployment failures and service warnings to an SNS topic that emails me.

Everything was built by hand. The only thing that runs as code is the two GitHub Actions workflows, and they only build, push and deploy app versions. They never create or change infrastructure.

Design decisions

Decision Chosen Alternative Why Trade-off accepted
How to build By hand in the console Terraform from day one Learn what every field does before automating it Nothing is reproducible; rebuilding means clicking again
Compute ECS on Fargate ECS on EC2, EKS AWS runs the servers, so the lab stays on deploys and failures Higher price per vCPU, no control over the host
Traffic Public task IP, my IP only Application Load Balancer Cost: an ALB bills hourly even when idle No HTTPS, a new IP on every deploy
Network Default VPC, public subnets Private subnets with NAT or VPC endpoints Private tasks need NAT or endpoints for ECR, S3 and Logs, all billed hourly Tasks are reachable from the internet, limited to one IP
Deploy trigger Manual, with a commit SHA Deploy on every merge Pick the version and watch every rollout while learning Someone has to start each deploy
Deploy check Poll the rollout and compare revisions aws ecs wait services-stable The standard wait passes after a rollback, on the old version More workflow code to maintain
Image tags Full commit SHA, immutable latest Deploy exactly the image I chose; a tag can't be overwritten Re-running the push for the same commit fails, by design
Alerting EventBridge → SNS email CloudWatch alarms The failure I cared about is a deployment event, not a metric Raw JSON emails, no paging
Rollback Rolling update with circuit breaker Blue/green Simplest safety net that keeps the old task serving Only catches tasks that fail to start
Budget $20 lab budget on the tag, plus an account-wide one One budget Untagged costs exist; the account budget catches them Two budgets to look after

Why I stopped where I did

The plan had four stages: CI, Fargate, ECS on EC2 and ECS Managed Instances. I stopped after the Express Mode comparison. Express Mode had already built what my next step was going to be by hand: a load balancer, a task security group open only to it, HTTPS and a rollback alarm. Building it again would have cost more for little new learning. ECS on EC2 is rarely the right fit for a small web app like this one, so I dropped those stages too.

Tech stack

Layer What I used
App Python 3.13, FastAPI, uvicorn, 9 pytest tests
Image Docker, linux/amd64, non-root user, ECR (private, immutable tags, scan on push)
Compute ECS on Fargate, 0.25 vCPU / 0.5 GB, platform LATEST
CI/CD GitHub Actions, OIDC, actions pinned to full commit SHAs, Dependabot
Operations CloudWatch Logs, EventBridge, SNS, AWS Budgets, cost allocation tags
Region ap-southeast-1

Deployment pipeline

CI runs on every pull request and every push to main. The test job installs the dependencies, runs pytest and builds the image. The push job runs only on main: it assumes an AWS role through OIDC, logs in to ECR and pushes the image. The workflow starts with no permissions and each job asks only for what it needs.

Branch protection requires a pull request and a passing test job before anything reaches main. Force pushes are blocked and nobody can bypass it. I tested each rule: a direct push to main is rejected, a pull request with a failing test can't merge, and a run from another branch can't assume the AWS role.

A terminal showing git push to main rejected: push declined due to repository rule violations.
A direct push to main, rejected

A GitHub Actions push job failing at Assume AWS role after 12 retries: Not authorized to perform sts:AssumeRoleWithWebIdentity.
A run from another branch can't assume the role

Deploy is its own workflow. It checks that it's running from main and that the input is a full 40-character SHA, then assumes a separate deploy role. Inputs reach the shell only through environment variables, so they can't inject commands. One deploy per service runs at a time.

The interesting part is how it decides whether a deploy worked. The obvious tool is aws ecs wait services-stable, and my first version used it. That run went red even though the deploy had worked: the wait returned about 3 minutes before ECS marked the rollout as complete. It also passes after a rollback, because the service is stable again, just on the old version. So the workflow now polls every 15 seconds, for up to 15 minutes, and goes green only when the service is on the revision it just registered and the rollout is COMPLETED.

Reliability

I didn't define an SLO for this lab. What I did define, and test, is what a deploy is allowed to do: either finish on the new version, or roll back while the old task keeps serving.

A normal update

The rolling update starts the new task before stopping the old one, and the old task shuts down cleanly on SIGTERM, about 150 ms after the signal.

The deployment timeline mid-update: the new task is running while the old task is still waiting to drain.
New task first, old task still serving

The same deployment completed in 2 minutes 48 seconds: one task started, then the old task stopped.
Update complete in 2 min 48 s

CloudWatch logs from the old task: a sigterm event, uvicorn shutting down, the app's shutdown event and application shutdown complete.
The old task shutting down cleanly, client IPs blacked out

Rollback, tested twice

Test What happened Time to roll back Site
A release that crashes on startup 3 tasks failed, the circuit breaker tripped and rolled back to the previous revision About 4 min Stayed up
A deploy with an image tag that doesn't exist 3 tasks failed to pull the image, the circuit breaker rolled back, the deploy job went red, the alert email arrived About 10.5 min Stayed up

The image pull failure took much longer because each task retries the pull 7 times before giving up. In both cases the old task was never stopped, so nobody would have noticed anything.

The crash test:

ECS service events: four tasks started between 21:34 and 21:38, then deployment failed: tasks failed to start, rolling back to the previous deployment, and one pending task stopped.
Service events during the crash test

The deployment timeline with the circuit breaker triggered at 3 of 3 failed tasks, then rolling back and rolled back.
Circuit breaker triggered, 3/3 failed tasks

The service deployments list: the crash deployment marked Rollback successful, above two successful deployments.
Rollback successful

The bad image tag, through the deploy workflow:

The deploy.yml run on main, manually triggered, status Failure after 11 minutes 28 seconds.
The deploy job fails instead of going green

The tasks list: the ecs-lab:5 task still running, and the ecs-lab:6 tasks stopped.
The old revision kept serving

A stopped task's reason: CannotPullContainerError, the image tag of forty zeros not found.
Why the tasks stopped

An email from ecs-lab-alerts with the subject AWS Notification Message.
The alert email from EventBridge and SNS

How long things took

Deploy Time
First deploy by hand 2 min 47 s
Update by hand 2 min 48 s
Deploy workflow, start to green 4 min 11 s
Express Mode, blue/green 8 min 23 s

Alerts

Rule Fires on Goes to Tested?
Deployment failed ECS SERVICE_DEPLOYMENT_FAILED for this service SNS → email Yes: the bad tag deploy emailed me
Service action ECS service events with WARN or ERROR SNS → email Built, never triggered

There are no CloudWatch alarms. The next step I planned would have added a health check alarm on the load balancer.

Express Mode, compared

With the hand build working, I deployed the same image with Express Mode and listed everything it created that I hadn't built:

It also had defaults I wouldn't keep. The load balancer's security group opens port 80 and IPv6 with nothing listening behind them, the log group never expires, and the rollback alarm has no actions. Deleting the service removed everything that bills, but left the log group, 4 task definition revisions and 2 IAM roles behind. I found those by comparing an inventory from before and after, and cleaned them up with a script.

If I ran this in production I'd choose Express Mode's blue/green pattern over my rolling update. My circuit breaker only catches tasks that fail to start. Blue/green with a rollback alarm also catches a release that starts fine but serves errors. The cost is time: 8 min 23 s per deploy against 2 min 47 s.

The lab app served through Express Mode: task 7217de24 in ap-southeast-1a on FARGATE, version 0b32eb9.
The same image, deployed by Express Mode

Security

Cost

These are actual charges from Cost Explorer for the whole lab, November 3 to 13, 2026, in ap-southeast-1, with credits excluded and filtered to the lab's cost allocation tag.

Service What it was Actual
ECS Fargate task time $0.14
Elastic Load Balancing Express Mode's load balancer, about 80 minutes $0.05
VPC Public IPv4 addresses $0.05
ECR Image storage $0.01
SNS, CloudWatch, ACM Alerts, logs, certificate $0.00
Total $0.24

Cost Explorer breakdown for November 2026: total US$0.24, Elastic Container Service $0.14, Elastic Load Balancing $0.05, VPC $0.05, ECR $0.01, and Certificate Manager, SNS and CloudWatch at $0.00.
The whole lab in Cost Explorer

The project-ecs-alarm budget: $0.24 spent of a $20 monthly cost budget, healthy.
The lab budget

It's this small because the service sat at 0 tasks between sessions. The whole account came to $0.26: the extra $0.02 was public IPv4 that doesn't carry the tag. That's why I kept an account-wide budget next to the $20 lab budget.

What broke

1. The deploy that went red but worked. The first workflow run failed while the service was actually fine. Cause: services-stable returned before ECS marked the rollout complete, so the check read "in progress" and failed. Fix: poll the rollout state and the revision instead. What I'd do differently: never trust a "stable" check to mean "my version shipped".

2. Express Mode couldn't write its own logs. Its first deploy failed 5 tasks with an access denied error on CloudWatch Logs. Cause: Express Mode picks its own log group name, and my execution role could only write to mine. As a first deploy it had nothing to roll back to, so it just stopped.

An Express Mode deployment whose rollback failed: No rollback candidate was found to run the rollback, after 7 minutes 8 seconds.
Nothing to roll back to
Fix: add its log group to the role. What I'd do differently: check which names a managed feature will choose before pointing a tightly scoped role at it. And it showed up only in the console: my alert rules covered my service, not the Express one.

3. An alert rule that would never have fired. The pattern the EventBridge console suggested matched on the cluster ARN, a field deployment events don't have. It would have failed silently forever. I caught it while building, and changed the rule to match the service instead. While testing I also found that the role the console created for SNS trusted only one rule, so the second rule failed silently too. What I'd do differently: trigger every new alert once, on purpose, before trusting it.

Evidence

Claim Proof I kept
CI pushes on merge, and only from main Screenshots: the image in ECR, a rejected direct push, a blocked pull request, a denied role assumption from another branch
First deploy worked Screenshots: the page showing the task ID and version, /healthz, JSON logs in CloudWatch
Rolling update replaces the old task cleanly Screenshots: service events, and the old task logging SIGTERM then shutting down
Crash rollback Screenshots: "Rollback successful", circuit breaker 3/3
Bad tag rollback and alert Screenshots: the failed run, the stopped reason, the alert email
Express Mode Before and after inventories
Cost Cost Explorer screenshot, $0.24

What's missing

Status

Archived and torn down on November 13, 2026. The code is on GitHub ↗.