Built It Before the Button
ECS on Fargate built by hand in the console, broken on purpose to prove rollbacks, then compared against ECS Express Mode.
Archived · $0.24/month
- ECS Fargate
- ECR
- GitHub Actions
- OIDC
- IAM
- EventBridge
- SNS
- CloudWatch Logs
- AWS Budgets
- Python
I wanted to learn how containers run on AWS, so I built an ECS on Fargate service by hand in the console, one piece at a time: the roles, the log group, the cluster, the task definition, the security group and the service. Then I broke it on purpose to watch it recover, wrote a deploy workflow that can tell a real deploy from a rollback, and only then pressed the button: ECS Express Mode, which builds most of it for you. Comparing the two showed me what the button actually does. Everything has since been torn down, so there's no live demo, only the write-up.
2 rollbacks tested, site stayed up · 4 min 11 s deploys · $0.24 total
Context and goals
I had already built a CI/CD pipeline as Terraform in an earlier project. This time I deliberately went the other way: no infrastructure as code until I understood every field by hand. Writing Terraform for something I hadn't built yet would have meant copying defaults I didn't understand, and as it turned out, several of the console's defaults were wrong.
What I was aiming for:
- Run a small web app on ECS on Fargate, built entirely in the console
- Prove that a bad release rolls back on its own, with the site staying up
- Deploy from GitHub without storing any AWS keys
- Get an email when a deployment fails
- Compare my hand build against ECS Express Mode
- Keep a toil log of everything slow or error-prone, to automate next time
Out of scope: EKS (it has its own project), infrastructure as code, and the later stages I planned and then dropped (more on that under What's missing).
The app is a small FastAPI service I wrote for the lab. It shows which
task served the page, has a /healthz endpoint, logs JSON to stdout,
shuts down cleanly on SIGTERM, and can be told to crash on startup so I
could test failures.
Architecture
There are two architecture diagrams here, because I ran the same app two ways. The first is the service I built by hand: one public task, no load balancer, my own deploy workflow and alerts. The second is what ECS Express Mode built from the same image with one click: a load balancer, HTTPS and a rollback alarm in front of the same task. The gap between the two is what the button does for you, which is the point of the project.


Build. Every pull request runs the tests and a Docker build. A merge to main does the same, then pushes the image to a private ECR repository, tagged with the full commit SHA. The tags are immutable, so nothing can replace an image once it's pushed.

Deploy. A separate workflow, started by hand with a commit SHA. It reads the service's current task definition, swaps only the image, registers a new revision, updates the service and then waits until the rollout is finished on that new revision.
Run. One Fargate task (0.25 vCPU, 0.5 GB) in the default VPC with a public IP. The security group allows port 8000 from my IP only. Logs go to CloudWatch Logs with one week of retention.




Watch. Two EventBridge rules send ECS deployment failures and service warnings to an SNS topic that emails me.
Everything was built by hand. The only thing that runs as code is the two GitHub Actions workflows, and they only build, push and deploy app versions. They never create or change infrastructure.
Design decisions
| Decision | Chosen | Alternative | Why | Trade-off accepted |
|---|---|---|---|---|
| How to build | By hand in the console | Terraform from day one | Learn what every field does before automating it | Nothing is reproducible; rebuilding means clicking again |
| Compute | ECS on Fargate | ECS on EC2, EKS | AWS runs the servers, so the lab stays on deploys and failures | Higher price per vCPU, no control over the host |
| Traffic | Public task IP, my IP only | Application Load Balancer | Cost: an ALB bills hourly even when idle | No HTTPS, a new IP on every deploy |
| Network | Default VPC, public subnets | Private subnets with NAT or VPC endpoints | Private tasks need NAT or endpoints for ECR, S3 and Logs, all billed hourly | Tasks are reachable from the internet, limited to one IP |
| Deploy trigger | Manual, with a commit SHA | Deploy on every merge | Pick the version and watch every rollout while learning | Someone has to start each deploy |
| Deploy check | Poll the rollout and compare revisions | aws ecs wait services-stable |
The standard wait passes after a rollback, on the old version | More workflow code to maintain |
| Image tags | Full commit SHA, immutable | latest |
Deploy exactly the image I chose; a tag can't be overwritten | Re-running the push for the same commit fails, by design |
| Alerting | EventBridge → SNS email | CloudWatch alarms | The failure I cared about is a deployment event, not a metric | Raw JSON emails, no paging |
| Rollback | Rolling update with circuit breaker | Blue/green | Simplest safety net that keeps the old task serving | Only catches tasks that fail to start |
| Budget | $20 lab budget on the tag, plus an account-wide one | One budget | Untagged costs exist; the account budget catches them | Two budgets to look after |
Why I stopped where I did
The plan had four stages: CI, Fargate, ECS on EC2 and ECS Managed Instances. I stopped after the Express Mode comparison. Express Mode had already built what my next step was going to be by hand: a load balancer, a task security group open only to it, HTTPS and a rollback alarm. Building it again would have cost more for little new learning. ECS on EC2 is rarely the right fit for a small web app like this one, so I dropped those stages too.
Tech stack
| Layer | What I used |
|---|---|
| App | Python 3.13, FastAPI, uvicorn, 9 pytest tests |
| Image | Docker, linux/amd64, non-root user, ECR (private, immutable tags, scan on push) |
| Compute | ECS on Fargate, 0.25 vCPU / 0.5 GB, platform LATEST |
| CI/CD | GitHub Actions, OIDC, actions pinned to full commit SHAs, Dependabot |
| Operations | CloudWatch Logs, EventBridge, SNS, AWS Budgets, cost allocation tags |
| Region | ap-southeast-1 |
Deployment pipeline
CI runs on every pull request and every push to main. The test job installs the dependencies, runs pytest and builds the image. The push job runs only on main: it assumes an AWS role through OIDC, logs in to ECR and pushes the image. The workflow starts with no permissions and each job asks only for what it needs.
Branch protection requires a pull request and a passing test job before anything reaches main. Force pushes are blocked and nobody can bypass it. I tested each rule: a direct push to main is rejected, a pull request with a failing test can't merge, and a run from another branch can't assume the AWS role.


Deploy is its own workflow. It checks that it's running from main and that the input is a full 40-character SHA, then assumes a separate deploy role. Inputs reach the shell only through environment variables, so they can't inject commands. One deploy per service runs at a time.
The interesting part is how it decides whether a deploy worked. The
obvious tool is aws ecs wait services-stable, and my first version used
it. That run went red even though the deploy had worked: the wait
returned about 3 minutes before ECS marked the rollout as complete. It
also passes after a rollback, because the service is stable again, just
on the old version. So the workflow now polls every 15 seconds, for up to
15 minutes, and goes green only when the service is on the revision it
just registered and the rollout is COMPLETED.
Reliability
I didn't define an SLO for this lab. What I did define, and test, is what a deploy is allowed to do: either finish on the new version, or roll back while the old task keeps serving.
A normal update
The rolling update starts the new task before stopping the old one, and the old task shuts down cleanly on SIGTERM, about 150 ms after the signal.



Rollback, tested twice
| Test | What happened | Time to roll back | Site |
|---|---|---|---|
| A release that crashes on startup | 3 tasks failed, the circuit breaker tripped and rolled back to the previous revision | About 4 min | Stayed up |
| A deploy with an image tag that doesn't exist | 3 tasks failed to pull the image, the circuit breaker rolled back, the deploy job went red, the alert email arrived | About 10.5 min | Stayed up |
The image pull failure took much longer because each task retries the pull 7 times before giving up. In both cases the old task was never stopped, so nobody would have noticed anything.
The crash test:



The bad image tag, through the deploy workflow:




How long things took
| Deploy | Time |
|---|---|
| First deploy by hand | 2 min 47 s |
| Update by hand | 2 min 48 s |
| Deploy workflow, start to green | 4 min 11 s |
| Express Mode, blue/green | 8 min 23 s |
Alerts
| Rule | Fires on | Goes to | Tested? |
|---|---|---|---|
| Deployment failed | ECS SERVICE_DEPLOYMENT_FAILED for this service |
SNS → email | Yes: the bad tag deploy emailed me |
| Service action | ECS service events with WARN or ERROR | SNS → email | Built, never triggered |
There are no CloudWatch alarms. The next step I planned would have added a health check alarm on the load balancer.
Express Mode, compared
With the hand build working, I deployed the same image with Express Mode and listed everything it created that I hadn't built:
- An internet-facing load balancer with HTTPS on an AWS-owned domain, with its own certificate
- A task security group open only to the load balancer
- Blue/green deploys with a 3-minute bake and a rollback alarm
- CPU auto scaling from 1 to 20 tasks
- Its own log group
It also had defaults I wouldn't keep. The load balancer's security group opens port 80 and IPv6 with nothing listening behind them, the log group never expires, and the rollback alarm has no actions. Deleting the service removed everything that bills, but left the log group, 4 task definition revisions and 2 IAM roles behind. I found those by comparing an inventory from before and after, and cleaned them up with a script.
If I ran this in production I'd choose Express Mode's blue/green pattern over my rolling update. My circuit breaker only catches tasks that fail to start. Blue/green with a rollback alarm also catches a release that starts fine but serves errors. The cost is time: 8 min 23 s per deploy against 2 min 47 s.

Security
- OIDC, no access keys. GitHub Actions assumes AWS roles with short-lived tokens. The trust policy only accepts this repository's main branch
- Separate roles. CI can push to one ECR repository and nothing else. The deploy role can update one service, and can pass only the execution role, only to ECS tasks
- Tight execution role. It can pull from the one repository and write to the one log group, and it can't create log groups
- No secrets. Nothing is stored in the workflows or the task definition, and the task has no role because the app doesn't call AWS
- Pinned actions. Every third-party action is pinned to a full commit SHA, and Dependabot keeps them updated
- Network: port 8000 open to my IP only
- Honest limitation: both OIDC roles trust the same main branch, so the CI workflow could assume the deploy role. The fix is a GitHub environment on the deploy job, with the deploy role trusting only that environment
- Skipped: HTTPS and a load balancer, for cost. Traffic to the task was plain HTTP
- Skipped: WAF, because there was no load balancer to attach it to
- Skipped: private subnets, because they need a NAT gateway or VPC endpoints, both billed hourly
Cost
These are actual charges from Cost Explorer for the whole lab, November 3 to 13, 2026, in ap-southeast-1, with credits excluded and filtered to the lab's cost allocation tag.
| Service | What it was | Actual |
|---|---|---|
| ECS | Fargate task time | $0.14 |
| Elastic Load Balancing | Express Mode's load balancer, about 80 minutes | $0.05 |
| VPC | Public IPv4 addresses | $0.05 |
| ECR | Image storage | $0.01 |
| SNS, CloudWatch, ACM | Alerts, logs, certificate | $0.00 |
| Total | $0.24 |


It's this small because the service sat at 0 tasks between sessions. The whole account came to $0.26: the extra $0.02 was public IPv4 that doesn't carry the tag. That's why I kept an account-wide budget next to the $20 lab budget.
What broke
1. The deploy that went red but worked. The first workflow run
failed while the service was actually fine. Cause: services-stable
returned before ECS marked the rollout complete, so the check read "in
progress" and failed. Fix: poll the rollout state and the revision
instead. What I'd do differently: never trust a "stable" check to mean
"my version shipped".
2. Express Mode couldn't write its own logs. Its first deploy failed 5 tasks with an access denied error on CloudWatch Logs. Cause: Express Mode picks its own log group name, and my execution role could only write to mine. As a first deploy it had nothing to roll back to, so it just stopped.

3. An alert rule that would never have fired. The pattern the EventBridge console suggested matched on the cluster ARN, a field deployment events don't have. It would have failed silently forever. I caught it while building, and changed the rule to match the service instead. While testing I also found that the role the console created for SNS trusted only one rule, so the second rule failed silently too. What I'd do differently: trigger every new alert once, on purpose, before trusting it.
Evidence
| Claim | Proof I kept |
|---|---|
| CI pushes on merge, and only from main | Screenshots: the image in ECR, a rejected direct push, a blocked pull request, a denied role assumption from another branch |
| First deploy worked | Screenshots: the page showing the task ID and version, /healthz, JSON logs in CloudWatch |
| Rolling update replaces the old task cleanly | Screenshots: service events, and the old task logging SIGTERM then shutting down |
| Crash rollback | Screenshots: "Rollback successful", circuit breaker 3/3 |
| Bad tag rollback and alert | Screenshots: the failed run, the stopped reason, the alert email |
| Express Mode | Before and after inventories |
| Cost | Cost Explorer screenshot, $0.24 |
What's missing
- No infrastructure as code: everything was built by hand
- No SLO, and no CloudWatch alarms
- No load balancer, HTTPS or second task built by hand. Express Mode built them, I reviewed them
- The service warning alert was built but never triggered
- The two OIDC roles share the same trust (see Security)
- Planned and dropped: an S3 task role, ECS Exec, scheduled scaling, Fargate Spot, Container Insights and private subnets with VPC endpoints, plus the ECS on EC2 and Managed Instances stages
Status
Archived and torn down on November 13, 2026. The code is on GitHub ↗.