· Gary O'Keeffe · Life Sciences · 5 min read
How We Cut a Cloud Bill by 45%
A midsize life sciences lab running a LIMS on AWS. We found the waste and worked with them to cut the bill by 45%.
We identified savings at a midsize life sciences lab running a LIMS on AWS, then worked with them to cut the bill by 45%.
The work was not exotic. Most of the spend was sitting in plain sight. Dev and test databases ran all night. Deleted files still lived in S3 Standard. The production database had no reservation. Nobody had looked at the bill line by line.
This post is what we found, what we changed, and the pitfalls we keep seeing. The lab is not named. Nothing here identifies them.
The bill
Monthly spend in scope was about $5,000. Four lines made up almost all of it:
- Managed Oracle databases, about $1,850 a month. This was the largest line. License-included Oracle, billed by the hour.
- S3 storage, about $1,160 a month.
- EC2 extras (EBS volumes and snapshots), about $1,450 a month.
- Windows EC2 servers, about $770 a month.
Cost Explorer shows these totals. It does not tell you which databases are only used during the workday, or that most of the S3 bill is old versions of files someone deleted years ago. That takes a line-by-line read.
What we changed
Databases and servers that ran all day, every day
Dev, test, and spare database copies ran 24/7. People used them during the workday. We scheduled them for 8 to 5 on weekdays.
The databases start before the app servers and stop after them. The app never comes up without its database.
A reference database was only needed during the first and last weeks of the month. It now runs in those windows only.
Non-production Windows app servers had the same pattern. They ran 24/7. We scheduled them with their databases.
The schedules use EventBridge Scheduler. There is no custom code.
A production database with no commitment
The production database ran all the time, as it should. There was no reserved instance on it. A one-year reservation cuts about a third off that line.
Old versions sitting in S3 Standard
Most of the S3 bill was not current data. Versioning was on. Files deleted long ago still had old versions in Standard storage. About 40 TB. About 300,000 objects.
We used S3 Inventory to list every object and every version. Then a lifecycle rule moved noncurrent versions to Glacier Deep Archive. That class is about 96% cheaper than Standard, and about 72% cheaper than Glacier Flexible Retrieval.
The one-time transition fees were about $15. The data stayed. We did not delete it.
Snapshots and volume types, left with the team
There were 612 EBS snapshots. Some dated to 2020. Nobody had expired them. Several gp2 volumes could move to gp3. The lab’s team is taking that work on.
The result
We identified the savings and worked with the lab to put the changes in. The in-scope bill dropped by 45%.
Old snapshots and the gp2 to gp3 moves were still open. Those are not in the 45%.
Common pitfalls
These show up often. They showed up here.
Dev and test run around the clock. A database used 9 hours a day, 5 days a week, does not need to run 168 hours a week. Same for the app servers in front of it. Schedule them. Start the database first. Stop it last.
No reservation or Savings Plan on steady production. If a production database has run for months and will run for months, on-demand pricing is a choice. A one-year reservation is the simple version of that choice.
Versioned buckets keep deleted files in Standard forever. Versioning is useful. It is also a quiet storage bill. Noncurrent versions of deleted objects sit at the Standard rate until a lifecycle rule moves them. Many teams never add that rule.
Old snapshots and AMIs never expire. Snapshots accumulate. Copies in a second region accumulate too. If no one set an expiry, they are still there.
gp2 volumes that could be gp3. gp3 is cheaper for most workloads and you can set IOPS separately. The change is small. The leftover gp2 volumes add up.
License-included database pricing hides in one line. A managed Oracle bill is compute, license, and storage in one hourly number. It looks like one database. It is often several copies, some of them idle.
A stopped RDS database restarts after 7 days. AWS does this on purpose. If you stop a non-production database and walk away, it comes back. A schedule is more reliable than a one-time stop.
Glacier has minimum storage durations and slow restores. Deep Archive is cheap because you are not supposed to read it often. Objects have a minimum storage duration. Restores take hours, not seconds. That is fine for old versions you are keeping in case someone asks. It is not a backup you would restore from at 2 a.m.
Lifecycle rules take a day or two to apply. After you write the rule, do not expect the bill to move the same afternoon. Inventory and transitions run on their own cadence.
The same audit on Google Cloud
The same waste shows up on Google Cloud under different names.
- Cloud SQL and Compute Engine have instance schedules, the counterpart to EventBridge Scheduler.
- Committed use discounts are the counterpart to reserved instances and Savings Plans.
- Cloud Storage has Nearline, Coldline, and Archive classes, and lifecycle rules for noncurrent object versions.
- Persistent disk snapshots pile up the same way EBS snapshots do.
We run this audit on AWS and on Google Cloud.
What a Cloud Cost Audit costs
A full audit is $5,000. One flat fee. You get a line-by-line read of the bill, a prioritized savings plan with estimated monthly savings per fix, and help putting the changes in.
