Skip to content

· Gary O'Keeffe · Life Sciences · 5 min read

How a Lab Cut Its AWS Bill From $70,000 to $36,000 a Year

A mid-size life-science lab cut its AWS bill from about $70,000 a year to about $36,000—a saving of about $34,000 a year, roughly half—after an audit and the changes it found.

The work was not exotic. Most of the spend was sitting in plain sight. Dev and test databases ran all night. Deleted files still lived in S3 Standard. The production database had no reservation. Nobody had looked at the bill line by line.

This post is what we found, what we changed, and the pitfalls we keep seeing. The lab is not named. Nothing here identifies them.

The bill

The AWS bill was about $70,000 a year. Four services made up almost all of it:

  • Managed databases, the largest line. License-included Oracle, billed by the hour.
  • S3 storage, most of it raw sequencing data.
  • EBS volumes and snapshots, accumulated over years.
  • Windows EC2 servers running the application layer.

Cost Explorer shows these totals. It does not tell you which databases are only used during the workday, or that most of the S3 bill is old versions of files someone deleted years ago. That takes a line-by-line read.

What we changed

Databases and servers that only worked 8–10 hours a day

Dev, test, and spare database copies ran 24/7 even though people used them 8–10 hours a day. We scheduled them to start before the workday and stop after it.

The databases start before the app servers and stop after them. The app never comes up without its database.

A reference database was only needed during the first and last weeks of the month. It now runs in those windows only.

Non-production Windows app servers had the same pattern. They ran 24/7 for a 10-hour workday. We scheduled them with their databases.

The schedules use EventBridge Scheduler. There is no custom code.

A production database with no commitment

The production database ran all the time, as it should. There was no reserved instance on it. A one-year reservation cuts about a third off that line.

Raw sequencing data moved to archive storage

Most of the S3 bill was raw sequencing data and old object versions that did not need to be in the expensive storage class. Versioning was on. Files deleted long ago still had old versions in Standard storage.

We used S3 Inventory to identify what could move. Then a lifecycle rule moved the sequencing data and noncurrent versions to Glacier Deep Archive. That class is about 96% cheaper than Standard.

Nothing was deleted. The data is still there, at a fraction of the cost.

Snapshots and volume types

There were 612 EBS snapshots. Some dated to 2020. Nobody had expired them. Several gp2 volumes could move to gp3. The old snapshot cleanup and the gp2 to gp3 volume moves are done.

The result

We identified the savings and worked with the lab to put the changes in. The bill is about $36,000 a year: a saving of about $34,000 a year, roughly half.

If your bill looks like this one, the Cloud Cost Audit is $5,000 for one account. A refund of the $5,000 if the final sheet comes to under 10% of your monthly cloud bill, refunded the same day, and we still send the sheet.

Common pitfalls

These show up often. They showed up here.

Dev and test run around the clock. A database used 9 hours a day, 5 days a week, does not need to run 168 hours a week. Same for the app servers in front of it. Schedule them. Start the database first. Stop it last.

No reservation or Savings Plan on steady production. If a production database has run for months and will run for months, on-demand pricing is a choice. A one-year reservation is the simple version of that choice.

Versioned buckets keep deleted files in Standard forever. Versioning is useful. It is also a quiet storage bill. Noncurrent versions of deleted objects sit at the Standard rate until a lifecycle rule moves them. Many teams never add that rule.

Old snapshots and AMIs never expire. Snapshots accumulate. Copies in a second region accumulate too. If no one set an expiry, they are still there.

gp2 volumes that could be gp3. gp3 is cheaper for most workloads and you can set IOPS separately. The change is small. The leftover gp2 volumes add up.

License-included database pricing hides in one line. A managed Oracle bill is compute, license, and storage in one hourly number. It looks like one database. It is often several copies, some of them idle.

A stopped RDS database restarts after 7 days. AWS does this on purpose. If you stop a non-production database and walk away, it comes back. A schedule is more reliable than a one-time stop.

Glacier has minimum storage durations and slow restores. Deep Archive is cheap because you are not supposed to read it often. Objects have a minimum storage duration. Restores take hours, not seconds. That is fine for old versions you are keeping in case someone asks. It is not a backup you would restore from at 2 a.m.

Lifecycle rules take a day or two to apply. After you write the rule, do not expect the bill to move the same afternoon. Inventory and transitions run on their own cadence.

The same audit on Google Cloud

The same waste shows up on Google Cloud under different names.

  • Cloud SQL and Compute Engine have instance schedules, the counterpart to EventBridge Scheduler.
  • Committed use discounts are the counterpart to reserved instances and Savings Plans.
  • Cloud Storage has Nearline, Coldline, and Archive classes, and lifecycle rules for noncurrent object versions.
  • Persistent disk snapshots pile up the same way EBS snapshots do.

We run this audit on AWS and on Google Cloud.

What a Cloud Cost Audit costs

The audit is $5,000 for one AWS or Google Cloud account. You get a one-page sheet listing every change with its resource ID, savings, risk level, and approver, plus a one-hour meeting to walk through it. Timeline: one week from working access to the sheet.

A refund of the $5,000 if the final sheet comes to under 10% of your monthly cloud bill, refunded the same day, and we still send the sheet.

→ See the Cloud Cost Audit

Share:

Download this article

How a Lab Cut Its AWS Bill From $70,000 to $36,000 a Year

The email is required for the PDF. See the privacy policy.

Related Posts

View All Posts »