WHYY.org
The Challenge
Years of publishing had filled WHYY's site with about 330 GB of media, roughly 2.4 million files. All of it lived on Pantheon's file system, right alongside the website. That created three problems.
No working backup. The library was larger than Pantheon can back up. The media hadn't had a working backup in years, so there was no copy to restore from.
Slow tools for editors. With more than 250,000 items in the library, a media library search took over a minute, long enough to time out.
Slow development. Each new copy of the site for development meant hauling all that media along.
The people closest to the site felt it most. WHYY's digital team carried the risk of having no restorable copy of the media. Their in-house developers copied the site between environments constantly. Editors felt it every time they searched for an image.
The Plan
The media had to come off Pantheon for backups to work again. The goal was to move the files without changing how readers reach the site or how editors publish to it. We chose to utilize Amazon S3 and AWS CloudFront to store and deliver the media and take the burden off of Pantheon.
Where the media lives now
Before, Pantheon held the website and its 330 GB media library together, which made the site too big to back up. Now Pantheon builds the pages, Amazon S3 stores the media, and CloudFront delivers it to readers.
How we did it: rehearse everything, then launch
Nothing touched the live site until it had been proven on a full copy. Pantheon calls these copies multidevs, and we used them at every step.
1. Build and test.
In July and August, we built and tested the whole setup on one multidev.
2. Do the heavy lifting off the live site.
We registered all 255,000 media items on that multidev, then carried the results over. The live site never had to do that work itself.
3. Rehearse the launch.
On September 20–21, we ran the full launch on a same-day copy of the live site. The real launch on September 22 took about 30 minutes.
4. Keep a safety copy.
Before deleting anything, we kept a complete copy of every file on another multidev, plus a locked copy in a second AWS region. Then, overnight on September 29, we removed 2.4 million files from Pantheon, checking each one against S3 first.
Folders the standard tool would have broken.
Nearly half the images sat in oddly named folders. The plugin's own migration tool would have broken them, so we built our own.
Social share images.
About 77,000 social share image links would have broken in the move. We rewrote them.
Six files that didn't make it.
We checked every file against S3 before deleting it. That check caught 6 files that never made it over.
Settings that don't travel with the code.
A few settings nearly tripped up the launch three times. We caught all three in rehearsal.
A doubled license count.
After launch, WHYY's old domains doubled the plugin's license count. We tracked that down and fixed it.
"New Relic hasn’t looked this good in quite some time - amazing what a buried setting can do.”
WHYY's media is protected again.
The media library now lives in Amazon S3, and only about 31 GB of other files remain on Pantheon. That is small enough to back up, so backups run nightly again after years of not working. A second copy of the media sits in another AWS region, locked against deletion.
Pages load faster for readers.
Alongside the move, ad and tracking scripts now load after the page, so readers see content first. On the live site, the home page's mobile Lighthouse score rose from around 50 to about 80.
The server does less work, faster.
After the migration, we kept tuning. Comparing New Relic before and after our October fixes:
- Response time down from 1.25 s to 0.58 s
- Database time down 82%
- Requests over 3 seconds down 95%, from 24,000 a day to 1,300
Existing links still work.
Old image links redirect automatically. Links from NPR, newsletters, search engines and social media keep working. Editors work in WordPress the same way as before, and media library searches now take about 6 seconds instead of timing out.
Built to keep growing.
WHYY's media library will keep growing. Now it can grow without putting backups or page speed at risk. That's what we mean when we say a website should be secure, stable and scalable.
Faster pages for readers
Google Lighthouse scores for WHYY's live site, measured in November, compared with early September, before launch.
Less work for the server
New Relic on the live site, comparing the weeks before and after our October tuning.
Backups, search and storage
Backups run nightly again, editors' media searches take about 6 seconds, and the whole media library has moved off Pantheon.