How the Nettl network came through it together
On 21 October, two critical flaws in PHP, the software behind most of the world’s websites, made front-page news.
Government departments, airlines and even Wikipedia took systems offline while they patched.
Nettl partners had a different week.
Their sites stayed protected, their questions were answered in minutes, and every site we host was fully patched by 4:30am the next morning.
Here’s how the network pulled together.
While others went dark
The flaws were serious enough that a PHP core developer suggested switching servers off for a day if a business could afford to.
Many did exactly that. Others handed the problem down the line.
Some hosting companies told their customers to apply the patches themselves, leaving small business owners to work out PHP versions and server updates on their own.
We did the opposite. Every patch was applied centrally, across every site we host, with nothing for partners or their clients to do.
Keeping every partner in the loop
Our developers were tracking the threat from 8:15am on 21 October. By 10:40am, a critical incident topic was live in the Arena, our partner community, and by 11:05am every partner had an email explaining what was happening and what we were doing about it.
We contacted partners whose businesses lean heavily on web work directly, from the UK to our partners in the US and New Zealand.
Through the busiest stretch, every question on the Arena, by email or on the Geek channel was answered within 15 minutes.
We also briefed Oodles, the AI in the Nettl System, so partners could get answers from it at any hour.
Three incident emails went out over two days, each one updating partners on exactly where things stood. No one was left wondering.
Protecting every partner site
Tuesday 20 October
-
PHP’s security team announces two critical flaws
Wednesday 21 October
-
Our developers start tracking the threat
-
Cloudflare reports the first attacks
-
Developers start applying the available updates
-
Decision made to put protective blocks across the whole network
-
Blocks in place on every site we host
-
Critical incident topic live in the Arena, with every question answered within 15 minutes through to 2pm
-
Incident email to every partner, plus direct contact with web-focused partners in the UK, US and New Zealand
-
Second incident email: patches expected at 3pm
-
PHP releases the official fix
-
Nettl System updated; rollout across our hosting begins
Thursday 22 October
-
Every hosted site confirmed fully patched
-
Third incident email with a full rundown of the changes
-
New Arena topic with advice on updating older PHP versions
All times UK time.
More than a network
An independent studio facing this alone would have spent the day reading security bulletins, chasing a host for answers and fielding worried calls from clients.
Nettl partners and system users spent it with a team behind them.
We pointed them to verified news sources to share with clients, and gave them talking points for those client conversations.
When partners raised ideas for handling PHP updates in future, the whole network discussed them in the Arena.
The next day, we opened a fresh topic with advice on updating older PHP versions.
That’s what a community looks like in a crisis. Partners got answers from people who know their business, and their ideas are helping shape how we handle PHP updates from here.
It also gave partners something to offer their own clients. Nettl Care Plans keep client sites updated and monitored, including their PHP version, so partners can give the businesses they work with the same protection the network got this week.
Want a team like this behind your business?
If you run a print, design, signage or web business, Nettl gives you the system, the support and a network of independent studios to back you up when it matters.