As outage continues into fifth hour, Facebook tries to restart

There are rumours whipping around the internet that, among other things, Facebook has been “deleted”, that it has been “hacked”, that is has “gone”. The truth appears to be a little less dramatic – though still plenty dramatic.

Essentially, Facebook has deleted its DNS, or the address book that helps people find its websites. When someone types “Facebook.com” into a web browser, the DNS should be able to provide it with the numerical address to get the data that represents the website. But that DNS is broken, so the web browser gets lost, and you get an error message. (The same thing happens, effectively, with the app, although not in such an obvious way.)

That’s so widespread because Facebook controls its DNS. And Facebook does that on behalf of all the services it owns: WhatsApp, Instagram, Messenger, Workplace, Oculus, and more besides. So everything has broken, all at once.

What’s more, Facebook also heavily relies on its own systems internally. So, for example, the fact the DNS isn’t working means that those engineers who should be able to fix it also can’t access it. That’s why staffers are now reportedly being sent to actual physical data centres to try and make things work again. And that’s where we are now.

To borrow an analogy from earlier, it is as if we (or our web browsers) have turned up at the Facebook office in the hope of getting into one of its many rooms. But the receptionist who usually keeps the record of which room is which is gone – they’ve been deleted – and so you’re stuck outside with no idea where to go.

Inside those rooms are not only Facebook but every Facebook property, which together makes up much of the internet. And all the people who would normally be able to hire a new receptionist can’t find their way around the building, too. We’re all stuck outside, looking in, even those people who would normally be on the inside.

It’s now been five hours since Facebook lost its receptionist. And there’s no sense when they’ll return.

© Getty Images instagram down facebook whatsapp.jpg

Twitter slows down

Everything is having a bit of trouble, presumably as many more people than usual switch to it to make up for lack of Facebook. Like Telegram, Twitter is struggling.

Telegram slows down

As Messenger, WhatsApp and other Facebook chat services break, some people have turned to Telegram. So many people, in fact, that it seems to be struggling – while it’s still working, it’s doing so much more slowly than usual.

Some details of outage begin to appear

Some details of what exactly has gone wrong, and how Facebook is rushing to fix it, do appear to be coming through. This, from security researcher Steve Gibson, offers something of a picture, though it’s not clear where it’s come from'

If this is the case, it means in short that a version of that address book we discussed earlier* has been deleted. But to put it back in place, those working remotely from the relevant servers need the address book to find where they’re going. You can see the difficulty.

Most likely, the fix would require actually going to those servers, and doing the work to get them going again. That’s probably what’s happening right now as Facebook engineers struggle to fix everything.

*Scroll down to the post after the one after this one if you want a pained analogy to explain all this.

Some details of outage begin to appear

Some details of what exactly has gone wrong, and how Facebook is rushing to fix it, do appear to be coming through. This, from security researcher Steve Gibson, offers something of a picture, though it’s not clear where it’s come from.

If this is the case, it means in short that a version of that address book we discussed earlier* has been deleted. But to put it back in place, those working remotely from the relevant servers need the address book to find where they’re going. You can see the difficulty.

Most likely, the fix would require actually going to those servers, and doing the work to get them going again. That’s probably what’s happening right now as Facebook engineers struggle to fix everything.

*Scroll down to the post after the one after this one if you want a pained analogy to explain all this.

Facebook chief technology officer offers ‘*sincere*’ apologies

Mike Schroepfer, Facebook’s chief technology officer, has posted his first tweet about the outage.

The wording is much the same as Facebook’s statement much earlier on, and gives no new information. But the fact it’s being posted at all probably says something about how hectic things must be at Facebook at the minute.

Facebook ‘withdrawn’ from the phone book of the internet

Facebook appears to have had its DNS records taken from the global routing tables. That’s according to Brian Krebs, a cyber security expert who runs a popular blog.

In slightly less nerdy speak, that means that effectively Facebook.com, Instagram.com and presumably the rest have had their records wiped from the internet’s address book. When you type one of those URLs into your internet browser, it should be able to speak to Facebook and ask it where it needs to go – but the system that does so has been withdrawn.

It’s like turning up at the Facebook office for a meeting but the receptionist isn’t there. You (or your computer) are just stuck at the desk, since you (or it) don’t know the number of the office door you’re trying to get to. (Or something like that analogy.)

It’s not clear why that happened. Facebook is so big that it runs its own DNS – unlike other, smaller companies – so only someone at Facebook would have the power to stop it running, too.

Here’s Krebs saying much the same thing, in a more legitimate way:

AOC mocks Facebook’s outage

Taking advantage of the outage, Alexandria Ocasio-Cortez has joked that people should use the opportunity to share “our favorite stories of democracy working in hopeful ways and coolest evidence-based reporting”. Examples can be found in the replies to the tweet below.

Users report internet problems – but it’s probably Facebook

Over at Down Detector – a website we’re probably all becoming very familiar with today – reports of issues at carriers and internet networks are surging. In the US, that’s T-Mobile, Verizon and AT&T and others; in the UKit’s Virgin Media, BT, Vodafone, Sky and more.

But it’s probably because Facebook is down, which in turn means Instagram and WhatsApp are down, which in turn means that almost everything a lot of people do with their phone is broken. So it looks like the internet as a whole is broken. But it’s not.

Twitter celebrates as users flock from Facebook and Instagram

At least Twitter is enjoying itself. It has responded to the outage from its official account.

When will Facebook be back and why is it, WhatsApp and Instagram not working?

What’s going on, and when will it stop? Well – nobody really knows. But here’s our best guess.

Read the full explainer here.

Keep up with our newsletter

Like to keep up to date with all the latest tech news? Then sign up to our free weekly IndyTech newsletter for updates and analysis on everything from the state of the International Space Station to latest news on Bitcoin. To sign up click here and scroll down to the tech icon.

WhatsApp outage unconnected to reports it is about to stop working for ‘millions'

There were reports last week that WhatsApp was about to stop working on a range of old phones. They turned out to be a little misleading: while it is about to cut off support for phones with older operating systems, the list of devices that it supposedly affected was incorrect. You can read all about that here.

But that’s unconnected to this. It doesn’t come into effect until 1 November, anyway. But hopefully it’s an interesting story to read while you wait for it to come back online.

Here’s a very simple version of one of the problems that Facebook is having: its domain name system, or DNS, is not working. That might be a symptom or it might be the cause – we should find out soon enough – but it is the reason that when you type “Facebook.com” into your browser, the computer is unable to have that converted into the actual data that makes up the Facebook website.

On Friday, it was Slack. Today it’s Facebook – and also Facebook Workplace. These tools we’re already much relied on, but since the advent of widespread home working, they can put whole offices out of operation.

Here’s our story on the Slack outage the other day.

Outages ‘can often point to a cyber attack’, expert says

Questions are often asked about cyber attacks after an event like this. But while they can point that way, that can also add to confusion, cautions Jake Moore, the former head of digital forensics at Dorset Police and now cybersecurity specialist at global cybersecurity firm, ESET.

“Outages are increasing in volume and can often point towards a cyber-attack, but this can add to the confusion early on when we are diagnosing the causes,” he said.

“As we saw with Fastly in the summer, web-blackouts are more often originate from undiscovered software bug or even human error. 

“Although these are increasing in frequency and require more failsafes in place, predicting these issues is increasingly more difficult as it was never thought possible before”.

Facebook employees take a ‘snow day'

The NYT’s Ryan Mac reports that, since everything is broken, Facebook employees are taking the equivalent of a “snow day”.

WhatsApp also says it is ‘working to get things back to normal'

An update from WhatsApp’s Twitter means that all the different apps have now commented on the problem. (All slightly differently.) Here’s WhatsApp’s:

Instagram asks users to ‘bear with us'

The official “Instagram Comms” account has tweeted about the problems.

Facebook outages can hit other services

Because Facebook provides services that are central to much of the internet – even sites that do not appear to have anything to do with it – outages like this can sometimes cause problems for other seemingly unrelated sites.

In July of last year, for instance, Spotify stopped seemed to stop working. But the culprit wasn’t actually Spotify itself but instead a Facebook piece of technology that was embedded within the app.

Facebook acknowledges issues

Andy Stone, a spokesperson for Facebook, has recognised the issue and says it’s being worked on:

Hootsuite posts warning to users

Hootsuite, the popular social media management tool, has acknowledged that there are issues and says that it is working with the company.

“Facebook and Instagram are currently experiencing issues,” it wrote in an update posted seven minutes ago. “You may encounter issues performing actions for any Facebook or Instagram profile within Hootsuite.

“Our team is working with Facebook and Instagram to resolve the issue as quickly as possible. We thank you in advance for your understanding and patience.”

Why does Facebook keep going down?

This is a fascinating account, from 2019, of a discussion between Mark Zuckerberg and other Facebook staff after a number of outages. It sheds light on why Facebook is so worried about outages, as well as why they kept happening then.

They’ve slowed down, somewhat, but it has had similar big outages to this in recent months.

You can read The Verge’s story here.

Facebook’s ‘Status’ page is down too

Facebook also maintains its own “status” page to give updates on how it is performing. But that site is down too, presumably because the whole site is.

(You can find it here, though of course you won’t see anything.)

No news from Facebook yet

Facebook does have ways of talking to the world: it has official accounts for itself, Instagram and WhatsApp on Twitter, for instance, which it has used for updates on outages in the past.

There’s nothing on any of them yet.

Hello and welcome

... to The Independent’s live coverage of a major Facebook outage.