I was supposed to fly home from Santiago de Compostella when the blackout happened. Me and my girlfriend had checked out of our hotel and headed to the bus stop to take the bus to the airport. The blackout had already started but we didn't realise (in hindsight, I do remember the pedestrian crossing not working. But I didn't think much of it). Anyways our flight was cancelled and it was clear we needed somewhere to stay the night.
I immediately rebooked the same hotel, but when we got back there the receptionist had left so you had to check in over the phone instead. Except WhatsApp wasn't working. Then mobile data went down. And before long we were walking through the old town going hostel to hostel looking for a place to sleep, as everything got darker and darker (due to the lack of powered street lighting). The old town in almost pitch black was pretty scary!
We ended up breaking back into the hotel, borrowing a bunch of towels from a laundry cart in the hallway and sleeping in this lockable room we found in the basement.
Besides that somewhat stressful part, it was a really strange but fun experience to see the city without power: no traffic lights, darkened shops with lots of phone lights, cafés still operating just with only outdoor seating and limited menus, the occasional loud generator, and most of all the people seemingly having a great time in spite of it.
I would've loved to have stayed out all night exploring the city, but finding somewhere to sleep that night was a bit more pressing!
I think people underestimate how valuable these reports are, so I’m very glad that detailed investigation is done here. Every major grid operator around the world is going to study this and make improvements to make sure this doesn’t happen on their grid.
In a lot of ways it’s like investigations into airplane crashes.
The root cause tree on page 452 gives a good overview of how complex the behavior can be.
The good news is that the grid operators have a good idea of what the problem was/is and it's well understood how to fix it. The downside is that it will require quite a bit of both time and money to reinforce the grid infrastructure.
Spotted the US-american assuming their law system is used world-wide.
If over here a lady buys a hot coffee in a McDrive, drives away, spills the hot coffee on their legs and makes a car accident due to this ... she won't be able to sue the McDrive. There's no fine-print or "Coffee is hot, you dumb person" writing needed anywhere. She could be lucky if she doesn't get fined for endangering others by her stupid actions.
So, if we have a power outage here, the courts don't suddenly get busy. Because there simply no one is suing.
Fun fact: despite this bad power outage, the power grid systems in Europe are still better (even way better) than in the US. There is a comparable statistics measure called "SAIDI" --- system average interuption duration index. And duration wise, per custom and year, the US power grids are worse than over here than in most of West Europe: (US SAIDI 2020: 1.3 hours, German SAIDI 2020: 0.3 hours). That's a factor of more than 4 on the worse-iness of US power grid!
That could be an indicator that suing at the tiniest chance isn't helpful macro-ecnomical. Or that a general suing culture (with legalese trying to protect one from the economic risks) aren't actually helping improving things in the general sense, although they reduce the risk of getting bankrupt. But society-wise, a sue culture is most probably a negative: you spend energy/time/money on things that aren't necessary in saner law systems.
Separate reply for a separate topic: You're repeating the urban-legend version of the McDonalds lawsuit, not the real story, which you can find in numerous places, e.g. https://www.ttla.com/?pg=McDonaldsCoffeeCaseFacts. tl;dr, it was repeated wilful negligence by McDonalds, they'd already injured several hundred other people through it. They knew it was a serious problem but kept doing it anyway.
We had a problem some time ago with a major power outage due to operator negligence. When it came to assigning blame it turned out the corporate structure was such that it was impossible to sue the operator. Since it was in effect publicly-owned, the public would have been suing itself.
The fact that there is not a single root cause but several ones makes me instinctively think this is a good report, because it's not what the "bosses" (and even less politicians) like to hear.
Yes, a lot of modern engineering is good enough that single-cause failures are very rare indeed. That means that failures themselves are rare, but when they do happen, they're most likely to have multiple causes.
How to explain that to non-engineers is another problem.
I think a better way of explaining it to people is that we've made critical systems so reliable that, in order for them to fail, the failures have to be quite complex.
This is almost universal in aviation. They always talk about the "accident chain." Essentially everything that can kill you with one mistake is illegal through training and operational requirements and engineering and maintenance regulations.
we are not as complicated as the national grid, I have been here for nearly 10 years now, and our outages have gone from single cause, two causes, or now its nearly always 3 things that need to go wrong at the same time.
Frequently, when you see these massive failures, the root cause is an alignment of small weaknesses that all come together on a specific day. See, for instance, the space shuttle O-ring incident, Three-Mile Island, Fukushima, etc. These are complex systems with lots of moving parts and lots of (sometimes independent) people managing them. In a sense, the complexity it the common root cause.
This is the same thing that happened with the 35W bridge collapse in Minneapolis. The gusset plates after the disaster were examined and found to be only 1/2" thick when the original design called for them to actually be 1" thick. The bridge was a ticking time bomb since the day it was built in 1967.
As the years went on, the bridge's weight capacity was slowly eroded by subsequent construction projects like adding thicker concrete deck overlays, concrete median barriers and additional guard rail and other safety improvements. This was the second issue, lining up with the first issue of thinner gusset plates.
The third issue that lined up with the other two was the day of the bridges failure. There were approximately 300 tons of construction materials and heavy machinery parked on two adjacent closed lanes. Add in the additional weight of cars during rush hour when traffic moved the slowest and the bridge was a part of a bottleneck coming out of the city. That was the last straw and when the gusset plates finally gave way, creating a near instantaneous collapse.
It's like the Swiss Cheese model where every system has "holes" or vulnerabilities, several layers, and a major incident only occurs when a hole aligns through all the layers.
> See, for instance, the space shuttle O-ring incident
That wasn't really a result of an alignment of small weaknesses though. One of the reasons that whole thing was of particular interest was Feynman's withering appendix to the report where he pointed out that the management team wasn't listening to the engineering assessments of the safety of the venture and were making judgement calls like claiming that a component that had failed in testing was safe.
If a situation is being managed by people who can't assess technical risk, the failures aren't the result of many small weaknesses aligning. It wasn't an alignment of small failures as much as that a component that was well understood to be a likely point of failure had probably failed. Driven by poor management.
> Fukushima
This one too. Wasn't the reactor hit by a wave that was outside design tolerance? My memory was that they were hit by an earthquake that was outside design spec, then a tsunami that was outside design spec. That isn't a number of small weaknesses coming together. If you hit something with forces outside design spec then it might break. Not much of a mystery there. From a similar perspective if you design something for a 1:500 year storm then 1/500th of them might easily fail every year to storms. No small alignment of circumstances needed.
They need more battery storage for grid health, both colocated at solar PV generators (to buffer voltage and frequency anomalies) and spread throughout the grid. This replaces inertia and other grid services provided by spinning thermal generators. There was no market mechanism to encourage the deployment of this technology in concert with Spain’s rapid deployment of solar and wind.
It is very carefully worded, but variable renewables are holding the smoking gun here. This is why spain now requests a better connection to french nuclear now. This reckless overbuild of variable generation is a valuable negative example, wind and solar without adequate hydro or nuclear is dead
As someone who lived through the blackout it was wild. I felt back into the pre-internet, pre-smartphone era. It was pretty cool actually. The rumor mill spread so fast that Within hours the official word on the street was that we were getting hacked by a foreign military and people were joking that we had nothing of interest to be conquered xD
472 pages. That's going to be a nice bit of reading this weekend. It is very nice to see such a comprehensive report as well as the fact that it was made public immediately.
In my case I was near and Bilbao the blackout ended around ~12PM while other towns around -up to 12km far- didn't got the power back until 22PM.
Everyone was outside with pocket radio receivers listening to instructions and having a good time around bars as if it were a European soccer final cup with the local team being playing...
On telecomms, I barely got 2G speeds and no Spanish newspaper (nor media) dared to set a https://lite.cnn.com or https://text.npr.org like page to read the news. I had to literaly use a Gemini (the protocol) client, Lagrange, tunneling the webs thru gemini://gemi.dev with their 'News Waffle' service. And for international news I just resorted to Gopher and magical.fish. The mainstream web barely loaded some of the text (if any), while several IM options ran at crawling network speeds. The lines were clogged and I coudn't even call my relatives.
Can’t read all of this since it’s 424 pages but i want to point out that Australia is beating Europe on grid connected storage. Not on a per capita basis. It’s beating all of Europe combined outright https://www.visualcapitalist.com/top-20-countries-by-battery...
We did have many many problems previously. The state of South Australia went out for a couple of weeks at one point in similar cascading failures. This doesn’t happen anymore. In fact the price of electricity is falling and the grid is more stable now https://www.theguardian.com/australia-news/2026/mar/19/power...
This price drop is inline with the lowered usage of gas turbine peaker plants (isn’t that helpful right now? No need for blockaded gas for electricity).
A lot of people say it can’t be done. That you can’t have free power during the day (power is free on certain plans during daylight due to solar power inputs dropping wholesale prices to negative) and that you can’t build enough storage (still not there but the dent in gas turbine usage is clear).
It’s one of these cases where you’ve been lied to. Australia elected a government that listened to reports battery+solar is great for grid reliability and nuclear was always going to be more expensive.
Lol it took about an hour for us to realize there was a blackout as our house had switched to island mode. Ended being the sort of organization point of the area for the neighborhood as we could provide mobile charging and light until the power came back around 22 at night
112 comments
I immediately rebooked the same hotel, but when we got back there the receptionist had left so you had to check in over the phone instead. Except WhatsApp wasn't working. Then mobile data went down. And before long we were walking through the old town going hostel to hostel looking for a place to sleep, as everything got darker and darker (due to the lack of powered street lighting). The old town in almost pitch black was pretty scary!
We ended up breaking back into the hotel, borrowing a bunch of towels from a laundry cart in the hallway and sleeping in this lockable room we found in the basement.
Besides that somewhat stressful part, it was a really strange but fun experience to see the city without power: no traffic lights, darkened shops with lots of phone lights, cafés still operating just with only outdoor seating and limited menus, the occasional loud generator, and most of all the people seemingly having a great time in spite of it.
I would've loved to have stayed out all night exploring the city, but finding somewhere to sleep that night was a bit more pressing!
In a lot of ways it’s like investigations into airplane crashes.
The good news is that the grid operators have a good idea of what the problem was/is and it's well understood how to fix it. The downside is that it will require quite a bit of both time and money to reinforce the grid infrastructure.
If over here a lady buys a hot coffee in a McDrive, drives away, spills the hot coffee on their legs and makes a car accident due to this ... she won't be able to sue the McDrive. There's no fine-print or "Coffee is hot, you dumb person" writing needed anywhere. She could be lucky if she doesn't get fined for endangering others by her stupid actions.
So, if we have a power outage here, the courts don't suddenly get busy. Because there simply no one is suing.
Fun fact: despite this bad power outage, the power grid systems in Europe are still better (even way better) than in the US. There is a comparable statistics measure called "SAIDI" --- system average interuption duration index. And duration wise, per custom and year, the US power grids are worse than over here than in most of West Europe: (US SAIDI 2020: 1.3 hours, German SAIDI 2020: 0.3 hours). That's a factor of more than 4 on the worse-iness of US power grid!
That could be an indicator that suing at the tiniest chance isn't helpful macro-ecnomical. Or that a general suing culture (with legalese trying to protect one from the economic risks) aren't actually helping improving things in the general sense, although they reduce the risk of getting bankrupt. But society-wise, a sue culture is most probably a negative: you spend energy/time/money on things that aren't necessary in saner law systems.
We had a problem some time ago with a major power outage due to operator negligence. When it came to assigning blame it turned out the corporate structure was such that it was impossible to sue the operator. Since it was in effect publicly-owned, the public would have been suing itself.
How to explain that to non-engineers is another problem.
As the years went on, the bridge's weight capacity was slowly eroded by subsequent construction projects like adding thicker concrete deck overlays, concrete median barriers and additional guard rail and other safety improvements. This was the second issue, lining up with the first issue of thinner gusset plates.
The third issue that lined up with the other two was the day of the bridges failure. There were approximately 300 tons of construction materials and heavy machinery parked on two adjacent closed lanes. Add in the additional weight of cars during rush hour when traffic moved the slowest and the bridge was a part of a bottleneck coming out of the city. That was the last straw and when the gusset plates finally gave way, creating a near instantaneous collapse.
https://en.wikipedia.org/wiki/Swiss_cheese_model
> See, for instance, the space shuttle O-ring incident
That wasn't really a result of an alignment of small weaknesses though. One of the reasons that whole thing was of particular interest was Feynman's withering appendix to the report where he pointed out that the management team wasn't listening to the engineering assessments of the safety of the venture and were making judgement calls like claiming that a component that had failed in testing was safe.
If a situation is being managed by people who can't assess technical risk, the failures aren't the result of many small weaknesses aligning. It wasn't an alignment of small failures as much as that a component that was well understood to be a likely point of failure had probably failed. Driven by poor management.
> Fukushima
This one too. Wasn't the reactor hit by a wave that was outside design tolerance? My memory was that they were hit by an earthquake that was outside design spec, then a tsunami that was outside design spec. That isn't a number of small weaknesses coming together. If you hit something with forces outside design spec then it might break. Not much of a mystery there. From a similar perspective if you design something for a 1:500 year storm then 1/500th of them might easily fail every year to storms. No small alignment of circumstances needed.
page 11 contains "Full root cause tree" - one image with all the high level info
Everyone was outside with pocket radio receivers listening to instructions and having a good time around bars as if it were a European soccer final cup with the local team being playing...
On telecomms, I barely got 2G speeds and no Spanish newspaper (nor media) dared to set a https://lite.cnn.com or https://text.npr.org like page to read the news. I had to literaly use a Gemini (the protocol) client, Lagrange, tunneling the webs thru gemini://gemi.dev with their 'News Waffle' service. And for international news I just resorted to Gopher and magical.fish. The mainstream web barely loaded some of the text (if any), while several IM options ran at crawling network speeds. The lines were clogged and I coudn't even call my relatives.
We did have many many problems previously. The state of South Australia went out for a couple of weeks at one point in similar cascading failures. This doesn’t happen anymore. In fact the price of electricity is falling and the grid is more stable now https://www.theguardian.com/australia-news/2026/mar/19/power...
This price drop is inline with the lowered usage of gas turbine peaker plants (isn’t that helpful right now? No need for blockaded gas for electricity).
A lot of people say it can’t be done. That you can’t have free power during the day (power is free on certain plans during daylight due to solar power inputs dropping wholesale prices to negative) and that you can’t build enough storage (still not there but the dent in gas turbine usage is clear).
It’s one of these cases where you’ve been lied to. Australia elected a government that listened to reports battery+solar is great for grid reliability and nuclear was always going to be more expensive.