Cardano developer IOG’s response on why 60% of all nodes went offline the other day

I’ve tried to find the original source but it’s been posted on Telegram, so here it is:

—-

Some of you will be aware of an incident last night at approximately 00:09:00 UTC (between block 8300569 and 8300570) which caused approx. 50% of Cardano nodes to disconnect and restart. This impacted relay nodes and block producing nodes - edge nodes appear to have been unaffected.

This appears to have been triggered by a transient anomaly causing one of two reactions in the node; some disconnected from a peer, others threw an exception and restarted. Such transient issues (even if they were to affect all nodes) were considered in the design of the cardano-node and consensus. The systems behaved exactly as expected.

Block production was only briefly impacted with a portion of the network falling out of sync for approximately https://cardanoscan.io/block/8300569 before nodes restarted. Therefore impact was low - akin to the delays that occur during normal operations and often seen at epoch boundaries. Most nodes automatically recovered – depending on the SPO deployment choice. We’re now investigating the root cause for this anomalous behavior and implementing further logging measures alongside our regular monitoring procedures. Thanks as ever to all the Cardano SPO community for its support in this.

—-

I’ll be interested in seeing the root cause, but it’s hard to see a negative here. It’s software, bugs happen, but it appears that Cardano’s design allows for things of this nature to happen and the blockchain will simply restart itself to keep going. This is what decentralisation and smart software design is about - no single point of failure, it’s something this space should be championing as one of the core pillars of our shared technology.