Cardano went DOWN!? 🚨 Is my ADA Investment Safe?

Everything's great I came back and you can 
see that she always breaks Sunday morning or   Monday morning you know late late at night what 
we don't know is we don't know the causal event   That would knock the system into that particular 
State something happened with cardano last night   Seems like there's a lot of 
confusion even for cardano holders   The cardano community interacted with this tweet 
in droves I appreciate cardano its Community the   Whole crypto space can learn from what happened 
with cardano last night so let's talk about it   Did cardana go down is my investment safe 
to fully understand why 60 of all Ada nodes   Went offline the other day this is a cardano 
developers take some of you will be aware of   An incident last night at approximately this time 
between these blocks which caused approximately 50   To 60 percent of cardano nodes to disconnect and 
restart this appears to have been triggered by a   Transient anomaly causing one of two reactions in 
the Ada node some disconnected from appear others   Through an exception and restarted I still don't 
understand what exactly happened listen to this   60-second clip Charles hoskinson on what exactly 
went down over the weekend today is January 23rd   2023 just a little bit of a postmortem on the 
weekend we had an interesting issue let me go   Ahead and share my screen if you look over here a 
lot of people recognize that the network stalled   For about two minutes so everything's great I came 
back and you can see that [ __ ] always breaks   Sunday morning or Monday morning you know late 
late at night while everybody's sleeping that's   Just how things work uh and anyway long and short 
is that it seems to be a transient issue and it's   Probably a combination of a collection of things 
that happened at the same time which means that   The reproducibility is unlikely so we pulled the 
team together Sunday night it was a Sam Jared   Um Arno and myself and we talked to the spos 
Marcus guffler and uh and a few others were   Willing to come on and we took a look at 
all the logs we have and because we don't   Have a rolling P CAP we couldn't actually see 
what the transaction or block that actually   Caused this trip of the of the balance R 
code that was put in for optimization was   Um we know where the air was called in the program 
but it's hard to actually as digging through to   Actually figure out what was the triggering 
event for that uh that particular issue   Um but it doesn't seem like it's reproducible so 
cardano was stalled and if I'm understanding the   Problem correctly such transient issues even 
if they were to affect all cardano nodes were   Considered in the design of the cardano node and 
consensus the system behaved exactly as expected   Block production was only briefly impacted with a 
portion of the network falling out of sync before   The nodes restarted therefore impact was low 
akin to the delays that occurred during normal   Operations and often seen as Epoch boundaries 
most nodes automatically recovered depending on   The stake pool operator deployment choice so the 
system behaved exactly as expected interesting   Let's listen to Charles hoskinson expand on 
this the good news is cardano did exactly  

What it was supposed to do when a stall occurs uh 
basically the system uh recovers itself heals so   The nose go back up and you'll notice you know 
everything's great and it's back you know and   That's exactly what we designed the nodes to do 
it's not very satisfying because I ideally every   Time a distributed system has a blip like this 
or a stall like this you'd like to know an exact   Cause but the problem is that distributed systems 
sometimes create what are called emergent bugs so   Locally it's not reproducible but a collection 
of things create a collective Global state   That for some reason triggers something in the 
system basically stops uh for some people uh so   Uh as I said there's a team working on it they're 
looking at it and they're chipping away at it uh   And we'll do a post-mortem inside the get issues 
and um obviously if there's a bug in either a   Haskell Library uh or a bug in the implementation 
of the balance R which was put in for optimizing   Uh cardano uh then you know it'll get patched but 
the problem is bugs do require understanding of   What triggered them and that's the thing we're 
looking for right now and it's going to take a   Little bit of time for us to get there some cases 
you can't actually figure these ones out and they   Come up once every five years once every six years 
and and you just say okay you know the network   Recovered we move on uh but uh sometimes you can 
uh but you never want to drive yourself nuts over   It listen in any distributed system stuff crashes 
all the time and that's okay listen why in any   Distributed system whether it be the Googleplex 
or Microsoft Azure or Amazon web services so these   Massive large-scale systems uh stuff crashes 
all the time systems go down all the time for   A variety of reasons it could be Hardware failures 
or software failures hacking attempts cosmic rays   All kinds of stuff they're even a famous cosmic 
ray issue with Google bigtable and that's okay   Because the second thing you do when you build a 
large-scale distributed system is you build it for   Resilience and self-healing so when these types of 
things happen the system recovers which is exactly   What occurred here it was really cool to see how 
the spos came together and how quickly uh they   Were on top of this and they're still looking in 
fact it's a race condition everybody seems to be   Looking for it and I'm really proud of uh Smog 
and Andrew Westberg and all the others that just   Showed up literally moments after this occurred 
middle of the damn night and started collecting   Large amounts of information and trying to 
debug the the issue so it just shows you how   Decentralized and resilient cardano has become you 
give me your thoughts about this whole ordeal in   The comments section below and the final question 
we need to answer is my investment safe it's   Important to remember this transactions were not 
lost blocks were not dropped no money was lost the   Network actually didn't halt you know it stalled 
for a little bit and recovered but the network   Was still moving forward it's still progressing so 
it's not like Panic calls had to go to exchanges   And they say up grade to this emergency node or 
all the consensus people had to get together and   Discord and figure out how to kick and restart 
cardano it's self-healed and that's the point of  

A decentralized distributed resilient system is 
it should have this capability to self-heal and   Rebuild itself in the event that it stalls from 
a transient event and you know I'm proud that   This happened get your tickets to bitcoin 2023 
Miami May 18th through 20th Miami Beach this   Year use code altcoin daily 10 off this is the 
biggest Bitcoin Conference of the Year ticket   Prices continue to increase as we get closer 
to the event use code altcoin daily 10 off get   Your tickets now if you're interested in making 
money in cryptocurrency Make sure you subscribe   To altcoin daily one video every day keeping you 
informed on the entire cryptocurrency market I'll   Leave you with this cardano holders will this 
issue that affected 60 percent of cardano nodes   Will this issue ever be a problem for cardano 
ever again this particular issue will never   Come up again likely in this format especially 
if it's an emergent transient air and because   Things are a moving Target when optimizations 
come in it may actually recover that part of   Code when Genesis comes in there's a different 
operating model and network upgrades come in   There's a different operating model obviously 
input endorsers are going to completely change   Everything so that's the other part of debugging 
is that as you'd like to find something you may   Actually just change the code that had the problem 
and begin with while you upgrade the module to a   Different state and not even know that you fixed 
the particular Puck that the issue is caused with