A look into what happened behind the scenes during the longest outage in Roblox history.

Sources:
https://blog.roblox.com/2022/01/roblo...
https://www.hashicorp.com/resources/h...
https://roblox.fandom.com/wiki/2021_R...
https://roblox.fandom.com/wiki/Timeli...
https://news.ycombinator.com/item?id=...
https://raft.github.io/
http://www.lmdb.tech/media/20130329d...
http://www.lmdb.tech/doc/
https://db.cs.cmu.edu/mmapcidr2022/
   • Free To Use Gameplay | Roblox | 4K | ...  

Chapters:
0:00 Intro
0:33 HashiStack Explanation
4:47 Outage Investigation
8:20 Root Causes Found
11:30 Return to Service
12:19 Slow Leaders
15:56 Resolution

Corrections:
At 9:44, the default unbuffered channel in Go does not hold any items and has a buffer size of 0. Sends to such a channel are blocked until another goroutine is ready to receive the value. The illustration in the video shows a *buffered channel of size 1* however the overall point still stands.

Music Credits:
Firecracker by LEMMiNO (   • LEMMiNO  Firecracker (BGM)  )
Impact Prelude by Kevin MacLeod
We're Finally Landing by Home