Slack’s 2021 Outage Postmortem Takeaways

-

Slack had an outage at 4 January 2021 for about ~1 hour on their web application (they mentioned it as “tier” but I am assuming that it relates to their client-facing application). This Slack blog post discusses about how the problem was discovered (it also had a neat timeline diagram) and the cause of the problem. This is my takeaway of Slack’s start of year 2021 outage.

  1. Monitoring services should be made reliable. One major hinderance that Slack faced during the incident was the monitoring services were unable to be used. As a result, they reverted to manual ways (such as querying directly to their backends) to troubleshoot and recover the impacted systems.
  2. Balance between upscaling and downscaling of instances. The article does not go deep dive about this but it seems this scaling have a major part. It only mentioned about disabling downscaling and adding 1200 servers for the upscaling. Now, from what I know, these scalings are usually handled by the cloud provider. My best guess on how they add servers (or should it be instances?) is by increasing minimum (healthy) instances of the server. Not sure how they disable downscaling but I guess the minimum instance may also “disable” downscaling to undesired instance count.
  3. If a service is expecting to experience a high usage after a long low one, there should be some measures taken to handle this short-timed surge. In Slack’s case, their measure is to “request a preemptive upscaling of our [Slack] TGWs at the end of the next holiday season” since one of the incident causes is the slow upscaling of TGW (AWS Transit Gateways).

This is additional resources that the article mentioned that I found interesting. It may supplement the lessons learned outside of the outage’s cause.

  • This is a guide on how to handle overload from Google’s SRE. There are practices on how to manage overload by artificial throttling (either on the backend or client side) when certain threshold is met. The threshold discussed is CPU usage and customer quota (the customer meant here is client application/service that uses a backend). I love this quote from the page because it embodies the general idea on how to gracefully handle overload: “At the end of the day, it's best to build clients and backends to handle resource restrictions gracefully: redirect when possible, serve degraded results when necessary, and handle resource errors transparently when all else fails.”
  • Circuit breaker in programming written by Martin Fowler. From what I understand, it is like a more advanced retry mechanism for functions or processes that may fail outside of logic or the expected. It is worth exploring for those process that relies on remote resources (which is the case for this Slack outage).

Comments

Post a Comment

Popular posts from this blog

Ruby on Rails Time, Date, DateTime Cheatsheet

-
Rails has a lot of handy methods built-in that sometimes all you have to do is know it. In consequence, there are an overwhelming methods available to get specific time and/or date. This cheatsheet should provide some practical methods for Time, Date, DateTime that may help you in developing with Rails. I have also included an example output of each method so you don't have to manually enter it in rails console to know what the outcome would be. Here is a brief overview of what methods will be covered in this cheatsheet. Feel free to skip to the methods that you need or scroll around to know what methods there are. Maybe it will help you in the future. Enjoy! Current time and date ( now , current , today ) Specific day ( yesterday , tomorrow ) Specific moment in time ( beginning_of_minute , end_of_minute , beginning_of_hour , end_of_hour ) Specific time of the day ( beginning_of_day , middle_of_day , end_of...
Read more

Experiment #1: Can JavaScript run something like a concurrent thread/process? No.

-
So, I know it is stated frequently that JavaScript is "a single-threaded" language. But it is non-blocking thanks to Web APIs. However, I was thinking whether it means that we can run something concurrently like in some other languages like Java, Go, or Python. I mean ES6 includes Promises and it was enhanced by async-await. So, is it possible to make use of Promises to run multiple things in the same time? It was fairly a simple experiment. All you need is A function that acts as a Promise (an async function), A method that will be run as a Promise and as a synchronous function, and A "container" function that holds both async function and plain function. With that three method in mind, this is the code that I came up with. function timelapse ( id ) { for ( let i = 1 ; i <= 1 _000_000; i++) { if (i % 500 _000 === 0 ) { console .log( ` ${ new Date ().getTime()} ${id} ` ); } } } async function asyncFunction ( id ) { ...
Read more

SQL WHERE and HAVING clause (with examples)

-
Image
In SQL, there is two clause used for filtering: WHERE and HAVING . Most of the time WHERE clause is sufficient. However, the reason why HAVING clause exist is because WHERE clause cannot filter aggregates. To show an example of how each clause can be used, let's say we are running an ice cream company. We produce several flavours of ice cream. We sell these ice creams to stores in multiple countries. Each store would have different stocks of our ice cream varieties. Our users are from all over the world. From that example, we have the following schema of the database. In the following section, I will show some examples for both clauses in PostgreSQL . I have created the database dump  so you can experiment with it yourself. WHERE clause We can use the WHERE clause to search stores that is available in a county with the following query. SELECT * FROM stores WHERE country = 'ID'; We can also use it to find users who cannot access o...
Read more