Gitlab’s 2017 Database Outage Postmortem Takeaways

-

The complete article of the postmortem can to read here: Postmortem of database outage of January 31. In summary, Gitlab.com has a database outage occurred that on 31 January 2017. This outage resulted in the lost of data spans over +6 hours.

There are several takeaways that I learned from this postmortem.

  1. Setup several recovery mechanism that works in case the first choice cannot be relied. Gitlab have 3 ways of database backup: running scheduled pg_dump to S3, disk snapshots using Azure, and LVM snapshot that usually used to copy data from production to staging environment.
    • The pg_dump cronjob failed because of a different version used for pg_dump (for PostgreSQL 9.2) and the database version (PostgreSQL 9.6).
    • The Azure disk snapshot is used to only backup NFS server (I am guessing data for their application and others..?). Moreover, this is a disk recovery data which means if Gitlab needed to only restore database data. I imagine they should manually to choose which part of the disk contains the database backup data if they use this restoration method.
    • LVM snapshot is the recovery mechanism that Gitlab chose for this incident. However, since they were transferring backup snapshot from staging (in Azure classic storage which has low network throughput) to production, it took them 18 hours to restore it along with webhooks that are not in staging database but are in production one.
  2. Recovery mechanism (and perhaps the procedure too) should be tested regularly. Gitlab only knew their recovery mechanisms are not robust as they thought are until the disaster already happened. The mechanism and the procedure should also be done in a safe environment in anticipation of human mistakes from engineers (which most likely going to happen, increasingly so in stressed time).
  3. Error reporting should be ensured that it has a way to also alert reports that failed to be announced (whether failure on sending email or messaging application). Gitlab’s error reporting relied on email with DMARC, a mechanism to validate the trustworthiness of an email sender. However, the error reporting for cronjob does not use DMARC which results in the report being rejected as an untrusted sender.
  4. User deletion should be soft deletion. I am guessing their hard deletion on the mistakenly flagged engineer user may cause foreign key errors on PostgreSQL which contributes to the lag they were facing, on top of the increased spams.

Comments

Post a Comment

Popular posts from this blog

Experiment #1: Can JavaScript run something like a concurrent thread/process? No.

-
So, I know it is stated frequently that JavaScript is "a single-threaded" language. But it is non-blocking thanks to Web APIs. However, I was thinking whether it means that we can run something concurrently like in some other languages like Java, Go, or Python. I mean ES6 includes Promises and it was enhanced by async-await. So, is it possible to make use of Promises to run multiple things in the same time? It was fairly a simple experiment. All you need is A function that acts as a Promise (an async function), A method that will be run as a Promise and as a synchronous function, and A "container" function that holds both async function and plain function. With that three method in mind, this is the code that I came up with. function timelapse ( id ) { for ( let i = 1 ; i <= 1 _000_000; i++) { if (i % 500 _000 === 0 ) { console .log( ` ${ new Date ().getTime()} ${id} ` ); } } } async function asyncFunction ( id ) {
Read more

Ruby on Rails Time, Date, DateTime Cheatsheet

-
Rails has a lot of handy methods built-in that sometimes all you have to do is know it. In consequence, there are an overwhelming methods available to get specific time and/or date. This cheatsheet should provide some practical methods for Time, Date, DateTime that may help you in developing with Rails. I have also included an example output of each method so you don't have to manually enter it in rails console to know what the outcome would be. Here is a brief overview of what methods will be covered in this cheatsheet. Feel free to skip to the methods that you need or scroll around to know what methods there are. Maybe it will help you in the future. Enjoy! Current time and date ( now , current , today ) Specific day ( yesterday , tomorrow ) Specific moment in time ( beginning_of_minute , end_of_minute , beginning_of_hour , end_of_hour ) Specific time of the day ( beginning_of_day , middle_of_day , end_of
Read more

SQL WHERE and HAVING clause (with examples)

-
Image
In SQL, there is two clause used for filtering: WHERE and HAVING . Most of the time WHERE clause is sufficient. However, the reason why HAVING clause exist is because WHERE clause cannot filter aggregates. To show an example of how each clause can be used, let's say we are running an ice cream company. We produce several flavours of ice cream. We sell these ice creams to stores in multiple countries. Each store would have different stocks of our ice cream varieties. Our users are from all over the world. From that example, we have the following schema of the database. In the following section, I will show some examples for both clauses in PostgreSQL . I have created the database dump  so you can experiment with it yourself. WHERE clause We can use the WHERE clause to search stores that is available in a county with the following query. SELECT * FROM stores WHERE country = 'ID'; We can also use it to find users who cannot access o
Read more