Menu

Server outage (Dec 24 2023) Root Cause Analysis

2 months ago
Starting at midnight last night the server became unresponsive to players. This was due to an issue with the Database backup routine failing at startup, leaving the database locked.

The outage lasted a total of 1 hour, with no rollbacks or other impact.

The database is backed up every 8 hours through an automated process. At the start of the process, the primary DB is locked and one of our secondary DB servers takes over. in this case the script failed immediately after locking the…

Server outage (Dec 24 2023) Root Cause Analysis

Tags
Written By