Server downtime and R106 Delay

1 year ago

The game server has been down since the attempted release on the 29th and Ravalox let the community know that R106 is delayed until Sept. 30th.

Update: Murphy was certainly looking over our shoulders this morning! A number of issues occurred, including uncharacteristic highway traffic that put the release in danger. The build was clean, the base patch applied with no issues, but then the Heritage migration had an exception that caused us to have to restore the database backup we made just after all the player logged out and the server was shut down (which is the first time *ever* that we had to execute our disaster recovery plans to restore the database!). Once this was resolved, we completed the patching process only to find that our internal sanity testing showed issues with recently claimed or moved lots not being visible in the game. (to clarify: none of the items or lots were deleted, everything is still in the item database.)

At this point we shifted into emergency triage and troubleshooting mode, getting as many of the relevant team resources together, including Atos, DevilCult, Undone, Allium, and myself. We hammered away at it all day and into the evening, we now have a new build going over night as a staging build to test for sanity and bring the live server back up. We believe the issue has been identified and are working on a fix that will be included in another build tomorrow.

Written By