Impact

At least 17 organizations experienced issues displaying information in certain platform views, such as Scope, Locations, and ToDo. Additionally, hackers encountered difficulties when attempting to clone repositories related to these organizations. The issue started on UTC-5 24-10-28 12:35 and was reactively discovered 2.6 hours (TTD) later by a client who reported through our help desk [1] that the error message There is an error :( appeared upon accessing findings, and the information failed to load correctly. The problem was resolved in 2.6 hours (TTF) resulting in a total impact of 5.2 hours (TTR) [2].

Cause

An adjustment to optimize the data cleanup process inadvertently altered a condition, leading to an unintended deletion of items [3].

Solution

A database backup was retrieved, and the affected items were restored for the impacted organizations [4].

Conclusion

Setting guidelines for testing data migrations is essential to prevent similar incidents. Best practices include running migrations in a local test environment with a robust sample dataset, validating the database schema within this environment, and performing comprehensive end-to-end tests to confirm all processes work as expected. FAILED_MIGRATION < INCOMPLETE_PERSPECTIVE