Loss of access to organizations
Incident Report for Fluid Attacks
Postmortem

Impact

At least five users experienced issues when attempting to access certain organizations on the platform. The issue started on UTC-5 24-10-30 09:53 and was proactively discovered 14.4 minutes (TTD) later by a staff member who reported through our help desk [1] that they could not access the organizations. The problem was resolved in 2.4 hours (TTF) resulting in a total impact of 2.8 hours (TTR) [2].

Cause

The issue was caused by enabling a feature in the database that automatically deletes records based on a set expiration time. When users accepted invitations to join an organization, the system did not clear the expiration_time field, leaving past timestamps on some records. This unintentionally deleted organization access records with expired timestamps [3].

Solution

The issue was addressed by updating the system to automatically clear the expiration_time field when an invitation is accepted, preventing active organization access from being deleted. A migration was also implemented to correct existing organization_access records with past expiration times, ensuring they would not be mistakenly removed. Additionally, a migration was added to restore items that had already been deleted due to the expiration policy [4].

Conclusion

Additional checks were added in the testing process to verify that the expiration_time field updates correctly after an invitation is accepted, ensuring the issue does not reoccur [5]. NCOMPLETE_PERSPECTIVE < MISSING_TEST

Posted Nov 06, 2024 - 18:10 GMT-05:00

Resolved
The incident has been resolved, and access to all organizations has now been fully resorted for affected users.
Posted Nov 06, 2024 - 15:11 GMT-05:00
Identified
Issues have been detected where some users cannot access some organizations within the platform.
Posted Nov 06, 2024 - 10:25 GMT-05:00
This incident affected: Platform.