Impact

An unknown number of users experienced issues accessing the platform. The issue started on UTC-5 24-06-12 at 11:27 and was reactively discovered 28.8 minutes (TTD) later by customers who reported through our help desk [1] that they were unable to log in with their accounts, encountering the `Unauthorized access` message. The problem was resolved in 1.6 hours (TTF), resulting in a total impact of 2.4 hours (TTR).

Cause

We have separate secret files for development and production. The secret was rotated in the development file, but this rotation was overlooked in the production file. This caused problems when logging in to the Platform, resulting in the `Unauthorized Access` message [2].

Solution

The secret was rotated in the production file [3].

Conclusion

To prevent similar incidents in the future, we are implementing a solution to merge the two secrets, ensuring that only one needs to be rotated [4]. ROTATION_FAILURE