Impact
An unknown number of users experienced issues accessing the platform. The issue started on UTC-5 24-09-13 11:13 and was proactively discovered 14.4 minutes (TTD) later by a staff member, who reported through our help desk that login attempts were failing, displaying an "Unauthorized access" message. The problem was resolved in 3.6 hours (TTF) resulting in a total impact of 3.8 hours (TTR) [1].
Cause
Several users experienced issues due to two main factors. First, those with stored preferences received an "Unauthorized access" message. Second, API Tokens generated before September 2nd became invalid due to a gap in the token rotation process. Although tokens are intended to be rotated every 6 months, this issue caused tokens to expire just a week after their previous rotation [2].
Solution
The main solution involves fixing the token rotation process to prevent future issues. As an immediate measure, users were instructed to clear their cookies. Furthermore, error handling was enhanced so that if the issue arises again, users will be prompted to re-enter their OTP for verification [3].
Conclusion
Detecting such issues before production is challenging due to the inherent nature of token rotation, which adheres to a standard expiration period of 6 months. To address this, we have improved error-handling processes [4]. We will document the rationale behind the minimum rotation frequency for tokens to ensure transparency and prevent similar issues in the future. ROTATION_FAILURE < INCOMPLETE_PERSPECTIVE