General difficulties related to data query and data displaying

Incident Report for Fluid Attacks

Postmortem

Impact

An unknown number of users found problems related to outdated, missing, and incorrectly displayed data in various parts of the Platform, such as To do tabs, vulnerability search, indicators, and ToE. The issue started on UTC-5 22-08-02 17:00 and was reactively discovered 15.4 months (TTD) later by a client who reported through our help desk [1] that there were inconsistencies with the findings' indicators. The problem was resolved in 8.1 days (TTF), resulting in a total window of exposure of 15.7 months (WOE) [2].

Cause

The Platform uses advanced filters that rely on a search engine fueled by data from the database through specific software. This implementation approach proved unreliable, causing multiple failures and periods of service disruption [3].

Solution

A different approach was considered when transitioning to AWS Lambda [4].

Conclusion

The issue arose in production due to the initial adoption of a suboptimal approach, failing to integrate seamlessly with our repository's continuous deployment dynamics. With the new implementation operating smoothly, the team is actively investing efforts in refining the technical decision-making process to anticipate and effectively address similar changes in the future. INCOMPLETE_PERSPECTIVE

Posted Nov 20, 2023 - 10:39 GMT-05:00

Resolved

The incident has been resolved and the Fluid Attacks Platform is now working normally.
Posted Nov 17, 2023 - 09:15 GMT-05:00

Monitoring

The team is currently performing the service migration. The affected areas of the Platform are gradually returning to their normal operating status.
Posted Nov 16, 2023 - 17:11 GMT-05:00

Update

The team has completed the necessary preparations and will proceed with a progressive service migration.
Posted Nov 15, 2023 - 15:27 GMT-05:00

Update

The team is planning and preparing a service migration. We keep working to solve this problem.
Posted Nov 14, 2023 - 08:14 GMT-05:00

Update

The team has made some changes to mitigate the impact. We are continuing to work on a fix for this issue.
Posted Nov 10, 2023 - 14:53 GMT-05:00

Identified

Difficulties are being experienced with data analysis and search, impacting different areas of the Fluid Attacks Platform.
Posted Nov 09, 2023 - 08:36 GMT-05:00
This incident affected: Platform.