Machine report processing error
Incident Report for Fluid Attacks
Postmortem

Impact

At least one group experienced issues with the automatic closure of certain vulnerabilities. The issue started on UTC-5 24-02-16 15:47 and was proactively discovered 6.2 months (TTD) later by a staff member who, while examining the stack trace generated during the Skims execution on the affected group, uncovered the error. The problem was resolved in 21.8 hours (TTF) resulting in a total impact of 6.2 months (TTR) [1].

Cause

The AWS CSPM module’s error handling overlooked a specific type of exception, leading to problems with closing certain vulnerabilities. Multi-region checks in the module caused exceptions in regions without resources, blocking vulnerability report updates [2].

Solution

Adding a verification step in the stack trace to check the type of exception being raised ensures that the now-expected exceptions do not block updates to existing vulnerabilities [3].

Conclusion

The team has already been working on improving the tests for the AWS CSPM module by utilizing the Moto library. This will provide more accurate mock responses similar to those from AWS, enabling better identification and prevention of similar errors in the future [4]. UNHANDLED_EXCEPTION < INCOMPLETE_PERSPECTIVE

Posted Aug 23, 2024 - 10:11 GMT-05:00

Resolved
The incident has been resolved, and now the reports are closing correctly as expected.
Posted Aug 22, 2024 - 01:45 GMT-05:00
Identified
A problem was identified in the Machine report processing. The feature designed to prevent instability in CSPM reports is inadvertently preventing some reports from being closed when they should be.
Posted Aug 21, 2024 - 02:53 GMT-05:00
This incident affected: Scanning.