Some users experienced failures in several of our automated flows: repositories could not be cloned, reattacks on automatic reports did not run, and SBOM requests were not fulfilled. The issue started on UTC-5 26-08-18 09:07 and was reactively discovered 6 hours (TTD) later by a staff member from the hacking team who noticed that reattacks on automatic reports had been queued for more than three hours without progressing. The problem was resolved in 1.2 hours (TTF), resulting in a total window of exposure of 7.2 hours (WOE).
These three flows are not run directly by the platform. The platform builds and enqueues an execution, and a set of state machines then carries out the actual work; the handoff happens at a stage called batch_dispatch. A change updated the definition of those state machines, along with the components downstream of them, to a new format for passing the parameters of each execution..) The component that sits upstream, the one inside the platform that generates and enqueues the executions, was not updated in the same change, so it kept dispatching against the previous version. From that point on, the parameters it sent no longer matched what the state machines expected, and every execution it enqueued failed at the handoff stage instead of running. The mismatch was not caught before reaching production because this kind of infrastructure change is not currently covered by an end-to-end test in our continuous integration pipeline, and the developer making the change did not have full visibility into how the two sides of the flow depend on each other.
The version of the state machines referenced by the platform's dispatching component was updated to the latest one, bringing both sides of the flow back into agreement..) Cloning, reattack and SBOM executions resumed processing immediately afterwards, including the work that had accumulated while the flows were failing. To prevent a recurrence, the team will evaluate testing alternatives that can detect this class of version mismatch from the continuous integration pipeline, before a change of this kind reaches production.
This incident showed that a change to the state machine definitions and a change to the component that dispatches their executions must be rolled out together, and that nothing in our pipeline currently verifies that the two remain compatible. An end-to-end test covering the dispatch path, together with clearer documentation of how the flow works from end to end, would have surfaced the mismatch before any user was affected.
MISSING_TEST < COMMUNICATION_FAILURE < INCOMPLETE_PERSPECTIVE