{
  "page": {
    "id": "ssg1m9rm289m",
    "name": "Fluid Attacks",
    "url": "https://status.fluidattacks.tech"
  },
  "status": {
    "indicator": "none",
    "description": "All Systems Operational"
  },
  "components": [{"status":"operational","name":"Platform","created_at":"2023-09-14T08:46:15.119-05:00","updated_at":"2026-11-04T17:39:19.744-05:00","position":1,"description":"The app that manages your application vulnerabilities and provides a complete security status overview","showcase":true,"start_date":"2023-06-02T00:00:00.000-05:00","id":"ljz5nvbn8tlv","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Web","created_at":"2023-09-14T01:31:54.355-05:00","updated_at":"2025-10-10T11:15:43.393-05:00","position":2,"description":"Our home page","showcase":true,"start_date":"2023-05-05T00:00:00.000-05:00","id":"nrf1rq8vdtnc","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Docs","created_at":"2023-09-14T01:42:04.633-05:00","updated_at":"2026-02-27T07:22:36.077-05:00","position":3,"description":"The homebase for our documentation","showcase":true,"start_date":"2023-05-05T00:00:00.000-05:00","id":"bb2pmf0j2md8","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Agent","created_at":"2023-09-29T14:37:40.896-05:00","updated_at":"2026-11-10T21:28:35.727-05:00","position":4,"description":"The DevSecOps Agent","showcase":true,"start_date":"2023-05-05T00:00:00.000-05:00","id":"4111gls5s8ht","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Cloning","created_at":"2023-09-14T08:47:24.983-05:00","updated_at":"2026-11-04T17:39:19.768-05:00","position":5,"description":"Our repository cloning tool","showcase":true,"start_date":"2023-05-05T00:00:00.000-05:00","id":"jmw3n0vyk2vf","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Scanning","created_at":"2023-10-05T08:16:24.438-05:00","updated_at":"2025-11-28T07:55:44.371-05:00","position":6,"description":"Our source code analysis tool","showcase":true,"start_date":"2023-05-05T00:00:00.000-05:00","id":"2jx10grc1r99","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Extensions","created_at":"2023-12-06T12:13:35.184-05:00","updated_at":"2026-03-24T11:29:51.380-05:00","position":7,"description":"Our Visual Studio Code extension","showcase":true,"start_date":"2023-03-27T00:00:00.000-05:00","id":"865dj18m1w5m","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Mailing","created_at":"2023-10-02T18:21:51.563-05:00","updated_at":"2025-10-10T11:15:43.435-05:00","position":8,"description":"Our mail delivery tool","showcase":true,"start_date":"2023-05-29T00:00:00.000-05:00","id":"gw11gg3czf82","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Integrations","created_at":"2025-10-10T10:38:13.088-05:00","updated_at":"2025-10-10T11:15:43.442-05:00","position":9,"description":null,"showcase":true,"start_date":"2025-09-12T00:00:00.000-05:00","id":"0zftw9vsd202","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Criteria","created_at":"2025-10-30T11:27:04.527-05:00","updated_at":"2026-03-18T15:26:50.910-05:00","position":10,"description":"Fluid Attacks Database","showcase":true,"start_date":"2025-10-30T00:00:00.000-05:00","id":"0j85hdt1khgc","group_id":null,"page_id":"ssg1m9rm289m"},{"status":"operational","name":"Dependencies","created_at":"2023-09-14T11:11:13.006-05:00","updated_at":"2025-10-30T11:27:22.227-05:00","position":11,"description":null,"showcase":false,"start_date":null,"id":"644cd4xqgpl6","group_id":null,"page_id":"ssg1m9rm289m","components":["h2t01rljvp32","khc699hndrbr","3b0wcjmnv5f2","2q75jcgvn6yz","2wvtq4c24f2n","4r6m6lblqcs5","v7m5r1bqkbk2","qh1w5v522drz","v0wlr8qdpvqk","f5r6hm93wrvc","18nv41k38g5c","mghny52c13pb","8l4w16jq7y63","jdw91nl1qvwp","15cwz50091sz","mnf21bb9l7b8","0w32pxn31y48","5296y31npm7b","7lr6q7sc1vhw","ql47d7fpwl7j","c0gxwncwx00z","mjng5q70lqvw","s72187wlffwb","0xdw8hnn5tk3","4l5rkbnlfs02","lnfqxqxbwdsd","31jw2jkfw39t","dcc7pnblhn5w","jtkj5l9q7y84","vjylvhyw4nq8","cvsbbp5l523b","p48gqby1vqp0","w7l4txyy4v0q","9nf9dqfc1h5t","pn89vyh2bnvn","jm0c83w2rh35","17949ypwd9jj","7cd25rlbpsxh","52ylw4yl805l","445mjrlxw7cq","wgp7yldlnb5f","yjd82p6sktt0","2mscjtc14jrd","wppzddq5vprv","7n4h9jjmqwnx","dg2cmjbntjs0","r69442kr8w5b","xbt8rksrjrk5","xtlcf73x6084","9vkrbycrps5x","l15pzh7573vr","9tr3473gklq1","1h0zzpxg2x8q","nj9pg0rxw112","0zljbhjlbtgc","223kwkppysjp","3z6tyry4qwcg","xbz61drl3yxb","rhfs2771bc8w","mc84mmpp29zb","b88jksksf6kc","p8npfbd9kjns","d5gx31vwcfkm","51kwkn444yxb","1ypxyk3gw7pt","b22my5yc97nd","sglxgrv4pmqc","ky7mpyy6zs5h"]},{"status":"operational","name":"API","created_at":"2026-09-04T20:14:23.118-05:00","updated_at":"2026-10-26T20:29:38.502-05:00","position":12,"description":"GraphQL API at app.fluidattacks.com/api","showcase":true,"start_date":"2026-09-04T00:00:00.000-05:00","id":"nl0dx177vqd1","group_id":null,"page_id":"ssg1m9rm289m"}],
  "incidents": [{"name":"Bitbucket Repository Cloning and OAuth Connection Unavailable","status":"resolved","created_at":"2026-11-03T17:11:04.792-05:00","updated_at":"2026-11-04T23:29:33.797-05:00","monitoring_at":"2026-11-04T17:39:19.803-05:00","resolved_at":"2026-11-04T17:43:52.956-05:00","impact":"major","shortlink":"https://stspg.io/13k0nvn6mpml","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-11-03T17:11:04.784-05:00","reminder_intervals":null,"id":"8hxgkxv576bm","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2026-11-04T17:43:52.956-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-11-04T17:43:52.956-05:00","display_at":"2026-11-04T17:43:52.956-05:00","deliver_notifications":true,"tweet_id":null,"id":"tv9w4fk0xgb4","incident_id":"8hxgkxv576bm","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"operational","new_status":"operational"},{"code":"jmw3n0vyk2vf","name":"Cloning","old_status":"operational","new_status":"operational"}]},{"status":"monitoring","body":"A fix has been implemented and we are monitoring the results.","created_at":"2026-11-04T17:39:19.803-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-11-04T17:39:19.803-05:00","display_at":"2026-11-04T17:39:19.803-05:00","deliver_notifications":true,"tweet_id":null,"id":"n9rq7w2l24w5","incident_id":"8hxgkxv576bm","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"partial_outage","new_status":"operational"},{"code":"jmw3n0vyk2vf","name":"Cloning","old_status":"partial_outage","new_status":"operational"}]},{"status":"identified","body":"We are continuing to work on a fix for this issue","created_at":"2026-11-04T10:40:13.257-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-11-04T10:40:13.257-05:00","display_at":"2026-11-04T10:40:13.257-05:00","deliver_notifications":true,"tweet_id":null,"id":"g0lvgcj9xrmn","incident_id":"8hxgkxv576bm","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"partial_outage","new_status":"partial_outage"},{"code":"jmw3n0vyk2vf","name":"Cloning","old_status":"partial_outage","new_status":"partial_outage"}]},{"status":"identified","body":"Since October 29 at 18:57 GMT-05:00, repositories connected through Bitbucket OAuth credentials cannot be cloned, and their cloning status shows “Invalid credentials, access could not be granted prior to the cloning task”. Adding or re-authorizing Bitbucket OAuth credentials from the Credentials section of the organization also fails, with the Bitbucket error “Invalid client_id”. Repositories that use HTTPS or SSH credentials, or other providers (GitHub, GitLab, Azure DevOps), are not affected, and no data was lost or exposed. Our team is working on a fix.","created_at":"2026-11-03T17:11:04.901-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-11-03T17:11:04.901-05:00","display_at":"2026-11-03T17:11:04.901-05:00","deliver_notifications":true,"tweet_id":null,"id":"zk8sh7fnpzjn","incident_id":"8hxgkxv576bm","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"operational","new_status":"partial_outage"},{"code":"jmw3n0vyk2vf","name":"Cloning","old_status":"operational","new_status":"partial_outage"}]}],"postmortem_body":"#### Impact\n\nSome users were unable to clone repositories connected to Bitbucket through OAuth, and could not connect or reconnect Bitbucket through OAuth. The issue started on UTC-5 26-10-29 18:57 and was reactively discovered 2.9 days \\(TTD\\) later by a staff member who reported, on behalf of a client, that one of its groups had lost access to its Bitbucket repositories. The problem was resolved in 1 day \\(TTF\\), resulting in a total window of exposure of 3.9 days \\(WOE\\).\n\n* **Cloning**: repositories connected through Bitbucket OAuth were marked as failed, so they did not receive new code for analysis, and their pending reattack requests were put on hold.\n* **Connections**: users who tried to add or reconnect a Bitbucket OAuth credential received an error from Bitbucket instead of the authorization screen. New connections had probably been failing since October 22.\n* **Scope**: signing in with Bitbucket, repositories from other providers, and Bitbucket repositories connected with other credential types kept working normally.\n\n#### Cause\n\nOur platform accesses Bitbucket repositories through an OAuth app, which must be associated with a Bitbucket account. The account that held this app became inactive, and as a result Bitbucket disabled the app and the access it had granted. This change occurred on Bitbucket’s side, with no changes to our platform. Our platform interpreted the resulting errors as a temporary Bitbucket issue, and we did not have an alert for failures affecting all Bitbucket connections at once, so the problem was not detected until it was reported.\n\n#### Solution\n\nWe registered a new Bitbucket OAuth app under a company-managed account, with the same read-only access as before, and switched the platform to use it[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/121323) New Bitbucket OAuth connections now work as expected. Since access granted through the previous app cannot be transferred, affected credentials need to be reconnected from the organization’s Credentials section; our team is reaching out to each affected organization to help with this. We will also document a rotation procedure for this app, add an alert for widespread Bitbucket connection failures, improve how the platform handles this type of error, and use a separate app for our development environment.\n\n#### Conclusion\n\nThis incident showed that a key integration depended on an account that was not centrally managed, and that our monitoring did not distinguish a widespread failure from a temporary provider issue. Managing integration apps under company accounts with documented ownership and rotation steps, and monitoring connection failures across all credentials, would have prevented this outage and allowed us to detect it sooner. **ROTATION\\_FAILURE \u003c INCOMPLETE\\_PERSPECTIVE \u003c MISSING\\_ALERT**","postmortem_body_last_updated_at":"2026-11-04T23:29:33.760-05:00","postmortem_ignored":false,"postmortem_published_at":"2026-11-04T23:29:33.760-05:00","postmortem_notified_subscribers":true,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Platform","created_at":"2023-09-14T08:46:15.119-05:00","updated_at":"2026-11-04T17:39:19.744-05:00","position":1,"description":"The app that manages your application vulnerabilities and provides a complete security status overview","showcase":true,"start_date":"2023-06-02T00:00:00.000-05:00","id":"ljz5nvbn8tlv","page_id":"ssg1m9rm289m","group_id":null},{"status":"operational","name":"Cloning","created_at":"2023-09-14T08:47:24.983-05:00","updated_at":"2026-11-04T17:39:19.768-05:00","position":5,"description":"Our repository cloning tool","showcase":true,"start_date":"2023-05-05T00:00:00.000-05:00","id":"jmw3n0vyk2vf","page_id":"ssg1m9rm289m","group_id":null}]},{"name":"MCP Server, AI Chat and PR Assistant Unavailable","status":"resolved","created_at":"2026-10-26T18:01:37.000-05:00","updated_at":"2026-11-04T23:30:57.552-05:00","monitoring_at":null,"resolved_at":"2026-10-26T20:29:38.546-05:00","impact":"critical","shortlink":"https://stspg.io/dw9n52rn8c0h","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-10-26T18:01:37.000-05:00","reminder_intervals":null,"id":"d3b5sps96f3f","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2026-10-26T20:29:38.546-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-10-26T20:29:38.546-05:00","display_at":"2026-10-26T20:29:38.546-05:00","deliver_notifications":true,"tweet_id":null,"id":"jz9sc3vm11mh","incident_id":"d3b5sps96f3f","custom_tweet":null,"affected_components":[{"code":"nl0dx177vqd1","name":"API","old_status":"major_outage","new_status":"operational"}]},{"status":"identified","body":"Since 11:35 GMT-05:00, several AI-powered services on our platform are unavailable: the AI agent chat fails with \"Failed to connect to chat service\", connections to the Fluid Attacks MCP server (https://app.fluidattacks.com/mcp/messages/) from MCP clients such as Claude Desktop, Claude Code and other AI assistants fail with the error \"Unexpected content type: text/html\", and pull request reviews from the PR Assistant may not complete.","created_at":"2026-10-26T18:01:37.310-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-10-27T15:52:26.876-05:00","display_at":"2026-10-26T18:01:37.000-05:00","deliver_notifications":false,"tweet_id":null,"id":"33rfw1fkp01j","incident_id":"d3b5sps96f3f","custom_tweet":null,"affected_components":[{"code":"nl0dx177vqd1","name":"API","old_status":"operational","new_status":"major_outage"}]}],"postmortem_body":"#### Impact\n\nSome users experienced an outage of the AI-powered services on our platform: the AI agent chat failed with \"Failed to connect to chat service\", sessions from external MCP clients could not start, and some pull request reviews from the PR Assistant did not finish processing. The issue started on UTC-5 26-10-26 11:35 and was reactively discovered 5.8 hours \\(TTD\\) later by a staff member who noticed that the AI agent had stopped responding. The problem was resolved in 1.2 hours \\(TTF\\), resulting in a total window of exposure of 7 hours \\(WOE\\).\n\n* **AI Agent**: users who opened the AI agent on the platform did not get an answer.\n* **MCP server**: connections from MCP clients such as Claude Desktop, Claude Code and IDEs failed with \"Unexpected content type: text/html\".\n* **PR Assistant**: some pull request review runs never finished processing.\n* **Scope**: the API, vulnerability management, reports and scans kept working normally, and no data was lost or exposed.\n\n#### Cause\n\nThe services inside our infrastructure locate each other through an internal name resolution service. A change that enabled a new network security layer on some groups of servers modified how name resolution works on those servers[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/118341) On those servers the internal name resolution service ended up forwarding requests to itself, detected the loop, and stopped. Nothing restricted this service to the servers where it works correctly, and since servers are replaced every day, one of its two instances landed on an affected server on October 19 and stopped working. The service kept running on a single instance for a week without any alert. On October 26, an unrelated infrastructure change automatically updated this service to a newer version[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/119719) The restart moved the last healthy instance onto an affected server, leaving no instances available, so every AI service that depends on internal name resolution stopped responding. The new version itself was not the cause; both versions behave the same way on the affected servers.\n\n#### Solution\n\nThe service was restored by restricting the internal name resolution service to servers without the new network layer, after which both instances started correctly. The permanent fix keeps this service on a dedicated group of base servers, the same placement already used by other core components, and raises that group to a minimum of two servers[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/120043) A complementary change fixes the versions of the base components of our infrastructure, so updates are deliberate and reviewed instead of a side effect of unrelated changes[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/120037) The team will also add an alert on the availability of this service.\n\n#### Conclusion\n\nThis incident showed that the change to the network configuration of our servers was validated for the services it targeted but not for the base components of our infrastructure that could run on those servers, so the defect stayed hidden until the daily server rotation exposed it. Without an alert on the availability of name resolution and with component versions not fixed, a routine infrastructure update turned a silent degradation into an outage. Placing base components on dedicated servers, fixing their versions and monitoring their availability would have prevented any user from being affected. **INFRASTRUCTURE\\_ERROR \u003c MISSING\\_ALERT \u003c INCOMPLETE\\_PERSPECTIVE**","postmortem_body_last_updated_at":"2026-11-04T23:30:57.548-05:00","postmortem_ignored":false,"postmortem_published_at":"2026-10-27T15:56:55.395-05:00","postmortem_notified_subscribers":true,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"API","created_at":"2026-09-04T20:14:23.118-05:00","updated_at":"2026-10-26T20:29:38.502-05:00","position":12,"description":"GraphQL API at app.fluidattacks.com/api","showcase":true,"start_date":"2026-09-04T00:00:00.000-05:00","id":"nl0dx177vqd1","page_id":"ssg1m9rm289m","group_id":null}]},{"name":"API has failed","status":"resolved","created_at":"2026-08-25T01:42:36.722-05:00","updated_at":"2026-08-25T10:04:55.095-05:00","monitoring_at":null,"resolved_at":"2026-08-25T10:04:55.078-05:00","impact":"none","shortlink":"https://stspg.io/l6gfvg2x8047","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"none","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-08-25T01:42:36.717-05:00","reminder_intervals":"2","id":"jvt55cr0tk3j","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"This incident has been resolved. Between 01:42 and 02:02 UTC-5, monitoring detected a single failing request out of 118 total checks (117 passing, 0 degraded). Average response time remained stable at 546 ms.\n\nThe failure was isolated to one query and did not represent a systemic issue or a sustained service disruption. No further failing or degraded requests have been observed since, and the API is operating normally.","created_at":"2026-08-25T10:04:55.078-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-08-25T10:04:55.078-05:00","display_at":"2026-08-25T10:04:55.078-05:00","deliver_notifications":false,"tweet_id":null,"id":"t1pf644xvxvq","incident_id":"jvt55cr0tk3j","custom_tweet":null,"affected_components":null},{"status":"identified","body":"An issue was found in API.\u003cbr\u003eClick for details: https://availability.fluidattacks.com","created_at":"2026-08-25T01:42:36.755-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-08-25T01:42:36.755-05:00","display_at":"2026-08-25T01:42:36.755-05:00","deliver_notifications":true,"tweet_id":null,"id":"8p7wl2pnxp23","incident_id":"jvt55cr0tk3j","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"API has failed","status":"resolved","created_at":"2026-08-16T18:11:26.951-05:00","updated_at":"2026-08-18T18:08:55.762-05:00","monitoring_at":null,"resolved_at":"2026-08-18T18:08:55.743-05:00","impact":"none","shortlink":"https://stspg.io/28t774nv7n7v","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"none","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-08-16T18:11:26.944-05:00","reminder_intervals":"2","id":"k4l82tcd5hhs","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2026-08-18T18:08:55.743-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-08-18T18:08:55.743-05:00","display_at":"2026-08-18T18:08:55.743-05:00","deliver_notifications":false,"tweet_id":null,"id":"7c7m8k2vy7vk","incident_id":"k4l82tcd5hhs","custom_tweet":null,"affected_components":null},{"status":"identified","body":"An issue was found in API.\u003cbr\u003eClick for details: https://availability.fluidattacks.com","created_at":"2026-08-16T18:11:26.998-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-08-16T18:11:26.998-05:00","display_at":"2026-08-16T18:11:26.998-05:00","deliver_notifications":true,"tweet_id":null,"id":"l3lrqkjd669b","incident_id":"k4l82tcd5hhs","custom_tweet":null,"affected_components":null}],"postmortem_body":null,"postmortem_body_last_updated_at":null,"postmortem_ignored":false,"postmortem_published_at":null,"postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Automated Cloning, Reattack and SBOM Flows Failing","status":"resolved","created_at":"2026-08-18T09:00:00.000-05:00","updated_at":"2026-08-19T23:28:46.391-05:00","monitoring_at":null,"resolved_at":"2026-08-18T09:00:00.000-05:00","impact":"none","shortlink":"https://stspg.io/996hfwqgrvy8","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-08-19T23:24:13.620-05:00","reminder_intervals":null,"id":"dz96jtkrwb4h","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"Some of our automated flows are currently failing: repository cloning, reattacks on automatic reports, and SBOM requests are not completing. The cause has been traced to an issue in our infrastructure, and our team is already working on restoring these flows. Existing information remains complete and intact, and there is no impact to security or to any other part of the platform.","created_at":"2026-08-19T23:24:13.664-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-08-19T23:24:13.664-05:00","display_at":"2026-08-18T09:00:00.000-05:00","deliver_notifications":false,"tweet_id":null,"id":"9tgktmp8c6gc","incident_id":"dz96jtkrwb4h","custom_tweet":null,"affected_components":null}],"postmortem_body":"### Postmortem\n\n#### Impact\n\nSome users experienced failures in several of our automated flows: repositories could not be cloned, reattacks on automatic reports did not run, and SBOM requests were not fulfilled. The issue started on UTC-5 26-08-18 09:07 and was reactively discovered 6 hours \\(TTD\\) later by a staff member from the hacking team who noticed that reattacks on automatic reports had been queued for more than three hours without progressing. The problem was resolved in 1.2 hours \\(TTF\\), resulting in a total window of exposure of 7.2 hours \\(WOE\\).\n\n* **Repository cloning**: cloning executions did not run, so repository contents were not refreshed during the window.\n* **Reattacks on automatic reports**: reattack requests were accepted but stayed queued without being executed.\n* **SBOM requests**: requested SBOM files were not generated or delivered.\n\n#### Cause\n\nThese three flows are not run directly by the platform. The platform builds and enqueues an execution, and a set of state machines then carries out the actual work; the handoff happens at a stage called `batch_dispatch`. A change updated the definition of those state machines, along with the components downstream of them, to a new format for passing the parameters of each execution[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/110512).) The component that sits upstream, the one inside the platform that generates and enqueues the executions,  was not updated in the same change, so it kept dispatching against the previous version. From that point on, the parameters it sent no longer matched what the state machines expected, and every execution it enqueued failed at the handoff stage instead of running. The mismatch was not caught before reaching production because this kind of infrastructure change is not currently covered by an end-to-end test in our continuous integration pipeline, and the developer making the change did not have full visibility into how the two sides of the flow depend on each other.\n\n#### Solution\n\nThe version of the state machines referenced by the platform's dispatching component was updated to the latest one, bringing both sides of the flow back into agreement[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/111060).) Cloning, reattack and SBOM executions resumed processing immediately afterwards, including the work that had accumulated while the flows were failing. To prevent a recurrence, the team will evaluate testing alternatives that can detect this class of version mismatch from the continuous integration pipeline, before a change of this kind reaches production.\n\n#### Conclusion\n\nThis incident showed that a change to the state machine definitions and a change to the component that dispatches their executions must be rolled out together, and that nothing in our pipeline currently verifies that the two remain compatible. An end-to-end test covering the dispatch path, together with clearer documentation of how the flow works from end to end, would have surfaced the mismatch before any user was affected.\n\n**MISSING\\_TEST \u003c COMMUNICATION\\_FAILURE \u003c INCOMPLETE\\_PERSPECTIVE**","postmortem_body_last_updated_at":"2026-08-19T23:28:46.373-05:00","postmortem_ignored":false,"postmortem_published_at":"2026-08-19T23:28:46.373-05:00","postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Cloning Failures for Repositories Using ZTNA or Egress Connectivity","status":"resolved","created_at":"2026-08-06T18:00:00.000-05:00","updated_at":"2026-08-13T00:14:57.509-05:00","monitoring_at":null,"resolved_at":"2026-08-06T18:00:00.000-05:00","impact":"none","shortlink":"https://stspg.io/1v1jp83cy8g8","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-08-13T00:12:12.003-05:00","reminder_intervals":null,"id":"p1vh7t86ktv9","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"Some groups are experiencing cloning failures on repositories configured with ZTNA or Egress connectivity. Affected repositories stay in a \"Cloning queued...\" state without completing, which prevents our automated scanners from retrieving recent changes and leaves vulnerability reporting for those repositories outdated. The cause has been identified in our connector infrastructure and a fix is underway.","created_at":"2026-08-13T00:12:12.062-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-08-13T00:12:12.062-05:00","display_at":"2026-08-06T18:00:00.000-05:00","deliver_notifications":false,"tweet_id":null,"id":"rbhypwcs043s","incident_id":"p1vh7t86ktv9","custom_tweet":null,"affected_components":null}],"postmortem_body":"## Postmortem\n\n### Impact\n\nAt least one user experienced automated cloning failures on repositories configured with ZTNA or Egress connectivity, which left the affected roots stuck in a `QUEUED` state and prevented automated scanners from retrieving recent changes, causing vulnerability reporting for those repositories to go stale. The issue started on 2026-08-06 at 17:48 \\(UTC-5\\) and was reactively discovered 1.9 days later by a staff member who received a help-desk report that a customer's root had been stuck in a \"Cloning queued...\" state for more than two hours with no vulnerabilities being displayed. The problem was resolved within an additional 1.8 days, resulting in a total window of exposure of 3.7 days. 26 groups were identified with affected repositories.\n\n### Cause\n\nAs part of a scheduled secrets rotation cycle, the WARP connector credentials were rotated to values that were not valid for the connector's registration. The rotation procedure documented for this credential pair was not followed, and the new values were never validated against the connector before being deployed. With an invalid registration, the connector could not establish the tunnel to the customer network prior to cloning, and every cloning job on the affected queue failed. Two properties of the platform kept the failure invisible: the connector error was unhandled, so the root's cloning status was never transitioned out of `QUEUED` and the interface continued to display \"Cloning queued...\" instead of a failure state; and a credential error inside the root-filtering step caused cloning to abort for every root in the group rather than only those bound to the broken credential. No alert was raised for the sustained cloning failures, so the outage only became visible when a customer escalation reached the help desk.\n\n### Solution\n\nThe credential rotation was reverted to the previous valid value, restoring the connector's registration and unblocking cloning for the affected roots. Cloning was then hardened so that failures of this class cannot be silent again: group cloning actions were wrapped in a general exception handler so that an unexpected failure in one group no longer blocks the rest; roots are now partitioned by credential health before validation, so a credential error marks only its own roots as failed instead of aborting the entire group gather; and logging was upgraded to emit real tracebacks on failures. A proper rotation process for this credential pair will be established with tighter controls and a step-by-step guide, and sustained cloning failures will now trigger an explicit alert.\n\n### Conclusion\n\nA credential rotation performed outside its documented procedure disabled the connector's network registration. Because the resulting cloning failures were neither surfaced in the root status nor alerted on, the outage persisted until a customer escalated it. Validating rotated credentials against the services that consume them before deployment, and alerting on sustained cloning failures, would have contained this within hours instead of days. **ROTATION\\_FAILURE \u003c MISSING\\_ALERT \u003c INCOMPLETE\\_PERSPECTIVE**","postmortem_body_last_updated_at":"2026-08-13T00:14:57.496-05:00","postmortem_ignored":false,"postmortem_published_at":"2026-08-13T00:14:57.496-05:00","postmortem_notified_subscribers":false,"postmortem_notified_twitter":false,"components":[]},{"name":"Analytics Charts Showing Outdated Data","status":"resolved","created_at":"2026-07-21T12:13:37.291-05:00","updated_at":"2026-07-23T20:53:24.129-05:00","monitoring_at":null,"resolved_at":"2026-07-22T14:30:49.297-05:00","impact":"none","shortlink":"https://stspg.io/1svs03f5ts2n","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":null,"scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-07-21T12:13:37.283-05:00","reminder_intervals":null,"id":"b3nwnpf5c2qp","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2026-07-22T14:30:49.297-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-07-22T14:30:49.297-05:00","display_at":"2026-07-22T14:30:49.297-05:00","deliver_notifications":true,"tweet_id":null,"id":"zc1txw6c1fvy","incident_id":"b3nwnpf5c2qp","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"operational","new_status":"operational"}]},{"status":"investigating","body":"The analytics charts (by group, by organization, and portfolio views) are currently displaying outdated information. The data behind these charts last refreshed on 2026-06-24 and has not updated since, so recent activity, including groups and repositories created after that date, is not yet reflected. This issue is limited to the charts only: the underlying data remains complete, intact, and unaffected, and there is no impact to security or to any other part of the platform. Our team is actively investigating the cause and working to restore regular chart updates. Further updates will be posted as we make progress.","created_at":"2026-07-21T12:13:37.342-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-07-21T12:13:37.342-05:00","display_at":"2026-07-21T12:13:37.342-05:00","deliver_notifications":true,"tweet_id":null,"id":"k0vss9dgvn8c","incident_id":"b3nwnpf5c2qp","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"operational","new_status":"operational"}]}],"postmortem_body":"### Postmortem\n\n### Impact\n\nUsers experienced stale analytics charts across the platform: the documents powering the \"by group\", \"by organization\", and \"portfolio\" views stopped being refreshed, leaving the vast majority of organizations displaying outdated data as of June 24. Groups and repositories created after that date did not appear in the charts even when data was available. The issue started on June 23, 2026 and was discovered approximately 20 days later by a staff member who noticed the charts were not up to date. Once identified, the problem was fully resolved within 1 day.\n\n### Cause\n\nA scheduled background job responsible for regenerating analytics charts was failing silently. When processing a large number of organizations, a single error during the generation of one organization's charts caused the entire job to stop, preventing all remaining organizations from being updated. Over time, increased load on the platform's data layer began triggering these errors more frequently and across more organizations, amplifying the impact. Because no alerts were configured to notify the team when this job failed, the issue went undetected for several weeks.\n\n### Solution\n\nThe chart generation process was updated to handle errors on a per-organization basis: if one organization's charts fail to generate, it is logged and skipped, and processing continues for all others. Additional safeguards were introduced to reduce pressure on the underlying data layer during high-load periods. Alerting was also configured so that failures in this job are now immediately surfaced to the engineering team, preventing similar silent failures in the future.\n\n### Conclusion\n\nA background job with no error isolation and no failure alerting can silently affect customer-facing data for an extended period before anyone notices. The key improvements, isolating errors per subject and adding alerts on job failures,  ensure this class of issue will be detected and addressed much faster going forward. We apologize for the disruption and are committed to maintaining the accuracy and freshness of your analytics data.","postmortem_body_last_updated_at":"2026-07-23T20:53:24.113-05:00","postmortem_ignored":false,"postmortem_published_at":"2026-07-23T20:53:24.113-05:00","postmortem_notified_subscribers":true,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Platform","created_at":"2023-09-14T08:46:15.119-05:00","updated_at":"2026-11-04T17:39:19.744-05:00","position":1,"description":"The app that manages your application vulnerabilities and provides a complete security status overview","showcase":true,"start_date":"2023-06-02T00:00:00.000-05:00","id":"ljz5nvbn8tlv","page_id":"ssg1m9rm289m","group_id":null}]},{"name":"Scheduler Incorrectly Opening Events Across Multiple Groups","status":"resolved","created_at":"2026-07-09T10:55:05.226-05:00","updated_at":"2026-07-09T23:52:34.665-05:00","monitoring_at":null,"resolved_at":"2026-07-09T12:50:54.273-05:00","impact":"minor","shortlink":"https://stspg.io/7hcbb5w963v5","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"minor","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-07-09T10:55:05.218-05:00","reminder_intervals":null,"id":"52jw3xgx18p1","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2026-07-09T12:50:54.273-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-07-09T12:50:54.273-05:00","display_at":"2026-07-09T12:50:54.273-05:00","deliver_notifications":true,"tweet_id":null,"id":"k0mcdj5hbbqd","incident_id":"52jw3xgx18p1","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"operational","new_status":"operational"}]},{"status":"investigating","body":"Some groups are receiving unexpected open events related to environment URL availability. These events were generated incorrectly and do not reflect actual issues in the affected groups. Our team is investigating and working on a fix.","created_at":"2026-07-09T10:55:05.277-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-07-09T10:55:05.277-05:00","display_at":"2026-07-09T10:55:05.277-05:00","deliver_notifications":true,"tweet_id":null,"id":"sfyg0lby431p","incident_id":"52jw3xgx18p1","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"operational","new_status":"operational"}]}],"postmortem_body":"### Postmortem\n\n#### Impact\n\nAt least one user experienced a flood of false-positive `ENVIRONMENT_ISSUES` events on their production groups, each dispatching an automatic email notification, for URL environments that were in fact reachable. The issue started on 2026-07-08 at 20:00 \\(UTC-5\\) and was reactively discovered 14.9 hours later by a staff member who noticed an unusually high volume of events of this type opened simultaneously across multiple production groups, all sharing an identical \"URL is not reachable\" description. The problem was resolved within an additional 2.6 hours, resulting in a total window of exposure of 17.5 hours.\n\n#### Cause\n\nA new URL environment connectivity check ran for the first time in production on its automated schedule. The logic that determines which URLs are eligible for the check contained a flaw: instead of correctly excluding URLs that require authentication or that are only reachable through special network channels, it included them in an unauthenticated probe. Upon receiving unauthorized-access responses, such as redirects to login portals, the system incorrectly classified those URLs as unreachable and opened an incident event for each one, also dispatching a notification email to the group's stakeholders. The affected URLs were fully functional; they simply required credentials that the automated probe never supplied. Approximately 850 spurious events were created across production groups.\n\n#### Solution\n\nThe approximately 850 incorrectly generated events were closed through an automated process that identified them by type, description, and creation time, and closed them directly without emitting further notification emails to users. The automated scheduler was disabled to prevent the issue from recurring. As a permanent fix, the connectivity check will no longer run automatically and will instead become an on-demand action that internal users can trigger manually from the interface. Additionally, the eligibility logic will be corrected to properly exclude authenticated environments and those operating through special network channels, and an exhaustive dry-run against real production data will be required before any re-enablement.\n\n#### Conclusion\n\nThis incident revealed that the URL eligibility logic and the handling of authenticated environments were never validated end-to-end before the first production run, allowing an unauthenticated probe to be misread as a connectivity failure at scale. Tests covering the eligibility logic and the authenticated-URL flow, along with a dry-run against real production data before enabling any automated task, would have prevented these events from being created. **MISSING\\_TEST \u003c INCOMPLETE\\_PERSPECTIVE**","postmortem_body_last_updated_at":"2026-07-09T23:52:34.649-05:00","postmortem_ignored":false,"postmortem_published_at":"2026-07-09T23:52:34.649-05:00","postmortem_notified_subscribers":true,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Platform","created_at":"2023-09-14T08:46:15.119-05:00","updated_at":"2026-11-04T17:39:19.744-05:00","position":1,"description":"The app that manages your application vulnerabilities and provides a complete security status overview","showcase":true,"start_date":"2023-06-02T00:00:00.000-05:00","id":"ljz5nvbn8tlv","page_id":"ssg1m9rm289m","group_id":null}]},{"name":"Evidence Page - Unexpected Application Error","status":"resolved","created_at":"2026-05-26T15:28:05.039-05:00","updated_at":"2026-05-30T00:50:58.501-05:00","monitoring_at":"2026-05-26T16:17:05.819-05:00","resolved_at":"2026-05-26T16:26:05.245-05:00","impact":"minor","shortlink":"https://stspg.io/4c7jlgr6hylj","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"minor","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-05-26T15:28:05.031-05:00","reminder_intervals":null,"id":"2hf0zj2bd20w","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"The issue causing the Evidence page to crash has been identified and resolved. The page is now loading correctly for all users. We apologize for the disruption and appreciate your patience while we worked through it.","created_at":"2026-05-26T16:26:05.245-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-05-26T16:26:05.245-05:00","display_at":"2026-05-26T16:26:05.245-05:00","deliver_notifications":true,"tweet_id":null,"id":"2jmqmrn901qc","incident_id":"2hf0zj2bd20w","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"partial_outage","new_status":"operational"}]},{"status":"monitoring","body":"A fix has been implemented and we are monitoring the results.","created_at":"2026-05-26T16:17:05.819-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-05-26T16:17:05.819-05:00","display_at":"2026-05-26T16:17:05.819-05:00","deliver_notifications":false,"tweet_id":null,"id":"lhvhjzbm55rf","incident_id":"2hf0zj2bd20w","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"partial_outage","new_status":"partial_outage"}]},{"status":"identified","body":"The issue has been identified and a fix is being implemented.","created_at":"2026-05-26T15:59:19.841-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-05-26T15:59:19.841-05:00","display_at":"2026-05-26T15:59:19.841-05:00","deliver_notifications":false,"tweet_id":null,"id":"97d48qvp64wf","incident_id":"2hf0zj2bd20w","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"partial_outage","new_status":"partial_outage"}]},{"status":"investigating","body":"Users may experience an unexpected error when accessing the Evidence page. The application fails to load due to a null reference issue while processing evidence data. Our team is investigating and working on a fix.","created_at":"2026-05-26T15:28:05.121-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-05-26T15:28:05.121-05:00","display_at":"2026-05-26T15:28:05.121-05:00","deliver_notifications":true,"tweet_id":null,"id":"qtslm6v4kjjk","incident_id":"2hf0zj2bd20w","custom_tweet":null,"affected_components":[{"code":"ljz5nvbn8tlv","name":"Platform","old_status":"operational","new_status":"partial_outage"}]}],"postmortem_body":"## Impact\n\nAll users on the platform experienced the inability to view or upload evidence on finding pages for approximately 58 minutes. Both the web interface and the desktop application were affected. Users without access to certain findings were additionally stuck in a reload loop during the same window. The issue started on UTC-5 26-05-26 14:35 and was proactively discovered 57.6 minutes \\(TTD\\) later by a staff member who noticed that evidence data was not loading and alerted the team. The problem was resolved in 57.6 minutes \\(TTF\\), resulting in a total window of exposure of 1.9 hours \\(WOE\\).\n\n## Cause\n\nA planned dependency upgrade deployed to remediate reported security vulnerabilities introduced an undocumented behavioral change in a core library responsible for mapping GraphQL field names to their internal data keys. Under the previous version, fields whose names were entirely lowercase were left unchanged during this mapping. Under the upgraded version, all fields, regardless of casing were passed through a name converter that inserts underscores before numeric characters. As a result, fields such as those used to store evidence items were silently redirected to keys that did not exist in the underlying data, causing them to return empty values instead of their content. The behavioral change was documented in the library's internal pull request history but was not prominently surfaced in the official changelog, and the existing test suite did not cover the full name-resolution flow against a live schema.\n\n## Solution\n\nAn emergency rollback to the previous dependency versions was deployed the same day, restoring all affected functionality within the incident window. A definitive fix was subsequently developed and deployed two days later: the dependency upgrade was re-applied alongside a custom name-mapping configuration that preserves the correct resolution behavior for all-lowercase field names while still applying the standard conversion to mixed-case names. Automated tests were added to explicitly validate that all numbered evidence fields resolve to their expected values, preventing silent regressions of this class in future dependency upgrades.\n\n## Conclusion\n\nMajor-version upgrades to core GraphQL infrastructure libraries must be accompanied by integration tests that exercise the full field-name resolution pipeline against a real executable schema not only unit tests of business logic. The failure was silent by design: fields returned empty values without raising errors, making it indistinguishable from missing data until users reported it. Future upgrades to schema-handling libraries require an explicit end-to-end assertion that no field resolves to null when data is present. **THIRD\\_PARTY\\_CHANGE \u003c MISSING\\_TEST \u003c INCOMPLETE\\_PERSPECTIVE**","postmortem_body_last_updated_at":"2026-05-30T00:50:58.486-05:00","postmortem_ignored":false,"postmortem_published_at":"2026-05-30T00:50:58.486-05:00","postmortem_notified_subscribers":true,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Platform","created_at":"2023-09-14T08:46:15.119-05:00","updated_at":"2026-11-04T17:39:19.744-05:00","position":1,"description":"The app that manages your application vulnerabilities and provides a complete security status overview","showcase":true,"start_date":"2023-06-02T00:00:00.000-05:00","id":"ljz5nvbn8tlv","page_id":"ssg1m9rm289m","group_id":null}]},{"name":"DOCS has failed","status":"resolved","created_at":"2026-05-20T08:37:09.991-05:00","updated_at":"2026-05-20T23:41:23.160-05:00","monitoring_at":null,"resolved_at":"2026-05-20T09:54:34.602-05:00","impact":"none","shortlink":"https://stspg.io/8r62mdsxjtsb","scheduled_for":null,"scheduled_until":null,"scheduled_remind_prior":false,"scheduled_reminded_at":null,"impact_override":"none","scheduled_auto_in_progress":false,"scheduled_auto_completed":false,"metadata":{},"started_at":"2026-05-20T08:37:09.982-05:00","reminder_intervals":"2","id":"rd5ktvh8w637","page_id":"ssg1m9rm289m","incident_updates":[{"status":"resolved","body":"This incident has been resolved.","created_at":"2026-05-20T09:54:34.602-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-05-20T09:54:34.602-05:00","display_at":"2026-05-20T09:54:34.602-05:00","deliver_notifications":true,"tweet_id":null,"id":"r10nzn9h5xrr","incident_id":"rd5ktvh8w637","custom_tweet":null,"affected_components":[{"code":"bb2pmf0j2md8","name":"Docs","old_status":"operational","new_status":"operational"}]},{"status":"identified","body":"We are continuing to work on a fix for this issue.","created_at":"2026-05-20T09:32:29.857-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-05-20T09:32:29.857-05:00","display_at":"2026-05-20T09:32:29.857-05:00","deliver_notifications":false,"tweet_id":null,"id":"tb48j8dwj27r","incident_id":"rd5ktvh8w637","custom_tweet":null,"affected_components":[{"code":"bb2pmf0j2md8","name":"Docs","old_status":"operational","new_status":"operational"}]},{"status":"identified","body":"An issue was found in DOCS.\u003cbr\u003eClick for details: https://availability.fluidattacks.com","created_at":"2026-05-20T08:37:10.056-05:00","wants_twitter_update":false,"twitter_updated_at":null,"updated_at":"2026-05-20T08:37:10.056-05:00","display_at":"2026-05-20T08:37:10.056-05:00","deliver_notifications":true,"tweet_id":null,"id":"y6smn8dx8rc3","incident_id":"rd5ktvh8w637","custom_tweet":null,"affected_components":[{"code":"bb2pmf0j2md8","name":"Docs","old_status":"operational","new_status":"operational"}]}],"postmortem_body":"## Impact\n\nOn May 20, 2026, users visiting [docs.fluidattacks.com](http://docs.fluidattacks.com/) may have encountered a \"Server failed to respond\" error for approximately 1.3 hours. The issue was detected internally by our team and resolved within the same window.\n\n## What happened\n\nDuring a routine deployment operation, an outdated version of the documentation service was accidentally restored to production. This older version was missing a configuration component required for the service to start up correctly, which caused the site to become unavailable.\n\nThink of it like accidentally restoring an old backup that was missing a key setting the service came back up in an incomplete state and was unable to respond to requests.\n\n## What we changed\n\nService was restored by redeploying the latest stable version of the documentation site, bringing `docs.fluidattacks.com` back online. We then applied a permanent fix to ensure the required configuration is always present and correctly loaded when the service starts, preventing the same failure mode from recurring.\n\n## Going forward\n\nWe have updated our deployment process to prevent outdated versions from being restored to production. Safeguards are now in place to ensure that only verified, fully configured versions of the service can be deployed.  \n**COMMUNICATION\\_FAILURE \u003c INCOMPLETE\\_PERSPECTIVE.**","postmortem_body_last_updated_at":"2026-05-20T23:41:23.143-05:00","postmortem_ignored":false,"postmortem_published_at":"2026-05-20T23:41:23.143-05:00","postmortem_notified_subscribers":true,"postmortem_notified_twitter":false,"components":[{"status":"operational","name":"Docs","created_at":"2023-09-14T01:42:04.633-05:00","updated_at":"2026-02-27T07:22:36.077-05:00","position":3,"description":"The homebase for our documentation","showcase":true,"start_date":"2023-05-05T00:00:00.000-05:00","id":"bb2pmf0j2md8","page_id":"ssg1m9rm289m","group_id":null}]}]
}