Users experienced stale analytics charts across the platform: the documents powering the "by group", "by organization", and "portfolio" views stopped being refreshed, leaving the vast majority of organizations displaying outdated data as of June 24. Groups and repositories created after that date did not appear in the charts even when data was available. The issue started on June 23, 2026 and was discovered approximately 20 days later by a staff member who noticed the charts were not up to date. Once identified, the problem was fully resolved within 1 day.
A scheduled background job responsible for regenerating analytics charts was failing silently. When processing a large number of organizations, a single error during the generation of one organization's charts caused the entire job to stop, preventing all remaining organizations from being updated. Over time, increased load on the platform's data layer began triggering these errors more frequently and across more organizations, amplifying the impact. Because no alerts were configured to notify the team when this job failed, the issue went undetected for several weeks.
The chart generation process was updated to handle errors on a per-organization basis: if one organization's charts fail to generate, it is logged and skipped, and processing continues for all others. Additional safeguards were introduced to reduce pressure on the underlying data layer during high-load periods. Alerting was also configured so that failures in this job are now immediately surfaced to the engineering team, preventing similar silent failures in the future.
A background job with no error isolation and no failure alerting can silently affect customer-facing data for an extended period before anyone notices. The key improvements, isolating errors per subject and adding alerts on job failures, ensure this class of issue will be detected and addressed much faster going forward. We apologize for the disruption and are committed to maintaining the accuracy and freshness of your analytics data.