Incident Date: 29/Aug/2025
Duration: ~4 hours from detection to rollback completion
Impact: Issue link synchronization behavior change affecting multiple customer instances
Status: Resolved with data restoration completed
The incident was first detected on 29/Aug at 17:56 when a customer reported issue links being removed after upgrading to v5.28. Within eight minutes, at 18:04, the support team received multiple similar reports and immediately requested a rollback. The formal incident response was initiated at 18:17, and the rollback process to v5.27.0 began at 18:48. The rollback was completed across all affected instances by 22:12 the same day, approximately four hours after initial detection.
Recovery efforts continued over the weekend. On 30/Aug at 09:43, the data restoration team was assembled and began developing recovery procedures. The comprehensive customer communication deployment occurred on 01/Sep at 17:05, providing affected customers with restoration instructions and individualized support.
(All times CET)
The technical issue stemmed from a synchronization enhancement in v5.28 that compared issue links between source and destination systems. The implementation operated under the assumption that links present on the destination but not on the source represented synchronization inconsistencies requiring removal. This approach failed to distinguish between links managed by Exalate and links created independently outside the synchronization scope.
Several contributing factors enabled this issue to reach production. The scope definition did not account for mixed environments where synchronized and non-synchronized links coexist on the same system. Test coverage focused primarily on standard synchronization flows rather than these mixed data environments. Additionally, the modification affected a broader scope than initially anticipated during the development and review process.
The impact characteristics were significant but recoverable. Multiple customer instances experienced removal of issue links that existed outside the intended synchronization scope. However, service functionality continued throughout the incident, and comprehensive audit logs preserved all affected link data, which enabled complete restoration of the removed links.Regarding data security and privacy, the audit logs contain only metadata elements such as issue identifiers, link types, and relationship mappings. No sensitive customer content, comments, or detailed issue information was captured in these logs. This metadata-only approach ensured that the restoration process could proceed while maintaining appropriate data protection standards.
On the positive side, the incident response team made a rapid rollback decision, quickly identifying this as the appropriate technical solution rather than attempting a forward fix. The comprehensive logging infrastructure proved invaluable, enabling complete data restoration from audit trails. Cross-functional coordination worked well across engineering, customer success, and operations teams. The structured communication approach maintained clarity and transparency throughout the resolution process.
The incident also revealed important areas for system enhancement. Synchronization logic in complex integration scenarios requires better handling of mixed data environments where multiple systems manage different subsets of the same data types. Testing scope needs expansion to cover mixed synchronized and non-synchronized environments comprehensively. Change evaluation processes require enhancement for modifications that affect customer data. Monitoring capabilities need improvement to enable proactive detection of synchronization behavior changes before customer impact.
Several process improvements have been (or are being) implemented as a result. Enhanced code review procedures now include additional requirements specifically for data modification operations. Expanded testing protocols cover scenarios involving mixed data environments and include dedicated data preservation validation. Improved monitoring systems detect synchronization pattern changes and unusual data modification activities. Updated release procedures include enhanced evaluation criteria for data-affecting changes and improved rollback automation capabilities.