Partial Outage of Exalate Nodes

Incident Report for Exalate

Resolved

This incident has been resolved.
Exalate version 5.7.4 containing the fix has been deployed on all customer instances.
Posted Dec 07, 2023 - 10:00 CET

Update

Dear customers,

We have successfully upgraded GitHub and Zendesk to the latest version, 5.7.4. Additionally, Jira nodes have also been updated to ensure compatibility.

Tomorrow, we will proceed with upgrading the rest of the connectors. This update is crucial to maintaining seamless connections, as both sides need to be on the latest version for optimal performance.

We appreciate your understanding and cooperation during this upgrade process. If you experience any issues, please don’t hesitate to reach out to our support team.

Thank you for your continued trust in our services.

The Exalate team
Posted Dec 05, 2023 - 22:03 CET

Update

We are pleased to inform you that the resource overallocation issue has been successfully resolved.

However, please be advised that some users may experience slowness in the processing of sync. Our dedicated team is actively investigating this matter to ensure a swift resolution.

We appreciate your patience and understanding as we work towards restoring normal processing speeds.
Posted Nov 30, 2023 - 17:03 CET

Update

We would like to extend our sincere apologies for any inconvenience caused by the recent service interruption experienced by some of our users.

Outage Information:
Our team has been actively working on resolving the issue that impacted a number of our customers. This issue led to the service interruption you may have experienced.

Issue Resolution in Version 5.7.3:
A solution has been implemented in our latest release, Exalate Version 5.7.3. This update not only addresses the resource overallocation issue but also includes key improvements to ensure the smooth and reliable functioning of our service.

Next Steps:
No Action is required from your end as the update has been deployed automatically, the consequences of the issue still can impact performance of the application which we are addressing as required.

Your Patience is Appreciated:
We understand the inconvenience this delay may have caused and appreciate your patience.
We can assure you that nodes are actively being monitored to ensure uninterrupted transaction flow.
Thank you for your understanding, and we will keep you updated on the progress.

If you have any further questions or concerns, please don’t hesitate to reach out to our support and customer success teams.
Posted Nov 29, 2023 - 19:03 CET

Monitoring

We are currently experiencing an incident affecting a limited number of nodes in our system. The issue revolves around the Resource overallocation, resulting in downtime for the affected nodes.

Current Situation:
Affected Nodes: Approximately 5% of nodes are currently experiencing downtime.
Recovery Attempts: Our team has been actively working on resolving the issue and has implemented various recovery measures.

Actions Taken:
Hot Fixes Released: We have deployed hotfixes on multiple platforms to address the Resource overallocation. The fixes have been applied to ServiceNow, Azure DevOps, Zendesk, Salesforce, and Jira Cloud (ongoing).
Impact on Users: While the upgrades are in progress, customers may notice a temporary slowness in the processing of their sync.

Ongoing Work:
Node Restoration: Despite our efforts, 5% of nodes are still affected and remain offline. We are diligently working to bring these nodes back online.

Sync Processing Investigation: Once all nodes are operational, we will conduct a thorough investigation into the processing of synchronization to ensure optimal performance.
Github Hot Fix: A hotfix for Github is currently under testing. We aim to release it as soon as it passes the necessary testing procedures.

We appreciate your patience and understanding during this time.
Posted Nov 28, 2023 - 18:59 CET
This incident affected: Exalate Cloud (connect.exalate.net (mapper), connect.exalate.cloud), Zendesk (Exalate Console), Jira Cloud (Synchronisation node), Azure DevOps (Exalate for Azure DevOps), Service Now (Exalate for ServiceNow in Exalate Cloud), GitHub (Exalate for GitHub), and Salesforce (Exalate for SalesForce).