Some nodes are down
Incident Report for Exalate
Postmortem

Only a small number of nodes were affected, we apologize to the affected users for any inconvenience caused by this issue.

Context

After creating a new Exalate Cloud cluster with improved security and resilience some months ago, we recently started migrating our older customers from our old server park to the new cluster. To make sure this migration takes minimum downtime and we can quickly roll back the migration if necessary the requests to these Exalate nodes are still being proxied through the old server park.

Issue

One of the servers in our old server park has failed, effectively disabling the proxying of requests.

Remediation

  • The DNS entries for affected nodes (jcloudnode3xx.exalate.net) were updated to point directly to our new cluster.
  • We will implement a faster schedule to remove the proxying for other older customers.
Posted Nov 12, 2020 - 11:35 CET

Resolved
This incident has been resolved.
Posted Nov 12, 2020 - 10:59 CET
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 12, 2020 - 10:24 CET
Identified
There is an issue with one of the servers in our park, nodes that identify jcloudnode3xx are affected. DNS changes were made to resolve the issue but it will take some time for DNS caches to be refreshed.
Posted Nov 12, 2020 - 10:09 CET
Investigating
We are currently investigating this issue.
Posted Nov 12, 2020 - 10:07 CET
This incident affected: Exalate Cloud (Hosting platform).