Today we had a major service incident on our site. As a result of this incident some of our customers lost their virtual servers.
We are still investigating the cause of the issue and our service will be shut down until the investigation is over.
# Here is what we know #
- There hasn't been any signs of security breach or abnormal activity anywhere on our systems.
- All sensitive information is encrypted throughout the system, including cloud API keys.
- The affected stacks were across Digital Ocean, AWS and Rackspace.
# Here is what we are doing #
- We are working hard to find the root of the issue, but we need to keep the systems shut down until we are sure our customers are not exposed.
# Here is what you can do to restore your service #
- If you are not affected by this issue, you will not be able to redeploy until the service is restored. We will keep you posted.
- If you are affected by this issue, we can help you with your latest deployment Git SHA (if you don't have it), redirecting your traffic from our DNS.
- If you are affected and were running on Digital Ocean, they might be able to restore your server from an automatic pre-destroy snapshot they take.
We are very sorry about this and understand the disruption it has caused to all of our users, we are working hard to restore the service as soon as possible.