
All HTTP-based services unresponsive - artf
https://status.bitbucket.org/?v20180202
======
david-giesberg
David from the Atlassian SRE team here. AWS Direct Connect is experiencing an
outage in their US East Region:
[https://status.aws.amazon.com](https://status.aws.amazon.com), which is
causing connectivity issues for most Atlassian products and services. We're
working hard to get everything back up and running. Please check
[http://status.atlassian.com](http://status.atlassian.com) for the latest
updates. We're posting regularly and will continue to provide updates there.

~~~
insomniacity
Why does Bitbucket depend on AWS DC? Why wouldn't I just connect to it over
the internet? Or is part of it in your own DC?

~~~
hkchad
Yea, this makes no sense to me. I have a pretty heavy workload in AWS (us-
east-1), don't use DC AT ALL and nothing is down for me today (except
Atlassian Jira/Confluence Cloud), we self host BB. Why their 'cloud' based
application relies on DC is very odd.

~~~
gtsteve
I don't know but my guess would be anything that isn't core storage - we know
they run their own SAN on their own hardware because that was the cause of
another outage a month or two ago.

At a guess:

\- Bitbucket Pipelines

\- Webhook workers

\- Front-end web servers

\- SSH push/pull workers

Basically anything that's elastic to demand. Presumably the cost of AWS
storage makes it not worth it for the Bitbucket team.

------
itake
its amazon having problems.

[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

~~~
colinbartlett
There are actually quite a lot of services down across the web now. Maybe they
are unrelated... but it could all be related to AWS.
[https://statusgator.com/services/amazon-web-
services](https://statusgator.com/services/amazon-web-services)

My side project, StatusGator, monitors something like 250 status pages and
there's quite a spike in warn or down notices at the moment that I can see.

~~~
edaemon
StatusGator is neat, thanks for linking that. Do you have graphs anywhere to
track the number of outages/problems over time? It would be nice to see if
there's been an uptick in problems generally, across certain services, etc.

~~~
colinbartlett
No, but that's a great idea! I have 3 years of data now from hundreds of
services including severity of reported outage and text about why it went
down. So I could show graphs over time for sure.

------
filchermcurr
GitHub DDoS, Sourceforge DDoS, BitBucket 'routing issues'... somebody hates
version control.

~~~
rambojazz
The target could be the companies, not VCSes per se. As long as they don't
ddos notabug I'm fine with it :)

------
thegeomaster
Another ongoing discussion:
[https://news.ycombinator.com/item?id=16501731](https://news.ycombinator.com/item?id=16501731)

Hosted JIRA is down too (at least for me).

Interestingly, I can seem to be able to find it only via search, it doesn't
show up on the frontpage at all.

------
arkad
XaaS has many benefits, but uptime is not one of them anymore. I self-host my
repos, had a few downtimes but thanks to this DDoS my local services have
better uptime. ( Disclaimer: I know it's not apple to apple comparison as
scale is massively different)

~~~
convolvatron
distributed source code management theoretically doing this in a robust and
replicated manner quite a bit easier. if you ignore partitions, it seems
pretty straightforward to make a git push-all, and a recovery process for
stale nodes coming back.

------
simlevesque
This is the last straw for me. I'm gonna stop using them to host my code. They
have been down way too many times in the last year. It's been six failures
from them in the last two weeks alone. I'm gonna self host Gitea to fix my
issues. I cannot believe that they fail so hard. Why does a failure mean that
I cannot read AND write from BitBucket ? Why are those two things even related
?

~~~
udia
Relying solely on one platform for your organization's code repository needs
is a bad thing in general. I have a workflow where I host my own Gogs instance
on a raspberry pi, as well as on Github. Maybe this would be useful to you?

I've configured my push such that it will deploy to both Github and on my Gogs
meaning that I always have an up to date repository in two places.

[https://github.com/gogits/gogs](https://github.com/gogits/gogs) (if you're
interested in setting up your own)

[https://stackoverflow.com/questions/14290113/git-pushing-
cod...](https://stackoverflow.com/questions/14290113/git-pushing-code-to-two-
remotes) (push to two remotes)

You don't have to use just BitBucket, or rely entirely on your self hosted git
service.

~~~
foxylion
I think hosting the repository is one thing.

But integrating those two repositories with all the automated workflows is
quite a headache. Jenkins won't automatically switch over to another git
backend if one fails. Other automated tools like code review are mostly
relying on a centralized repostiory. I don't see any simple solution to solve
these issues. Bitbucket, GitHub, GitLab don't have any easy fallback solutions
when you relying core workflows on their services.

Self hosting is maybe one solution (we do this with bitbucket), but this
requires major administrative effort to keep it running reliably (always
available, no data losses on hardware/software failures).

~~~
udia
I agree. Having CI with multiple git repositories is still a painful thing. (I
continue to rely primarily on GitHub here. My backup repos don't have CI
enabled.)

I think if your primary repo that's hooked into CI goes down, you're still
SOL. Having your own repo just enables you to continue local development among
your team.

I don't see any great solution, other than making your app distributed to
begin with and doing your build/deploy manually.

As a complete aside, I've fantasized about deploying to all cloud vendors
(Azure/GCE/AWS/Heroku/DigitalOcean/misc.) each with their own specific
build/deploy and having persistent state shared with something like
CockroachDB. Having some load balancer managing state between all instances.
Taking advantage of the free/basic tiers provided by all of the vendors.

------
edwinksl
Looks like access over SSH is still working. Not the worst.

~~~
maemilius
I'm currently only able to read over SSH. Write operations seem to hang
forever.

~~~
gtsteve
Keep trying, it eventually goes through.

------
krallja
Stop using AWS US East!

~~~
igammarays
Curious, why? Is us-east-1 known to be problematic? What about us-east-2
(Ohio)?

~~~
tedmiston
us-east-1 is the oldest region with the oldest hardware which is probably why
it has more issues than others

------
smaili
Appears to be due to an upstream dependency:

> Some component services are currently unreachable due to an upstream
> incident on a cloud provider. We're attempting to route as much traffic as
> possible away from the affected components, and are working with our vendor
> now.

