
Windows Azure Storage certificate expired? - jborden13
http://social.msdn.microsoft.com/Forums/en-US/windowsazuredata/thread/751c85c5-b3b5-43ba-9d5b-770472ad79e1
======
varunsrin
The Azure Dashboard is here, for anyone looking for updates on this:
[https://www.windowsazure.com/en-us/support/service-
dashboard...](https://www.windowsazure.com/en-us/support/service-dashboard/)

~~~
photorized
I am not sure I trust that dashboard. Day before yesterday, when Azure SQL was
down for the whole day, it took serval hours of 100% outages until the
dashboard got updated with red "problem" icons. Before that, it was
"information", "routine maintenance", etc.

Amazon likes to downplay their AWS issues as well (their console is always
green), but at least they provide a very detailed postmortem, and issue
credits proactively.

Have not seen that from MS yet, and it's been two days.

~~~
jspaur
Any info on Sql Azure being down a whole day? We didn't see anything. This
issue on the other hand, a real bummer.

~~~
tarekayna
Yeah I didn't see anything wrong with my services 2 days ago, which rely
heavily on Sql Azure.

Right now everything is down.

~~~
photorized
And we are seeng sporadic SQL connection errors today (between Azure VMs to
Azure SQL), so whatever the issue was - doesn't appear to be fully resolved.

------
sriramk
I was involved with obtaining a previous version of the same certificate when
I was at Windows Azure. There were several safeguards in place to stop exactly
this scenario from happening. I'm wondering how they broke down.

~~~
magic_haze
Dev churn, probably. Could you expand a bit on the sort of safeguards that
were in place? A couple of comments here mentioned trello, build scripts or
just plain post-its, it'd be interesting to know how bigcorps do it...

~~~
sriramk
The most basic one was that really large sets of people that would get emailed
a long time before expiry by the central crypto/cert management system.
Microsoft has a very streamlined system internally for obtaining/managing
cents since they do so much of it.

~~~
YokoZar
Perhaps that's precisely the problem -- a large set of people were notified,
rather than anyone in particular

~~~
mpyne
Admiral Hyman Rickover (the "Father of the nuclear Navy") had as one of his
basic principles that "if you can't point your finger at the person
responsible for something, then _no one_ is responsible for it".

------
yajoe
I thought this was a repost of the same story from _last_ February related to
expired certs:

<http://www.wired.com/wiredenterprise/2012/02/azure_outage/>

I'm pretty tolerant of mistakes (I make them all the time), but I have little
patience for repeating mistakes.

~~~
smarx
In this case, an SSL certificate has expired. The leap year issue last year
was a bug about date addition: 2/29/2012 + 1 year was calculated as 2/29/2013,
which is a date that doesn't exist.

The only similarity is that both involved certificates. (The invalid date was
used internally as an expiration date for a certificate.)

~~~
vyrotek
Glad to see you around these parts Smarx :) You maybe not remember me but I
came to Redmond for an Azure bootcamp you hosted for a few companies a few
years ago. (I'm sure you did lots of those) I was with IActionable. We've had
a few 'encounters' on the MSDN forums too. Azure sure has changed since those
days... Definitely for the better!

I see you've been busy since you left MS. Somehow I missed all the news about
Site44!

------
gabbo
If this outage is solely caused by an expired SSL certificate then this is
really bad. It makes Azure look like amateur hour (even though it's definitely
not). This may be acceptable for a smaller operation, but not Microsoft's
cloud platform. There should be processes (manual and automated) to deal with
this.

On the bright side, after this you _know_ they're not going to make the same
mistake twice.

~~~
niggler
"It makes Azure look like amateur hour (even though it's definitely not)"

Why do you think Azure is not?

~~~
gdc
Do you think that it is?

~~~
niggler
You can't deny that the current issue is certainly not something you would
expect of a serious offering

~~~
gdc
Every major cloud offering on the market has had outages that were traced down
to silly issues. I'm not sure if your dislike is targeted at just Azure, or
the cloud in general.

~~~
niggler
I'm always apprehensive of outsourcing critical business infrastructure
without a failover plan, and I find the excuse "amazon is down" or "azure is
down" unsatisfying (especially when its a service customers pay for)

~~~
suresk
Having a "hot spare" cloud provider is going to be so complex and expensive
that it probably would kill the idea of using Azure, Amazon, etc for most
companies.

Unless you run your own DC, you are still outsourcing part of your
infrastructure to a colo facility, which will not be immune to power and
connectivity issues, and of course, issues with your setup.

Saying "<cloud provider> is down" certainly isn't very comforting, but if the
aggregate downtime is less than if you were running your own setup, I'm not
sure that being powerless in the event of downtime is reason enough to not use
them.

------
mikeocool
Added this tool to my suite of monitoring tools last time an outage caused by
an expired cert popped up on hacker news:
<http://prefetch.net/articles/checkcertificate.html>

Runs on a daily cron job and emails if any of the certs it's monitoring are
within 30 days of expiration.

~~~
awa
is there something similar for windows?

------
moonboots
I've been using the following script in my build process to avoid certificate
expiration surprises:

    
    
      currentDateEpoch=$(date +%s)
      expirationDate=$(openssl x509 -in $my_cert -enddate -noout | sed 's/notAfter=//' | date -f -)
      expirationDateEpoch=$(date -d "$expirationDate" +%s)
      diff=$((expirationDateEpoch - currentDateEpoch))
      
      oneMonthInSeconds=$((30 * 24 * 3600))
      oneWeekInSeconds=$((7 * 24 * 3600))
      
      if [ "$diff" -lt $oneWeekInSeconds ]; then
        printf "Certificate $my_cert needs to be renewed! Expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2
        exit 3
      fi
      
      if [ "$diff" -lt $oneMonthInSeconds ]; then
        printf "Certificate $my_cert expires in less than a month! Expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2
      else
        printf "Note: certificate $my_cert expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2
      fi

------
rmason
Letting domains and SSL certs expire has been a consistent problem for
Microsoft. A few years back they lost a key domain that took down Hotmail:
[http://news.cnet.com/Good-Samaritan-squashes-Hotmail-
lapse/2...](http://news.cnet.com/Good-Samaritan-squashes-Hotmail-
lapse/2100-1023_3-234907.html)

Couldn't find it but I know they've had other embarrassing domain or certs
expire over the years. You would think they'd fix it before they lose
microsoft.com!

------
vyrotek
Don't worry guys. Scott is on it! :)

<https://twitter.com/shanselman/status/305064837434724353>

------
smhinsey
If your connection strings are set in .cscfg config files, remember that they
are editable through the Management Portal and you can swap them to HTTP.

Edit: Maybe not such a great idea. It looks like this relies on Storage under
the hood as well. Now my instances are recycling.

Later: It appears doing so actually corrupted the instance and required a
redeploy, which is of course not currently possible. Awesome.

------
LogicX
A few services to check your SSL certs:

    
    
      http://checkmyssl.com/
      https://sslcheck.globalsign.com/en_US
    

Email notifications:

    
    
      https://www.sslshopper.com/ssl-checker
      http://www.serverexec.com/ (does domain expiration too)
    

Script to run yourself, does email notification:

    
    
      http://www.prefetch.net/articles/checkcertificate.html
    

Any others to recommend?

------
runesoerensen
I'd like to invite you to try AppHarbor. We provide Windows application
hosting on top of AWS, and strive to deliver a higher quality of service than
Azure. We're standing by if you need help to migrate immediately.

We do a number of things to keep our platform and your apps more available.
For instance we set up reminders before our certificates expire...

~~~
fmax30
I have used Appharbor in my class to host a Web Pages project (Basically the
prof made the whole class host it). My experience is , that while the customer
support is good. The Service I used ( the free one) was a bit slow. My site
took quite sometime to get loaded ( around 5-10 secs where my page weight was
around 300-400 KB ) , the same page took less than a second on my local
service. But all in all I really liked AppHarbor :)

~~~
runesoerensen
The free worker is allocated the same amount of resources as paid ones, so
that shouldn't be the problem. It definitely sounds like there must have been
an issue _somewhere_ if you experienced that kind of performance difference.

I don't think it was related to the page weight (unless you're on a very slow
connection), nor the platform for that matter. I hope we were able to figure
out a solution or give you advice on how to resolve it :-)

------
breck
Why do SSL certificates expire? Seems like a suboptimal design decision.

~~~
hwatson
Take a look at me.com, for example. Before Apple bought it, it was owned by
SnappVille.com. If SSL certificates didn't expire, SnappVille could have
continued using their certificates for me.com.

~~~
paxswill
That still doesn't fully explain why they expire, as CRLs and OCSP allow
certificates to be revoked. I can't quite explain why having an expiration
date is safer, I just feel it's a good practice, to protect against possible
key compromise.

~~~
tptacek
SSL certificate revocation is extremely fragile.

<http://www.imperialviolet.org/2012/02/05/crlsets.html>

------
rwg
At ex-work, SSL certificates expiring was probably #3 in the list of reasons
for services being unavailable. It's a really easy thing to overlook in
service monitoring.

~~~
Ironlink
Note to self: Compulsively add every certificate expiry as an event on my
calendar.

~~~
koenigdavidmj
Better yet: set up a Nagios (or your site's equivalent) watcher for the
expiration.

Example: [http://exchange.nagios.org/directory/Plugins/Network-
Protoco...](http://exchange.nagios.org/directory/Plugins/Network-
Protocols/HTTP/check_ssl_certificate/details)

------
Gargol
Can't logon to management dashboard. Staging instances went crazy. Live
environment looks OK so far. Should I be worried and stay up all night
monitoring this stuff ?

------
taylorbuley
The Microsoft employee who managed to let the hotmail.co.uk domain expire is
suddenly looking a little less irresponsible.

[http://www.theregister.co.uk/2003/11/06/microsoft_forgets_to...](http://www.theregister.co.uk/2003/11/06/microsoft_forgets_to_renew_hotmail/)

------
727374
It's the annual February Azure meltdown. Last year they had a leap year bug
that decimated Compute in multiple regions.

[http://www.wired.com/wiredenterprise/2012/03/azure-leap-
year...](http://www.wired.com/wiredenterprise/2012/03/azure-leap-year-bug/)

------
dangrossman
For those of us using Trello to manage projects, it's worth adding a list of
reminders for your certificate renewals. Set a due date on each card and
Trello will highlight when they're coming up soon.

------
InvisibleCities
Looks like somebody forgot to set up a share-point reminder.

------
niggler
Does anyone use Azure for a production service? I'm interested in hearing
people's thoughts ...

~~~
photorized
We do, for some of our infrastructure at www.itrendcorporation.com. We use
Azure Websites, Azure Storage, Azure SQL. Websites has been great. SQL East
had a major (I mean, complete 100% outage for more than 5 hours) outage, so we
are now making some changes internally.

I like Azure, but I don't think I am ready to use it in production for
anything critical without another standby infrastructure in place.

~~~
niggler
"I like Azure, but I don't think I am ready to use it in production for
anything critical without another standby infrastructure in place."

Seems like a really good general rule for any service :)

------
pasbesoin
Someone over there might want to fire up Project and put in some hard
deadlines.

------
Scorpion
It _might_ have taken out Xbox Live cloud storage as well.

~~~
vyrotek
Your comment reminded me of an article that mentioned that Halo 4 would be
'backed' by Azure [1]. I wonder if that's down as well.

[1] <http://mashable.com/2012/10/31/halo-4-windows-azure/>

------
bogrollben
Don't worry - they found the guy that did it:
<http://memegenerator.net/instance/35258710>

