Hacker News new | past | comments | ask | show | jobs | submit login
Major Incident at Fastly (fastly.com)
126 points by afrcnc on Jan 12, 2021 | hide | past | favorite | 61 comments



Is it just me or is the text practically non readable with very light font on a white background?


I thought maybe that was the major incident. All their design people have left and messed up their CSS on the way out.


I haven't seen so much barely visible grey text since the last Hacker News thread discussing current affairs.


“It’s the wild colour scheme that freaks me,” said Zaphod whose love affair with this ship had lasted almost three minutes into the flight, “Every time you try to operate on of these weird black controls that are labelled in black on a black background, a little black light lights up black to let you know you’ve done it. What is this? Some kind of galactic hyperhearse?”


I just ran the Chrome Lighthouse accessibility report again. They increased the contrast and went from 74 to 90. Well done!


I am the head of Product Design at Fastly. We updated the design of the page to include text with the proper color contrast to improve accessibility. The colors are all now WCAG AA compliant. Thank you for the feedback.


I'm a little embarrassed by my earlier comment.

Thank you for taking this in stride and for updating your site.


It's not just you, there are 73 contrast errors (WCAG): https://wave.webaim.org/report#/https://status.fastly.com/in...


As compared with 139 contrast errors on this HN page as of 2021-01-12 19:04 GMT: https://wave.webaim.org/report#/https://news.ycombinator.com...


The 139 contrast errors are all the headers "jshprentz" "12 minutes ago" not having enough contrast.


That "12 minutes ago" has insufficient contrast is still a valid critique from an accessibility standpoint, even if the design choice seems reasonable at first. There will be sight-impaired people who struggle to consume those "age" snippets, as opposed to perceiving them as legible but de-emphasized.


I don't think the point was that they didn't matter, but structurally, it is 2 errors repeated 70 times as opposed to 140 distinct errors- might even just be one if they are lumped together as "post label" rather than "name" and "time".


HN has always had horrible UX and no desire to fix it. This tracks and does not excuse the original site at all.


> HN has always had horrible UX and no desire to fix it.

I thought the other day that's their way of discouraging new users, thereby slowing down the local Eternal September.

https://en.m.wikipedia.org/wiki/Eternal_September


I just don't understand how UI/UX designers, managers, whoever it was, managed to convince the world that grey on white makes any sense. Pure trend-following


When asked why going with a light on light colored web page rendering readability to almost nil, the Designer spoke at length eloquently stating obscure mission statements, having the site born at the intersection of arts, activism and the global community at large. Finally they lastly mentioned "because its the right thing to do". /s


Designers just want their work to be unique and eye-catching. They don’t care about readability. They care about user remembering that page. If most pages have good contrast, designers will want to use light contrast to stand different. If light contrast would become the norm, they will reinvent hard contrast. It all goes in cycle.


I remember that page as something I can't read and will tell everyone I know about how horrible it is. Is that really what "designers" want?


Is this a cry for help? :)


There's an argument that high contrast ratios are too high on the eyes, which is why "not quite black" is used on "not quite white" backgrounds.

How people ran away on this to 50% brightness text on 70% brightness backgrounds is another question.


If you have something else other than text that you want to be the focus of the page like a photo, reducing the contrast on the text allows the photos "pop" more if it using the range of light and dark.

My personal color schemes are all dark background and light grey text. So even in my world of IDEs, consoles, gvim, sublime and emacs where text is king, I prefer having less than 100% contrast between background/text. I am not using any kind of custom color scheme so at least 1 other person prefers this as well!


I come the world of video, and old enough to have spent time with graphics for SD. Specifically, NTSC compliant video. Since it was analog, there were severe consequences of color choices. Red was just a solid NO. On an 8-bit 0-255 scale, white was limited to 235, and black was limited to 16. The reasons for this are beyond the scope of this chat, but that limited range has still stuck with me. I never use #000 or #fff and opt for #111 to #eee for my limits. Seems like a decent enough compromise in web graphics.


Isn't the idea here that they don't want people snooping around and hanging out on the part of their website that describes all the problems going wrong with their service?


How does any site in the US not run afoul of Americans with Disabilities Act (ADA)? Surely color choices like this are an issue that someone can raise; see WCAG 2.0 AA. which talks about how there must be a color contrast ratio of at least 4.5:1 between all text and background.

Even W3C reminds people that colors are important.


The font colour they choose was named “really don’t want people to know but still have to”


I'm 25 and had to use reader mode for this site, never seen a site this bad before


Very true - I did Inspect Element and changed #808080 to #202020 without even thinking. I guess I am by now used to these low contrast designs :-)


You should use Firefox and toggle Reader Mode. No CSS changes necessary.


For some reason, Reader Mode isn't available on all sites. It wasn't available on this one, either.


Reader mode requires a certain layout of text and paragraphs.

You can add

  `about:reader?url=`
before the URL to force enable it.


Oh nice, thanks for that!


Safari's reader mode works on this site.


Unreadable gray text on white background is the latest in Web Designer fashion.


Luckily I think it’s on the way out, thank goodness. I agree that was an awful trend.


Dark mode browser extension is a wonderful thing.


I was wondering what everyone was taking about. I totally forgot I installed the dark reader extension on my phone... Thanks for reminding me!


They definitely need to work on accessibility.


HTML is accessible by default, so nobody really has to work on accessiblity; they just have to refrain from working on breaking it. Unfortunately that seems too difficult for many web designers.


It's a status page, so do they need to fix the font color or the actual problem first?


They obviously need to fix the actual problem first, but they only just found out about the new "actual problem". Surely somebody somewhere has actually looked at this status page before now, so this should have been a known issue already.


I guess the litmus test is that it doesn't become inaccessible when infrastructure goes down.


It gives it that really cool look of weatherworn text chiseled into stone.

What you should do is spread some parchment over it and get some charcoal and rub the charcoal on the parchment over the chiseled letters... oh. yeah. sorry...



If you turn up the brightness to max when your eyes hurt the text is legible.


Yep, it's next to unreadable. Who thought this was a good idea?


Fastly: if you could add context to the incident as you track it, it would help customers and users feel better.

Also, linking a public postmortem to the incident would be great. It doesn't inspire confidence as a customer if we can't see that you are working on identifying the cause and are preventing it from recurring. (We don't need to see the full RCA or other process, just an overview of the problem, the cause, the fix, and the follow-up. This is a great way to iterate & improve on your RCA process)


Affected live streaming on both Mux.com and Vimeo.com (the two streaming companies I use for my training classes, hahaha - so much for redundancy with paid companies. Had to switch to YouTube.)


With Mux you can enable "Redundant Streams" (multiple CDN's used in HLS manifest) which I think solves the issue here. We are also a big Mux customer but admittedly haven't made the full switch to redundant streams, although our testing was very promising. Since their ingest is all GCP I don't think that was affected by this, and it would have just been delivery.

https://www.mux.com/blog/survive-cdn-failures-with-redundant...


there is a good chance that redundant streams would have helped here assuming that the player could load the manifest to fail over, but that manifest sadly hits Fastly. the problem with redundancy is that it really only works if you are redundant all the way up and down the serving stack. in our standard serving path (even without redundant streams) we automatically select the optimal CDN for chunk serving which does take into account availability, but during this incident we measured Fastly as about 5% less available than normal which wasn't enough to trigger a full automatic drain off of them. turns out, a 5% failure rate when you are serving 50 chunks to enable a video playback means 1 of them will likely fail and you get a playback error or in the best case, rebuffering.

sadly, Mux's manifest servers have not been redundant because the edge logic is a pile of Fastly's VCL. as mmcclure says, we're working on adding another CDN that can run our manifest serving logic so that we can be fully redundant.

I am really looking forward to a standard and portable edge platform that makes redundancy at this application layer easier. wasm here we come?


Thanks for the kind words, Andrew!

Redundant streams would definitely help mitigate the problem, but we're also days away from shipping a feature that would have allowed us to seamlessly switch over to another partner in a situation like this. C'est la vie.


Ugh, on pins and needles over here. We have a handful of Vimeo livestreams going on right now.


Ah, that explains the minor hiccup I just saw on my Github Pages. I thought it was odd that Github status was all green.


Is this major (per title)? Anyone experiencing it? I’ve got two sites in different affected regions that do not seem to be affected at all.


PyPi is hosted through Fastly, and pip installs were all failing in our automated builds this morning, so I would call that pretty major.


Things like this can sink entire companies.

I'm at a livestreaming service / production company that uses Vimeo as our primary.

An outage like this when (as an example) "Procter & Gamble" is giving their yearly sales conference via a livestream due to COVID (instead of a large corporate event at a convention center) can cause a company to loose them as a client.


It's intermittent but I'm seeing it on sites that are using fastly (EG: terraform runs).


Noticed a few 503 and 502s around 10:40AM EST from different POPs.


“A fix has been implemented”: for those just now seeing this, you’ve probably read this post unnecessarily :)


I noticed that pythonhosted.org was shaky, which meant that `pip install` failed over and over again on different packages. CircleCI and Github were also affected.


#hugops


And yet, their stock is up +5.32% today.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: