Hacker News new | past | comments | ask | show | jobs | submit login
DuckDuckGo was down (duckduckgo.com)
179 points by AlphaWeaver 28 days ago | hide | past | favorite | 98 comments



Hey folks - we're back in business. Clearly Dax the Duck had a case of the Mondays!


The Mondays? No, man. I believe you'd get your ass kicked sayin' something like that, man.


One time I was at a festival where it wouldn't stop raining. It had been a shit show. Come Monday, all the cars were stuck in mud. Then there were tornado warnings. Then they let down a damn up river and the whole area started to flood. Whilst holding down a shade structure, waiting for updates on the tornado path, and standing in slowly rising water, someone made the joke:

"Mondays"


Ha. Reminds me of Norm Macdonald’s “Real Jerk” joke.

https://youtu.be/yrbZxtuUdsQ



... that's better than a lot of postmortems I've had on the job. Like the junior employee who insisted we not write one at all when we lost the ability to deploy to a test server, because "that wasn't our fault" and "it probably won't happen again".


I'd love to read a post-mortem.


Sometimes ducks just have a bad day.


Some days you’re the jet engine; some days you’re the duck.


You just did.


Me too


Mundays* ;-)

Corporate accounts payable, Nina speaking. Just a moment!

...for 10 hours: https://youtu.be/oFwkL1WgD_0


And it's back, after 10 minutes. Nice job to the response team, that was fast !


All I know is that I would not want to be the one who had last committed to production.


Good teams don't blame individuals. You can praise individuals, but you take blame as a team.

Hopefully at least two people reviewed the commit before it was put into production. Someone set up the post-launch smoke-tests/api-test-suits. Someone built the CI/CD-system. Etc etc.

My point is, it is rarely a single individuals fault when something at this scale goes down.


>Good teams don't blame individuals. You can praise individuals, but you take blame as a team.

I could not agree more with this statement.


I also think a lot of praise should generally be a team thing too (but I also love this: "You can praise individuals, but you take blame as a team.")

When discussing success to an outside group (giving a presentation or something) or higher up (especially here!) you always use "we". Even if it is a section that only you worked on or you did most of the work. You can often pick out good managers by just seeing if they talk like this, if their focus is around what they individually did or what their team accomplished.

My point is, it is rarely a single individual's success when things work.


I also really like it when managers mention significant individuals (as suggested by the phrasing that started this thread), as long as everyone on the team is getting mention-worthy projects. Both group and individual recognition, as long as it's fairly used.


Yes, exactly. I don't want to downplay the importance of praising individuals, but at the same time I don't want it to shadow praising teams.

There's a balance and I do think we tend to focus on the individual as a society. We can only climb mountains by standing on the shoulders of giants. One person gets to the peak and we should praise them for that, but we shouldn't ignore their foundation either (which I think currently happens). E.G. Kepler was an essential part of science and needs to be recognized, but if it wasn't for the work that Tycho Brahe did, Kepler's achievements couldn't exist. The "single person" mentality ignores the importance of the foundational work that needed to be done and frequently causes many to feel that they are not achieving simply because they are working in these roles, which are essential.


I agree with everything you say.


You're absolutely right both about how teams should function and about the reality of how failures happen.

But the parent is right that it still sucks to be the persons whose work was the proximate cause of an outage, even if nobody is going to blame/punish you for it. It doesn't make for a day where you feel good when you get home from work.


>Good teams don't blame individuals. You can praise individuals, but you take blame as a team.

This is a form of illusion. A cover up for what is essentially in reality a mistake made not only by the team but by the individual as well. To mask part of the truth and lay only the blame on the team is an effective form of not hurting someones feelings but also an effective form of avoiding the full reality.

Do people not see how illogical it is to blame the team but only praise individuals? The precedence being set here is that: People succeed as individuals but fail as teams.

The harder ideal to strive for is that both the team and the individual take the blame, but it's harder because people are prideful stubborn and easily hurt.

If you make a mistake step up and admit your mistake. Don't hide in the corner and expect the team to change all their processes to account for your mistake. Yes the team should do this, but yes you should stand up as an individual and do things yourself as well.

There are times when you must blame an individual. Let's say a team member repeatedly pushes bad and buggy code to production. Is it the teams fault or the individuals fault when such actions are repeated? Does it lay on the team or the team member to make sure buggy/bad code doesn't go into production?

The line is blurry here and I feel it is ultimately the wrong stance to say absolutely good teams don't blame individuals. Good teams and good people take responsibility so there is no need to dish out blame.

What happens when the person who spilled coffee all over the on premise servers doesn't take the blame? The team sees this and decides to blame nobody. Is this a good team? No.

The good team member volunteers and states publicly that the fault is his own. The good team agrees with this stance and also says that the fault is with the team as well and both the individual and the team take steps so that this mistake cannot happen again.


The reason the blame is taken as a team is so that methods are put in place to prevent those problems in the future. If it were simply a blame game those issues would not be resolved. Humans do not operate on pure reason, and if you decide to do the blame game they will rarely resolve the underlying issue that caused the problem in the first place. The blame game is a stupid game, and as we know playing stupid games wins stupid prizes.

It isn't about eliminating recurring problems, it's having the highest chance of reduction. If someone continually performs poorly then thats management's responsibility to replace them.


It's also that we (the team) failed to put steps in place to prevent the problem. The person who pushed the button that took down prod is only the last person who made a mistake. Plenty of mistakes were made before, otherwise their mistake would've been a non-issue.

If your system can't tolerate one person's mistake, it's not robust.

None of this means you can't praise a whole team (you should! The original statement just said you CAN praise individuals), or discipline team members who regularly show poor judgement.


>or discipline team members who regularly show poor judgement.

Isn't this contradictory with the original premise? If you discipline an individual you are blaming the individual.

You're saying you can praise individuals and you can discipline individuals and you must also follow this: "Good teams don't blame individuals. You can praise individuals, but you take blame as a team."

Do you not see the contradiction? I'm literally getting voted down because of slightly miffed feelings.

Despite all of this the logical correctness of my statement stands and you have inadvertently agreed with every aspect of my statement.


People are down-voting because they disagree. While it's not something I agree with, it's officially sanctioned on HN.

If I'm continually pushing buggy code, that's something that I'm going to be personally held accountable for. Blamed if you will. That's because I'm the only one involved. My team isn't constantly writing buggy code, I am.

If I push buggy code into production and take down the site then the team is to blame. Why is our process setup such that I can do so. EVERYONE, even the best programmers will make mistakes, and if the process allows those mistakes through to production then the TEAM has failed to build a suitably robust process, not the person who happened to write a bug, because every human being will be that person one day.

Your taking a once sentence philosophy about how to deal with technical incidents and treating it like we're advocating it as a universal and moral absolute. No one is claiming that if I throw a rock through someone's windshield my whole team should be arrested. What we are saying is that is I make a slip up and write a bug that takes down DDG, I shouldn't be held accountable for taking down DDG. If there's a larger pattern of poor decision making I'll be help accountable for THAT, but it's completely 100% separate from the outcome of knocking production offline.


>People are down-voting because they disagree. While it's not something I agree with, it's officially sanctioned on HN.

I think a lot of vote downs are emotional and reactionary rather than genuine disagreement. I think that's the majority. I can't fault people for this as it's part of human nature but I don't think this type of voting is ever officially sanctioned by HN.

>If I push buggy code into production and take down the site then the team is to blame.

And you are not? I propose that you and the team should take the blame. People make mistakes but all the time and they shouldn't be singled out for it, but I don't think it's good etiquette for the team member who made the mistake not to own up to the mistake either.

>Your taking a once sentence philosophy about how to deal with technical incidents and treating it like we're advocating it as a universal and moral absolute

It's a one sentence philosophy that doesn't hold any logic in my opinion. I'm not saying anything is morally absolute here, I'm trying to say that the quotation doesn't even hold up in even the most basic situation.

It's elegantly written, no doubt, but I think that elegance is deceptive and that's why I wanted to say something. The real world in my opinion is nowhere even remotely close to that philosophy. I would argue that philosophy is just a surface level special case. In the real world people get blamed and fired all the time.


Blame is for incidents. Discipline is for patterns.

Also, just a bit of friendly Internet advice, this isn't the best way to interface with people:

> Despite all of this the logical correctness of my statement stands and you have inadvertently agreed with every aspect of my statement.


>Also, just a bit of friendly Internet advice, this isn't the best way to interface with people:

Well I'm getting massive vote downs and the original poster literally called what I'm stating as initializing stupid blame games to win the stupid prize. Kind of rude and insulting. I think it's reasonable to say that my reaction is reasonable given the attack.

It's not exactly the real world on HN either as people are harsher than normal, ruder and less receptive of differing opinions.

I would say that Giving people a piece of "friendly advice on interfacing with people" won't exactly win you any friends either. If you teach people things using that methodology you will get a retaliatory response whether it's on the internet or the real world as the tone is a bit domineering and elitist.

>Blame is for incidents. Discipline is for patterns.

This doesn't make sense to me. I can't blame people for patterns? I can't discipline people for incidents?

The sort of short and terse elegant expressiveness of the phrase ultimately hides a statement that makes no logical sense. Typically someone is blamed first before he is disciplined. Both go hand in hand.

Either way, you're probably just trying to say that let the team get blamed but if you see patterns then blame the individual. Not as elegant but no linguistic tricks that decorate ultimately illogical statements to appear as philosophies of life.


> If it were simply a blame game those issues would not be resolved. Humans do not operate on pure reason, and if you decide to do the blame game they will rarely resolve the underlying issue that caused the problem in the first place. The blame game is a stupid game, and as we know playing stupid games wins stupid prizes.

I never said play the blame game. I never said the team must not take the blame. Please don't put words into my mouth.

I literally said it's the teams fault and the individuals fault. Both parties must take responsibility.

Take it this way, if a single team member repeatedly makes mistakes then is it the teams job to put methods in place just for that team member? Or is it the teams job to help that team member as an individual?

The blame game is a stupid game so don't play a blame game. Take responsibility both as a team AND as an individual. The team taking the blame exclusively is AVOIDING individual responsibility. It is not a sign of a healthy team.

I am essentially saying the solution is not so clear cut. The team can't always take the blame just to "avoid a blame game." The reality of the universe is that problems aren't always team level problems, that problems at the individual level exist as well.

To exclusively avoid addressing problems at the individual level is a stupid and delusional endeavor and you also win the stupid prize for ignoring reality.

>If someone continually performs poorly then thats management's responsibility to replace them.

This is called blaming an individual then firing him. It entirely contradicts your main argument. I didn't even go there yet, I advocated blaming the individual than helping him improve as a team. You immediately cut his head off and let management do the dirty work.


Unlikely it is their logical app code , at this scale you likely will do canary releases or similar , failing for all users should be super rare

It perhaps is related to devops or the search index cache servers is broken


Or a bad config. 90% of the time it's a bad config.


No, because,

* You probably got a review on your change before it was submitted, so you weren't the only one to think it was an ok change to make

* If you "revert first, ask questions later", you might also be the first one to fix it

* Hopefully the team recognizes that there are higher-order causes that caused your submit to break things than your own personal skill


Or the one in charge of testing. You'd think this is a day one unit test.


Right, which probably means the problem was quite subtle and was not detected by unit tests.


Image and video search works. Regular text search, maps and news are down. Did they forget to bay their Bing bill?

EDIT: scratch that. DDG is back in business again. I would love to read a post-mortem.


If just Bing broke, then I wouldn't expect to see zero results unless they didn't handle that edge case gracefully, since Bing isn't their only source.


Do they still use Bing ? I thought DDG had in-house search db.


They still use Bing, and others.

https://help.duckduckgo.com/results/sources/

"These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google)."


The Duck Duck Bot crawls the web in some fashion but it is my understanding most of what's in their search result comes from their Bing indices. Correct me if I'm wrong.


Afaik, it's still mostly from Bing.


Came here to see if I was crazy. Should be an interesting post-mortem, I wonder if someone dropped the production database...


Autocomplete in the search bar still works, so there’s some data out there.


Most of my searching is through the firefox address bar so I didn't notice, I just kept coming up on empty search pages. It did not compute at first, I must have done 3-4 searches before I realized what was happening.


I would have thought that the type of person compelled to use DDG, would disable autocomplete.


It really depends. On the continuum of "I don't care about privacy" to "borderline paranoid", there's certainly a middle ground of "I don't trust Google but do trust DDG to not record every keystroke I send and add it to my profile forever" that would be happy to enable autocomplete if it was going to a search engine with a meaningful privacy stance.


If I remember correctly their autocomplete might use probabilistic data structures to preserve some privacy, so you download a set of random completions rather than just a single completion. But I don't know where I read that or if it was implemented.


I don’t use DDG. I just followed this HN link, then tried typing some other things in the search bar.


Probably an operational issue, like an expired internal certificate I'd imagine. Very curious to see the post morten and root cause analysis.


Image search still works for me. Odd.

Edit: strike that, only for some entries. Must be some cache?


It still works if you're hungry: https://duckduckgo.com/?q=food&ia=recipes


Haha, I was also searching "food" to see no results and just started seeing recipe results returned.


Guess I'm not hungry, because nothing shows up for me.


Changing the time range to "Past Month" yields results


Can confirm.



Can confirm, from Switzerland.

I was quite confused why my query for a well-known brand name was returning nothing. Bang queries still work.


Even the conversion magic doesn't work (eg "1cm in inches") which one would assume is a separate system


As well as bangs, which give a different error saying to email ops@duckduckgo.com.


Oops, not fun. I think they must have been having problems for a while, I recently switched to try qwant because DDG was frequently slow to respond, but it was still working...


I know I was freaking out and had to use Google directly. :)


It's back up now. Should know soon why it was down


Seems like other type of searches (images) still work.


It is just the "All" and maps tabs that are not returning anything as of now. News/images/videos are still working.


Possible issues:

- It could be an issue with DNS servers - Certificates that got expired - Latest push to production that got wrong - DDoS attack


Wow, they must be having a pretty bad morning


Recipe results are still coming through


What a tough time for the devs. Troubleshooting at such a scale must be a nightmare


Came here to say this. Still broken for me. "NPR" was last query.


It's working again! But wow, that was quite an interesting 9 minutes!


I had this issue one time, and it got fixed when I reloaded the page.


Interestingly it looks like image search is still working though


Works again. I wonder if the fix was just rebooting a server


Normally that does the trick, but it doesn't tell what went wrong so it could repeat again.


wow, they are third in search engine usage, not sure whats going on!

[EDIT] removed the metrics due to downvotes! check below for traffic information and how much is lost..


where did you get this number from?


https://duckduckgo.com/traffic June 2020 Daily Average 62,489,472


Has this ever happened with another large search engine?


Yes. : https://www.cnet.com/news/google-goes-down-for-5-minutes-int...

"Google suffered an outage late on Friday night, though you might not have noticed -- it only went down for five minutes. During that brief window, Internet traffic around the world dropped by a massive 40 per cent, according to Web analytics firm GoSquared, Sky News Online reports. "

Bing also: https://www.pcworld.com/article/183770/bings_30_minute_outag...

"Microsoft's Bing search engine was offline for about 30 minutes on Thursday evening."

Qwant also: https://blog.qwant.com/qwants-unprecedented-service-failure-...

"Qwant’s infrastructure broke down around 12.00 UTC, and the service could only be restablished hours later, around midnight"


also a case in 2009 when google flagged all search results as malware for 55 minutes: https://techcrunch.com/2009/01/31/google-flags-whole-interne...


nope, guess we are witnessing it


How do you know it doesn't happen every day at Bing?


Fixed now it seems


I wonder if they have a status page?


I tried to find one. The closest I came is their Traffic dashboard [1], which is updated daily. I also checked their Twitter account [2] but no mention of problems yet.

[1]: https://duckduckgo.com/traffic [2]: https://twitter.com/duckduckgo


I looked at status.duck.com or status.ddg.com and both timed out so if they do, it's not there.


Ironically, the best way to find it would be to use a search engine.


Looks like it's back up now!


fixed


now it's good. It's searching.


They're back. So that lasted about 4 minutes..


still broken, and it was broken for me maybe 10 min ago, so the window is getting pretty long for something as critical as the competitor to google search..

Edit, and now it's back up for me, so ~15-20min of downtime from my perspective


I lied. A search for "whatsapp" works, but not for "Hacker News"


Still broken for me.


Still broken for me.


same, still broken.


Still broken for me


its still down




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: