Hacker News new | past | comments | ask | show | jobs | submit login
Cisco Fixes RV320/RV325 Vulnerability by Banning “curl” in User-Agent (twitter.com/redteampt)
734 points by pjf on March 27, 2019 | hide | past | favorite | 310 comments

Of all the security post-mortems I’ve ever wanted to read, it’s sad I’ll probably never get to read this one and its tale of how a team of well-paid comfortable engineers got together and decided this patch was a good idea.

Having been involved in meetings where "stop ship" was the phrase of the day, I'd bet money that the following at least vaguely resembles a real conversation:

Engineer Alice: We should really fix this properly.

Manager: How sure are you that the proper fix won't break something else for $BIG_CUSTOMERS who are responsible for $OBSCENE percent of this product line's revenue?

Engineer Bob: Uh, ten percent on a good day? Do you remember the last time we applied a "simple" hotfix to this function? I almost had to sleep under my desk.

Manager: And we can just prevent it with an extra regex rule in the front end code?

Engineer Alice: ... Yes. irrepressibly existential sigh

Manager: How much QA effort to call this good?

QA Engineer: We can run through a cut-down version of the acceptance test suite in four or five hours.

Manager: And for the proper fix?

QA Engineer: Oh, hell, at least a week of stress testing to really be sure.

Manager: Add the regex rule.

On HN its always big bad management who is the cause of every security problem or shoddy piece of engineering. If only that pesky management would screw off then we could do things "properly".

You'd be suprieed at how many incompetent engineers there are out there. If "engineer" Alice in your story was actually competent they would never agree to implement the proposed "fix". Its not a fix. To pass it off as one would be malpractice for a real Engineer.

"Engineer" conveys image of middle class white-collar job with relatively high status, good education and responsibilities. That word now used for everyone doing programming related jobs inside office space for no good reason.

I think the word "tehnician" should be used to describe most grey-collar ICT jobs, including most programmers. Their responsibility and scope of their work is limited. Many programming jobs are closer to blue collar factory level mechanic than engineer.



Yup. The reality of this industry is that most programmers are closer to the handymen you call to fix your faucet or lay floor tiles than to actual engineers. And we all have first-hand or second-hand horror stories related to handymen.

I've come to the conclusion recently that we're all just barbers, bloodletting and burring holes. Until we have proper certifiable and reproducible competency, we'll never become surgeons.

Don't you need a license to be a barber?

In modern times yes, here's some interesting reading on the guilds licensing originated from - https://en.wikipedia.org/wiki/Worshipful_Company_of_Barbers

And yet the US has consistently been in the vanguard of software development worldwide. Do we need surgeons? Do you need a four year postgraduate degree and three more years of apprenticeship before slinging together a web app? Certainly the industry has its problems, and some applications demand more rigor, but I don’t see this as much of a general solution.

The market didn't demand a high survival rate or efficacy for barbering either, but do you think that the (conscious or unconscious) that software engineers are some sort of magical wizards will last forever?

There may be explanations beyond wizardry - e.g. maybe barbers were historically often self-employed, small business owners who realized they stood to gain from creating barriers to entry such as licensing requirements.

Thankfully there are countries where Engineer is still a proper word, not something that you are allowed to call yourself after a 6 month bootcamp.

Yup, in the UK it bugs me when I keep meeting people who introduce themselves as an 'engineer'. When I ask them if they do mechanical or civil engineering, then I usually get to say "ah, so you're a programmer, just like me".

One thing I did sometimes like about being in Austria is the obsession with academic titles they have there... if somebody is a qualified engineer, they invariably use "Ing" as a title in place of "Mr" or "Ms"; people with two doctorates (not uncommon) really do call themselves Dr Dr So-and-so.

Austrian here. "Ing" (Ingenieur, so basically Engineer) is a protected title (not an academic one, though) but to be honest, it does not signal status or qualification to me. The actual academic titles are worth more.

Still doesn’t really mean anything. Due to a loophole in major assignment I can legally call myself Ing., but what I’ve really studied is more or less computer science.

What is qualified in Austria? In the USA, in most states, you must be licensed to have engineering in your business name. Your name is usually suffixed with PE.

To use the "Ing" title you have to have a certain education and a few years of practice. To be honest, it doesn't mean much.

But in the USA you can refer to yourself generically as a Software Engineer without any formal education.

Yes. I don’t think there is even a PE licensing path for software, so even with a CS degree you could never become a licensed engineer. You have to work under a PE for 5 years, then take the PE exam, then CE credits.

I have an MSEE, but can’t put engineer in my title. Though Ing. sounds kind of cool.

There actually was a PE for Software Engineering, but it is being discontinued, because almost no one took it.


Wow, didn’t know they had one. I wonder how they handle the licensing as there would be no one to complete the 5 year understudy; chicken and the egg.

It bothers me to no end when I hear programmers here in Ontario call themselves engineers when they neither graduated from an engineering program nor completed their PEO exam. It's very much illegal, and the PEO _will_ go after companies that allow their employees to title themselves as engineers.

How dare you leave out electrical engineering!

That said, there's many variants that are perfect valid: network engineer, sales engineer, systems engineer, etc.

And most states in the US too. Just rarely enforced. Only a licensed PE can do business as an “Engineer”. (For example have a company with the word Engineer or Engineering in the name).

I believe the city of Portland OR recently sued a Danish man for "practicing engineering without a license". They were upset at him for generating strong proof of shortened yellow light times at intersections with red light cameras. He had a degree (and I believe a certificate) from Europe, but it wasn't recognized by the Authorities.

Protected term in Canada as well, but I don't know the last time anyone that was used against in any level of court. During my education there was a case study of some guy digging basements calling himself an engineer when he wasn't but that's about it.

And still others have, though it may not be appreciated in all industries, _Chartered_ Engineers.

And how many 'engineers' would tack on 1000x too much complexity to justify their 'engineering' effort?

software in many cases is more art than science.

I think this talk is relevant https://youtu.be/lLnsi522LS8

Please bitch "associate professional"

If you're a company the size of Cisco, and you have two widely publicised vulnerabilities like this, then this sort of failure is _always_ management's responsibility.

That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers, but it can take other shapes too. It might be poor oversight and QA processes, or a failure to hire and/or assign competent engineers for the task.

> That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers

This is the narrative that always shows up in HN. Big bad management vs the hero engineer. It makes us feel good about ourselves while also absolving us of all responsibility. Its a bullshit narrative.

You live in a nice world, because usually the actual power (and duty) of a dev team member is to advise, not to agree or refuse.

You can always quit. Or force them to fire you for refusing to implement a non fix.

In reality though I doubt this narrative even occurred. Some incompetent engineer likely proposed this fix thinking that it was actually a fix.

Edit: I see I've been downvoted for this comment. If we were real engineers working on things like cars and bridges we'd actually be held accountable. Take some pride in your work people, this is one of the most in demand professions in today's economy.

At a large organisation I worked at years ago we used to call it Coder Shopping - when a manager would do the rounds trying to find someone who would say "yes" in a situation like this.

Invariably there would be someone who would, and if you were the person who had initially refused you probably wouldn't even be aware that it had happened until the bug reports came in...

If there were actual accountability, software engineers would have a much better lever against management. For some software this is case. If you write safety critical software you can be held personally responsible for accidents. In those industries engineer pushback is much more effective.

My background is safety-critical and -- speaking from first-hand experience -- it was truly incredible how much push-back we'd get when raising safety issues.

System tick timer rolled over after 2^32 1ms clock ticks (a little over seven weeks) and the software mishandled it by rebooting and losing control of the process. It wasn't until QA got involved (it turned out they saw it on a long-term test) that we were cleared to actually fix it. It was a two-character change in an inline function!

In some places "cover it up, don't discuss it in email" is the natural response. It really depends on the management and the organisational culture. You can have management who support the engineering team, or you can have management who micromanage and overrule and end up turning a two-day hotfix cycle into ten months of "try soldering a resistor to every single pin on the PCB".

Ha ha, a resistor on every pin is a signal integrity hack.

Quitting assumes the possibility of finding a job around the corner and not having people that depend on you.

You might be surprised, but if you move to Europe, it is actually so! Move to Berlin of Frankfurt. There is always work for Software Developer here, it is hard not to get one and you get spammed by recruiters all the time. You don't have to take shit from management, you can just go elsewhere. Everything is also very near.

I live in Germany, now try to convince the family to move along.

Well as a European I find the "uproot your life for a better job" mentality completely alien. Short of a war I'm not sure what would make me leave the place I call home.

> Quitting assumes the possibility of finding a job around the corner and not having people that depend on you.

Quitting assumes the possibility of finding a job after spending 4 months doing leetcode 6 hours a day and not having people that depend on you.

I like your spirit, but there's another conclusion we can draw: very few of us outside of a few special areas in embedded are "engineers". We tinker with bits and bytes to make business systems. Sometimes we bump up against some general engineering fundamentals. But we're just technicians meeting business goals.

Yep, and the same can be said for “real engineers”. Most EEs, civil engineers, etc work on boring stuff as well with negligible risk of doing any harm.

You suggest it was just one lone engineer being incompetent. Then you respond to downvotes with the non-sequitur "take some pride in your work" - are you suggesting pride is a fix for incompetence? What?

Absolutely: Cisco made this happen, not Engineer Alice. If your boss asks you to do something within my ethical boundaries and you have 5 tired co-workers in favour of shipping it, eventually you'll break and say "fine". It doesn't make you responsible or the one that pulled the trigger.

The fact that this fix made it to prod tells you that something is deeply deeply broken with cisco’s processes that goes well beyond any individual’s responsibility.

Exactly. It's not like somebody just did this like in a three people startup. There are code reviews and quality management and many more, and this lousy "fix" has passed all those stages and instances. If nobody declined this, there's a serious problem within Cisco.

We can "No True Scotsman" this all day long.

It is bad management to hire incompetent engineers.

It is bad management to fire competent engineers that push back.

It is bad management to make employees feel as though they would be fired for standing up in any way to management.

Take away the rhetorical shortcuts, and responsibility still always falls at the feet of the ones that have the power to make the decision. As long as engineers are hierarchically bound to obey managers upon pain of sudden and unplanned unemployment, the managers are ultimately responsible for everything the company does. If the company does not have tenured engineers that have stop work authority, the managers are fully to blame. And if they do, management still might be partially at fault.

We have this headcanon where a competent engineer could stand up, sweep away the slide deck on screen with a wave of the hand, and offer ironclad mathematical proof that the quick fix is no fix at all, and that the long, painful, but correct fix is what is Right for the Company (tm). The manager would then do the slow clap and tell that engineer to fix it right this time. It is a fantasy that is a great comfort to those of us doing maintenance work on a massive crock of shit, making the payments on technical debt that we are expressly forbidden from making any attempts to retire. But it is a fantasy. A real manager would cut the engineer off in mid-proof and say that the quick fix is happening, and that would be the end of the story.

The only way for an engineer that is "at will" and without a contract to push back is by falling on their own sword. They can quit, at great cost to themselves, to create a very minor inconvenience to the company, who will end up doing what it wanted anyway.

Being in this situation myself, you're damned right I have no responsibility for the crap code that goes out, because I have almost no autonomy in the manner in which I do my job. I cannot work outside the scope of the maintenance tickets I am assigned. And my company rents me out by the hour to the customer, so it has negative incentive to allow me to build up infrastructural capital that would allow me to do the same work in less time. I will blame management all day long, and then a bit more on evenings and weekends.

There are plenty of poor engineers, but this “fix” has to go through a process defined and enforced by managers. In a good process, multiple engineers would review the code, and if the office culture is good, then objections will be brought up and listened to, and if a fix is insufficient, management will understand and assign resources to make it right. None of those things depend on any one engineer or manager. They’re a culture issue. But the only people with the power to fix a broken culture is management. Engineer Alice doesn’t have that power herself.

In large orgs there would also be some kind of security team which would have input on the CVEs. Cisco does have them. It sounds like they were not involved at all here for some reason.

The only way, and I mean the only way, this gets blocked is because a manager (not a dev) assesses the risk of improper fixing to be higher than the reward of not doing a shit ton of work justifies.

A significant portion of that is making sure the manager is fully cognisant of what the quick fix means. Alice's job is to ensure that happens. The managers job is to advocate to his bosses that this has to be done and is a net win to do it properly.

It is as likely manager person failed in advocacy as it is she failed in information transfer, but there are plenty of gradients here.

And I might add: especially at big companies where being yes men is a higher valued characteristic than being technical

Oh, so much this.

"I'm sick of you bringing me problems, bring me solutions! And I want that blockchain based big-data cloud IoT solution by the end of the week!"

It really is amazing the kind of mental gymnastics HN commenters will display in order to shift the blame to everybody else, isn't it?

The question here, is if it would stop 90% of the attacks, would it be a worthwile way to spend 5 minutes.

And the answer to that question is almost invariably yes.

Well, no.

That's like stopping 90% of a flood. The other 10% does more than enough.

Yes, but this is software engineering. Where there is no such thing as malpractice, and where managers can not be prosecuted for criminal negligence even when their overriding of the engineers professional position gets dozens of people killed. In this delightful world, managers rule the roost and engineers do not have the ability to refuse to do anything.

If management let this important security fix get executed and deployed without a competent engineer in the loop, that's a horrible mistake in itself, but it's far less likely than management choosing this alternative based on accurate information from engineers.

Big bad management gets to keep their jobs when they make bad product decisions, so let's call it a wash.

If you read the parent post and think "bad management" I don't think the message was clear.

Stereotypes are valid first order approximations.

Who hired the incompetent engineers?

I've been Engineer Alice before (not with a security issue, but a pretty bad systems issue). In case anyone is ever in that situation, here are some more productive replacements for that existential sigh (in that they sometimes work). Choose the one most suited to manager's biases. The cliches are important: think of it as giving manager an easy way to explain it to their manager.

> Yes, but that would only help with this specific case. The proper fix would ensure that we don't see a similar issue tomorrow. We don't want to prepare for yesterday's battle.

> Yes, but the extra regex rule would be new code, which would need its own maintenance in perpetuity. Fixing it properly would let us remove the existing code. Removing code is the most efficient thing an engineer can do, and will save man-hours in the long run.

> Yes, but the extra regex rule has worse runtime performance than the proper fix. [Performance is our big competitive advantage | We don't want to fall behind in performance | We are already behind in performance]. We don't want to create a customer-visible performance regression.

No! The answer is an unequivocal “No“, without any „but“s or anything like that. The fix does not fix the problem, it is not even a fix, just a wrong code change that sets out to do something but does not achieve it.

Answering „yes“ is a lie here. This is different from a fix that fixes the problem in an ugly way, where „yes, but...“ is applicable.

In this specific case (I have never encountered a proposal quite this bad), the correct answer is probably "Give me the proof of concept code and about ten minutes, and I'll give you a version that works despite the patch." You're right about that. "No" is likely to bog down the discussion with "but Bob says it will work", and it's better to skip to the end.

I'm envisioning more of a middle-ground case, where the code is still obviously wrong but it's harder to cleanly demonstrate why.

Sometimes you need an interim solution. A decent organization will be able to manage them by replacing them with actual resolution.

Cisco has a bad track record in that regard and software in general.

My employer has reached a settlement with Cisco after buying their "firepower" solution which has been plagued with basic software and usability bugs.

Ten minutes? How long does it take to type a user agent string into the curl command line?

In my case, a user agent string pretending to be Firefox is already part of my .curlrc, so it's no time at all, just grab the PoC code and fire away. (I've got commented out user-agent lines for a bunch of other browsers that I can uncomment as the need or mood strikes me.)

I'm basically assuming five minutes to read the script, negligible time to change it, and another five to make sure it worked.

You must live in a very nice, ideal world, where simply saying "no, that's not the right way to do it" will convince managers to ignore the pressures placed on them to, at times, value speed over correctness.

What they are saying is that this isn't just "the wrong way to do it". It doesn't actually fix the problem it's supposed to fix, therefore it's actually wrong to say it's a functioning patch.

Of course, but their boss doesn’t know that (or care about it), he just wants to see the fixed bug graph line approach the total bug graphline.

It’ll be totally explainable as engineer failure if something somehow doesn’t work.

Apparently I do? Not sure what you want to hear here, but management does listen to me and my engineering peers. Sometimes things are a bit gray and fuzzy, but in this case here where they definitely are not, I am absolutely certain that there would be no overriding of our arguments.

That's awesome, actually. For me, it's not as bad as Cisco seems, but there are definitely times I get overruled and have to take shortcuts that I'm not always comfortable taking.

Indeed, and what's worse is that many managers[1] will stop listening after "Yes, ...".

[1] The incompetent ones, of which there are many.

How about:

No, it won't actually fix anything, it'll only look like it did for most trivial cases, and 6 hours after releasing it our company will be on the frontpage of Reddit.

I actually said something like that during a meeting. "Yes, you can do it that way, but if someone finds out we will be in the news." This argument worked surprisingly well.

Or you can just do the minimum just like the manager and cruise on the ship until it sinks while surveying for another ship to jump on at the first opportunity.

> Manager: And we can just prevent it with an extra regex rule in the front end code? > Engineer Alice: ... Yes. irrepressibly existential sigh

The answer is "no", though. This exchange sounds like the decision was between "fixing it cleanly" and "fixing it with a hack", which is roughly equivalent with fixing the problem at the expense of technical debt. But this change does not fix the problem at all.

I have to agree that the answer is no.

curl -A "anything but Curl" x.x.x.x

the problem has not been fixed by a user agent string.

Engineer Alice is the only person incompetent at their job in this conversation. If "irrepressibly existential sigh" is how you argue security with your bosses then maybe you're not senior enough to be in meetings like these.

The Manager and QA Engineers here depend on the expertise of the engineers. If the engineers fail to communicate key details of the situation, then that's on them. Sure, the boss is still at fault for depending on a shitty engineer for decision making, but that's all.

If you answer with the existential sigh, you’ve had at least 20 conversations with a similar outcome before. Nobody arrives at a new company that jaded.

I see your point and I suspect that that's what the author meant.

At the same time, I know lots of, well, arrogant nerds who get into "existential sigh" mode as soon as someone with less technical skill than them says something stupid. That's more how I read it.

Ah, yes, I see how that is another valid interpretation.

I guess for me the word existential is a bit different. For an arrogant developer it would have been something like ‘long-suffering sigh’.

Unless, I guess, you feel you are so smart that your entire life is defined by interacting with morons :P

> The Manager and QA Engineers here depend on the expertise of the engineers.

Both should be sufficiently competent and knowledgeable to know this is a terrible idea, especially in the long run. Otherwise they do not understand the task they have been given. So no, Alice is not the only incompetent one here.

In similar situations I

1. Explain that the 'hack' does not actually fix the problem. This is absolutely the engineers responsibility. In your example Alice does not say that. This has always been sufficient.

2. I make a habit of sending emails or keeping minutes confirming such meetings and the details of who said what.

This is about all you can do. But you're right. You have to explain it and document it: "It's trivial to change User-Agent, and any decent hacker would probably not be using 'curl' as their User-Agent. We can look at logs from past events to validate this assumption."

Bob is either a brown-noser or a complete turncoat. Don't be like Bob.

Bob is probably an engineer with twenty years experience, who's been thoroughly ground down by management and peer power-plays and has realised that the best solution is to keep your head below the parapets!

Lol, this is too true. Though I think I learned 80% of this in my first couple months at my first real full time job

Ironically, this is the change most likely to break things for customers that use curl for automation.

Ouch, that hits close to home. It's like you've sat in on all the same meetings I have, word for word.

Manager: Bob get to work on the quick fix that will mitigate some of the attacks based on the published exploit to go out within a day, Alice get to work on the correct fix to address the underlying issue to go out in a few weeks when it's fully tested and ready.

Manager: Alice, yeah, I know I said we'd carve out time this sprint for you to work on the long-term fix. But this other hot feature request came in, and there's potentially a lot of money riding on getting it done quickly. So can you put off the long-term fix, maybe a week? Two tops. You can just put the ticket.. yeah that one.. back on the backlog. There you go. Thanks so much!

Then someone realises one of their enterprise clients is using curl for a critical system, and said patch has completely broken it. Or more likely, they realise after customer/company calls complaining about how their systems are now busted...

This reminds me of the old BOFH posts from days past.

The one where the "appeasement engineer" shows up to fulfill the guaranteed response time requirement.

Wow, You got this one!

>how a team of well-paid comfortable engineers got together and decided this patch was a good idea

Test-driven development

Only if you are really bad at it. Really, really bad.

It's completely unrelated to any development methodology, this is someone unaware of basic HTTP protocol semantics.

So just bad development, period.

It looks like this is a low end Cisco device, released over 5 years ago. Patches are probably handled by underpaid interns, not well-paid engineers.

i'll raise you a case where a team of very well paid senior engineers/architects and PMs dismissed a remote execution vulnerability down to a very low "some next release" priority on the grounds that "notepad.exe doesn't seem to do any damage" - as you may have guessed the vulnerability PoC used notepad.exe . After seeing that with my own eyes, this Cisco curl is just "meh" for me :)

Every time I see an examle of this I think "holy crap we're all going to die". The bus drivers are blind.

I've had some visibility to big networking equipment manufacturers... the security or even stability on their web interfaces is terrible. It's like a strange rule of thumb for all the networking equipment manufacturers. If you have choice, disable the web interface.

> how a team of well-paid comfortable engineers got together and decided this patch was a good idea.

That's assuming that the well-paid and comfortable engineers decided on and implemented the patch.

I don't see what the fuss is about. This is an effective mitigation, given that software can't just arbitrarily lie about its user agent.

I guess the biggest problem is that the mitigation isn't future-proof. Someday, there might be other command-line tools besides cURL that can be used for this kind of attack.

A better fix would be to just drop all incoming IPv4 packets that have the "evil" bit set. (https://www.ietf.org/rfc/rfc3514.txt) Extending this protection to IPv6 is left as an exercise for the reader.

Ah, but what about packets that aren't evil, just gullible and manipulated by some third party?

Well if you’d read the RFC it’s obvious it’s your routers responsibility to flag those bits as evil. And if you’re using heuristic intrusion detection systems you need to do flag each bit as evil 50% of the time. It’s good we have such amazing protections in place.

Then add a "tainted by evil" flag to the ipv7 spec, or better, just add a "pure" flag to future ip spec.

I think I'd almost prefer an "alignment" flag, so we don't mischaracterize all the Chaotic Neutral packets.

But then what to do with the Lawful Evil packets ?

Add an x-Lawful-Warrant parameter to provide a warrant. It can’t be read by anything but the best AI technology and is included in every HTTP request. No warrant, no response!

It's not like curl can't easily spoof a user agent either.

We can claim the LSB of the sequence number.

I think you should have added a /s tag... some people don't get jokes unless you hit them over the head with it.

I will never stoop so low as to telegraph my own joke.

I couldn't agree more. The joke construction was a bit weak though, when you miss a good part of the audience. Add something like, "I mean, could you imagine the chaos it would cause if IE told websites it was really Mozilla?" and you demonstrate mastery of the subject matter, which should be enough to let other experts know you were facetious rather than ignorant. Unless you have timing issues... or need the comedian's plausible deniability.

I don't find that as funny, I think making the sarcasm more obvious makes the joke less funny as well. I prefer to take my chances than to go overboard and neuter the joke.

I like the original more without your embellishment. The addition sounds like every attempt at follow up humor on reddit.

Yeah but it's not Reddit, it's HN. You have to know your audience, read the crowd. On Reddit it's 90% sarcasm so there's no fixing it. Here it's the reverse and people take things seriously without a tell. You just have to bury the tell in another joke or it will ruin the funny.

"given that software can't [ridiculous thing no one has ever claimed]" is plenty for a tell.

Just the fact that this “fix” exists tells you some people believe differently.

If you can’t discern the joke, the joke isn’t for you to begin with; so it doesn’t matter.

Sir, your assessment of this particular joke is excellent. We should start a joke approval committee to avoid further confusion.

Yes, very well, as first action of the Joke Approval Committee, I propose that this august body instantiate the Recursed Approval Subcommittee, whose role will be to instantiate the Recursed Approval Subcommittee.

That's a perfect example you should post to JAC before. It would not pass, please never post this "joke" again.

Welcome on board :)

As soon as the subcommittees hit their break condition (0), they'll break for coffee and donuts, and I can submit my "joke" to the JAC. And I don't make a habit of repeatedly posting the same joke, but I cannot make the promise in this case.

It puts the motion in the basket.

In textualform, sarcasm and ignorance look the same. otoh, it can sometime be quite funny to see responses from people that take it seriously...

I've never posted sarcasm online without at lease one person (that guy) who takes it seriously and decides to reply. But I think it's worth it because the people who get it will get it. If a small group doesn't then who cares (if everyone doesn't then you did a bad job constructing a joke).

Then you must encounter Poe's Law frequently....?

All the time. After having seen the kinds of things some people will say in earnest, I will never begrudge someone missing my sarcasm.

Exactly. Hacking your client to use another user-agent or another IP address is illegal under CFAA. See Craigslist Inc. v. 3Taps Inc. etc...

So when I check desktop site in chrome I'm hacking?

Of course! So is right clicking and selecting inspect element to modify the page!

That's why super secure websites block right-click events on their webpage. Makes them unhackable!

I would call it hacking.. you sometimes run into those websites with 50 different opacity bits for all their web app things, and you don’t want to deal with all that rubbish so you set a “opacity: 1 !important” and the whole website looks off but it’s finally useable. I’d call that a hack.

Or also when you run into those websites designed on a $2000 screen that thinks it’s tres moderne to use 808080 text, so you “hack” it into readable text.

I think those are all happy little hacks :-)

Actually you can use -A to change the user agent in Curl itself. For example:

  curl -A "YourNewUserAgentString” http://url.com
Furthermore, there are free plugins or add-ons for both Chrome and Firefox that let you change the user agent and say you’re using IE, a mobile browser, a google bot, or anything you want the user agent string to say.

Most programmers should be able to write a small script using Python’s urllib or urllib2 to do a basic PUT or GET request with any User Agent. Heck, just type “change user agent python urllib” into google, click on the first stack overflow link, and copy/paste the answer. You can use Go, ruby, java, nodeJS, or PHP instead as well.

Edit: Maybe your comment was sarcasm :)

> Edit: Maybe your comment was sarcasm :)

I still appreciate you putting in the effort to explain. Have an upvote!

I have to fix my sarcasm detection AI again because of this comment

You're welcome!

Exactly, once they roll out the patch with a boolean OR in the conditional as '$http_user_agent ~* "wget"', it'll pretty much be hack proof!

It does seem so obviously simple to sidestep that it makes me wonder if they pushed it because they had an active attack they believed it would mitigate. Were they concerned that the attack was already baked into some automated script in the wild and this could perhaps at least trip up the script long enough for them to engineer a real fix?

Scripted attacks can of course be modified too, but it still takes more than zero human effort to do it.

Well, they can, but first they have to flip the RFC3514 "Evil" bit.

Curl can, easily. Curl is not just a client to get a file and pipe it to other programs, it's a diagnostic tool, and it has all sorts of parameters to help it simmulate all kinds of interaction with a webserver, including the -A parameter to change the user-agent, this is why this 'fix' is so stupid.

I see, thank you for the explanation!

Well played, sir.

It was nice of them to explain it, though, in case I meant it seriously!

This is quite funny

I am not laughing one bit let me tell you. Honesty in user agents is one of the last few defenses we have against hackers running rampant on the internet and turning it into some lawless wasteland where people don't even answer truthfully about their A/S/L.

> "A/S/L"

I have not seen that in a long time.

I guess American Sign Language isn't as widespread as it used to be, now that everyone just texts all day anyway.

They might be able to fix it by making the only valid user agent “not-curl-honest”.

curl -A "iswearimnotcurl" <exploit>

libcurl is open source, I would imagine it'd not be difficult to make it "lie."

Someone who also did not get the joke is going to read your comment, and suggest that libcurl being closed source somehow fixes the problem.

(Hint: the curl command already has a --user-agent parameter which allows the user to set the user agent string.)

Eh? Software can do exactly that.

How? Do you think developers would be so crazy as to add some sort of "--user-agent" option to software like curl that would cause arbitrary strings to be presented as the user agent? Why would somebody write software to do this, just go on the internet and tell lies?

I thought all this security nonsense ended in the 90s when they introduced the IS-MALICIOUS-REQUEST header. It must be, because I haven't seen any web traffic declaring itself malicious since then.

I believe most devices implemented now 11 years old RFC 3514. So evil packets and non-evil packets can be easily distinguished.

Even without --user-agent, you can still pipe 'echo -e "GET ...\rnHost: ... "' through nc, or even telnet.

Therefore, we obviously need to patch echo (and all echo shell builtins) to refuse to output strings containing "GET", "POST", "HTTP", or "Host:".

> Therefore, we obviously need to patch echo (and all echo shell builtins) to refuse to output strings containing "GET", "POST", "HTTP", or "Host:".

Unfortunately it is also possible to do this using file redirection or to write a new program that will make a TCP connection and send arbitrary data through it, making it necessary to do the same for all editors, compilers and interpreters.

That sounds like a lot of work though (there are many such programs), so maybe we ought to just patch the kernel to prohibit any of those strings from being written to a file descriptor. There can't be that many false positives.

Why stop there? The user can easily modify the kernel to disable such prevention measures for malicious usage.

We must have a regulation to require a hardware level detection and prevention features of curl which all hardware vendors must follow.

oh god stop some politician might see this and take you for being srs

But users have fingers, which can be used to disable or subvert said "features". Thus, those fingers are just gonna have to go. They're a security risk, and security risks are against the regulations.

Ooh boy I'm imagining some bureaucratic version of an operating system that uses this sort of security up and down the entire stack like some rube Goldbergian contraption to whack a mole a particular vulnerability. Like an operating system designed by doctor Seuss.

Your downvoters are terrible at detecting satire.

I know, that top comment has been a rollercoaster, +6, +2, 0, -3, -1, 2, and so on.

Thanks. I needed that laugh.

While that's probably technically feasible, that sounds like something only a terrorist would do anyway. All the more reason to ban the entire program, I say!

Only software developers with no honour would ever do that.

Pretty sure that was sarcasm haha

And it is also very good that regexes are simple, unambiguous, easy to debug and their engines are heavily scrutinized that no code execution/undefined behavior is possible.

Cisco is crumbling under its own weight.

This is a symptom of the rot in their management, and probably also a sign that they have hired too many incompetents.

It probably also is a sign of the current age. After the recovery from the IT-bubble programming got really hot. Thus: Too many of the new programmers wants to be programmers because it pays well - not because they love their craft. So therefore we have a bunch of well-paid but uninterested people seeking jobs at prestigious companies.

Culture matters. I want my socially maladapt terminal junkies back plz.

> I want my socially maladept neckbeards and terminal junkies back plz.

the MBA types don't like these hacker types - personality clash and whatnots.

But the MBA types control the company from above, and the hacker types don't like to do management work. The result is obvious.

Some time ago I went through the list of all the major router manufacturers and rated them on 1) security, and 2) long term usability, and 3) culture.

My conclusion was that I would buy my infrastructure from Allied Telesis. It's pretty much a Japanese version of Cisco, but it's still healthy.

Ubiquity was number 2. I refrain from buying from them only because of their glossy UI.

Mikrotik was on that list. Until I saw how horrible their winbox protocol was. And their implementation of SMB.. I must assume there are still plenty of unknown RCEs there.


At least Allied Telesis documents their backdoors :)


I guess there is nothing good.. what is wrong with people :(

Oh god...

I had several Allied Telesis switches at a previous job. The hardware was fine but resetting the configuration password was a remarkably painful process.

Can you elaborate on Mikrotik? I've only heard good things about them and am very satisfied with the one hAP I bought from them.

They have had several remote code execution vulnerabilities lately (summer 2018). While they were very quick to patch them they did not notify their customers in any way. There was nothing on their website that said anything about the urgency.

Instead of reusing functionality that exists in the router already (ssh?), the authentication for winbox is something they built themselves. It was in the winbox auth that the main security flaw was. It just looked really bad to me.

The winbox client also downloads and runs any DLL that is sent by the winbox server. The winbox client has a windows certificate so all it's code is trusted. So own the router and you get the admins workstation too.

It just feels like maybe they hired some random guy without much appreciation for security for doing winbox.

The SMB server also had a rce a while ago.

That said, I guess that if you disable winbox and stuff that should not face the internet, you are probably safe?

Too much for me though. I would not feel safe.

WinBox is not that simple and cannot be replaced with SSH. WinBox works on Ethernet level, so one can connect to router by MAC address and recover when IP level configuration is invalid.

disable winbox and use their webui?

also, while not perfect, what is so horrible about their winbox protocol?

Culture matters. I want my socially maladapt terminal junkies back plz.

I know some fastidiously groomed VSCode front-end hipsters whose entire bodies involuntarily tense up if you suggest a lazy technical workaround to them. It's funny and makes me feel good about the future.

But I get you; I agree completely. You do need people who really care. They're just not as easy to stereotype any more (if they ever were.)

Your first point is just blabber. Cisco is a large company and this is a $100 product that they don't really care much about. It's the type of product that is produced at the cheapest possible cost, by interns. The way this product has been developed and managed is completely intentional and not a sign of mismanagement. Come back in 3 and then 6 months and let's see what effect this has had on their revenue and market cap, and then you can start talking about mismanagement.

As a sign of the quality of the current median developer, I think you are spot on. But I don't know that it's worth griping about. That is the world in which we live ... so live in it. In fact, all the better for those of us (if I may dare to put myself in a more elite group) that are highly skilled ... we don't have to work on some me-too unimportant SOHO router, at correspondingly low me-too unimportant wages.

Bwahaha, hired too many incompetents?

They have outsourced the fuck out of their projects to uh, questionable means.

This "fix" seriously hurts Cisco's credibility. How can you trust their products? Perhaps they are thinking that noone gives a damn anyway after no less than five backdoors² were found in their products in 2018 alone? Just incredible.

² https://www.tomshardware.com/news/cisco-backdoor-hardcoded-a...

Cisco have a loong history of things like the. I mean go back to 2005 when they actually sent people to a security conference to rip pages out of the handouts that mentioned a security vulnerability in their products (https://www.computerworld.com/article/2482483/cisco-s-blunde...)

That (and other bad security responses) have not really hurt them, so at this stage, this won't make a lot of difference, I'd expect.

Cisco is in the business of selling big, black, expensive boxes that have a lot of security badges and fancy icons. People who buy such boxes don't care if they actually work, they want a big box so that they can claim they "invested in security".

Actually, they want the Cisco box because it has those 'badges' and that then allows them to check off all the check-boxes on the semi-annual security compliance report paperwork they have to file with some other department.

But you are right, they don't care if the box is secure, they just care that they have a CYA that allows them to "check off their own boxes" on their reports.

What would you buy instead of Cisco?

HP and Dell are the same thing. From experience a decade ago, the HP usually did not have the enterprise features they advertised.

The other minor brands are very hard to procure if you're not in the US or a primary English speaking country.

Juniper? Extreme? The list is not very long, but doesn't contain just Cisco.

Last I checked, long ago, they were impossible to procure in France.

Pretty sure Extreme is still non-existent in Europe as of today.

For all the flaws of Cisco, well the only flaw is the price, they can deliver in any language anywhere in the world.

Juniper has offices and partners all over EU, for example in Paris [0]. Similar to Extreme [1].

I don't know what problems you encountered in France, but I can assure you both of them are very well represented across the whole EU, even less developed parts of it.

The reason managers pick Cisco is mostly that "nobody gets fired for picking Cisco" imho.

[0] https://www.juniper.net/us/en/contact-us/sales-offices/paris...

[1] https://fr.extremenetworks.com/

Do people trust their products?

Meanwhile the US goes around telling other countries not to use Huawei because they can't guarantee security. [1]

[1] https://www.forbes.com/sites/zakdoffman/2019/02/19/huawei-fo...

I doubt they are referring to their security bugs.

There can be more than one company that's a security risk.

User agents shouldn't exist any more. They serve only to help unsuspecting users be fingerprinted.

Considering recent chrome, firefox user agent contains id string of almost every browser in existence, this is so true:

Mozilla/5.0 (...) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36

You just wait for Edge-based-on-Chromium's User-Agent :D

Per a buddy who is using said Edgium (Chromium-based Edge): https://i.imgur.com/p6ZMoJY.png

Oh ffs


I assume real live websites are detecting "Edge/" and applying workarounds for the old rendering engine.

Like the maybe-not-actually-true reason why "Windows 9" was skipped and we got 10 instead?

It was too edgy for them.

I vote we change all user agents to just "curl".

“Apache” would be ironic.

Might as well save 3 bytes and just call use “c”.

Well, the next best thing is what webkit is doing (https://webkit.org/blog/8042/release-notes-for-safari-techno...) by freezing the user agent.

So as time goes by the user agent becomes increasingly homogenous and so, less useful for tracking.

I believe this change was rolled back, since my User Agent is "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.2 Safari/605.1.15".

Rubbish. They are incredibly useful for debugging.

They would be a lot more useful if they were more straighforward and honest.

Just give me "Chrome 68" instead of "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"

firefox has a sane user agent. mine is:

>Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0

edit: sorry, accidentally submitted early

So what. Removing one of the largest sources entropy available for fingerprinting users is important. We shouldn't maintain the terrible long-term effects[1] of tracking everything just to help you have an easier time debugging.

[1] http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and...

Does it really provide that much entropy? Maybe back in the days when browsers didn’t auto-update, but intuitively it should only provide about four bits at most (os: Android iOS macOS windows, browser: chrome safari edge ie ff) except for a few rare users who don’t stay up to date or choose a weird browser (and those folks probably are a little confused about how privacy works).

ohhhhh you betcha.


i ran this on my work laptop (windows 10 / firefox - mostly default settings). the "details" tab shows a full breakdown of each fingerprinting component with a "similarity ratio" (percentage of fingerprints that share the same value).

every entry is 20% or greater, most of them around 50% except for timezone (4%), canvas (1%), and user agent (0.12%).

so, yeah, user agents are still the largest source of entropy by a wide margin for fingerprinting scripts.

I suspect this is just stale data. I can’t think why the most recent version of a browser would put me at one in a thousand.

There are better solutions available than hanging functionality on unreliable vestigial bits and pieces that shouldn't be there anyway which, because others abuse the functionality, you can't trust to be correct for debugging purposes.

99% of the traffic has the correct user-agent which is useful in tracking down issues that are browser specific. The other 1% will just get ignored as noise. And it's not like we can't tell what type of browser they are using with browser specific objects that we can pull from javascript. We just can't infer the version of the browser which is critical for debugging issues.

I use a randomized user agent and every once in a while get rejected by a site that has decided my browser can't be supported. Usually when I'm on some Safari version. I doubt much effort is being done to identify fake user agents since there are better fruit to pick in the fingerprinting game.

That might be true, although you don't list the solutions, but it doesn't refute my point in any case.

Per https://webkit.org/blog/8042/release-notes-for-safari-techno... the user agent is frozen in webkit now.

You're writing your websites wrong.

I've had to debug around a short-lived bug in the Firefox rendering engine that only showed up in the FF rendering engine before. The site itself? Standards-compliant; there was flexibility in the standard (ironically, for flexboxes ;) ) about performance constraints and our site just fell down Gecko's slowness tree and hit every branch on the way.

Given the engines are allowed to vary, this experience is inevitable from time to time. The user-agent is really valuable in helping us figure out when that's what we're seeing.

If there’s a way to write them without the occasional browser-specific bug, I’d sure like to know what it is.

They help you figure out if a user agent wants a mobile view of a page without Javascript... which is pretty useful.

We don't need user agents for that. There's an easy way to tell whether a visitor wants a cleaner view of a page, without Javascript: Yes.

I said "mobile", not "cleaner". Sizes and aspect ratios are different on phones than PCs.

> I said "mobile", not "cleaner". Sizes and aspect ratios are different on phones than PCs.

User agent doesn't tell you size or aspect ratio.

Even if you know I'm running Chrome on Android on a Galaxy Note 8, you don't know the size (physical or pixel) or aspect ratio of the viewport, because: (1) I may or may not be viewing in side by side windows, (2) the screen can be in any of three resolutions, (3) the screen can be portrait or landscape, (4) the browser may or may not be set to use the full space on the long dimension of the screen.

OTOH, CSS Media Queries answer these questions.

Not necessarily. Websites should use CSS media queries to work at all aspect ratios and sizes without needing server-side code or JavaScript.

the could be accomplished just by having User-Agent: mobile, all the other data in the user-agent are useless for this purpose.

There is presumably some way, at least hypothetically, to fix the mess that User-Agent has become. (Everybody is 'Mozilla' now! And yet somehow a unique snowflake 'Mozilla' that lets you be tracked...)

But it's not quite that. It's not that a device is "mobile" instead of "stationary" that matters. It's maybe that it's a touch screen. It's maybe that it's a small screen. (With an especially uneven aspect ratio?) Maybe that it's handheld. It's maybe the combination of all of those things. But it's not really that it's "mobile" exactly. And there are, and possibly will be increasing proportions of, devices that have some but not all of these qualities we package together in our brains as 'typical mobile'

All of those small differences of mobile devices are better served by media queries I personally think. You could replace "mobile" in my example by "small-screen" if you prefer.

Agreed. Media queries need to be improved though. There are no good media queries to detect "touch screen" and/or "no keyboard or mouse/pointer, only touch screen", only JS hacks that attempt it. This can matter. But yes, improving media query (or input method query?) APIs for both CSS and JS is, I agree, the correct solution to User-Agent madness. It's just not entirely straightforward or obvious or easy, and as always getting all browsers on the same page is an issue.

Well at least we acknowledged there's some utility.

Now consider that mobile devices are different as well..

The current user agent does not give you much insight about what type of mobile device you have anyway, especially on Android.

I use Firefox for Android on ChromeOS -- it produces interesting results.

I have been having a shitty day and your comments in this article have honestly cheered me up a bit. Thank you!


So instead of sending a user agent, send only the page ratio!!

This is not too helpful -- you can resize the window in such a way that the ratio changes...

Send the screen resolution

For which screens ? Why would that be helpful in any case, given the browser window is not the size of any of the screens ? I'm not sure what you would gain from getting this information.

I think CSS is probably good enough that we can live without UA sniffing for this.

We sometimes render different HTML to handle browser and device behavioural variations, including FOUT-management variations in how they load assets that therefore can't be handled in JS even if we wanted to. Historically we also redirected to a pure HTML/CSS version of our tools when IE9 or earlier was detected.

I'm pretty sure deleting User agents field will introduce another non-standard X-prefixed header.

That is not correct. They also serve to exclude users of browsers you dont know or like.

Applications are open for YC Winter 2024

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact