Of all the security post-mortems I’ve ever wanted to read, it’s sad I’ll probably never get to read this one and its tale of how a team of well-paid comfortable engineers got together and decided this patch was a good idea.
Having been involved in meetings where "stop ship" was the phrase of the day, I'd bet money that the following at least vaguely resembles a real conversation:
Engineer Alice: We should really fix this properly.
Manager: How sure are you that the proper fix won't break something else for $BIG_CUSTOMERS who are responsible for $OBSCENE percent of this product line's revenue?
Engineer Bob: Uh, ten percent on a good day? Do you remember the last time we applied a "simple" hotfix to this function? I almost had to sleep under my desk.
Manager: And we can just prevent it with an extra regex rule in the front end code?
On HN its always big bad management who is the cause of every security problem or shoddy piece of engineering. If only that pesky management would screw off then we could do things "properly".
You'd be suprieed at how many incompetent engineers there are out there. If "engineer" Alice in your story was actually competent they would never agree to implement the proposed "fix". Its not a fix. To pass it off as one would be malpractice for a real Engineer.
"Engineer" conveys image of middle class white-collar job with relatively high status, good education and responsibilities. That word now used for everyone doing programming related jobs inside office space for no good reason.
I think the word "tehnician" should be used to describe most grey-collar ICT jobs, including most programmers. Their responsibility and scope of their work is limited. Many programming jobs are closer to blue collar factory level mechanic than engineer.
Yup. The reality of this industry is that most programmers are closer to the handymen you call to fix your faucet or lay floor tiles than to actual engineers. And we all have first-hand or second-hand horror stories related to handymen.
I've come to the conclusion recently that we're all just barbers, bloodletting and burring holes. Until we have proper certifiable and reproducible competency, we'll never become surgeons.
And yet the US has consistently been in the vanguard of software development worldwide. Do we need surgeons? Do you need a four year postgraduate degree and three more years of apprenticeship before slinging together a web app? Certainly the industry has its problems, and some applications demand more rigor, but I don’t see this as much of a general solution.
The market didn't demand a high survival rate or efficacy for barbering either, but do you think that the (conscious or unconscious) that software engineers are some sort of magical wizards will last forever?
There may be explanations beyond wizardry - e.g. maybe barbers were historically often self-employed, small business owners who realized they stood to gain from creating barriers to entry such as licensing requirements.
Yup, in the UK it bugs me when I keep meeting people who introduce themselves as an 'engineer'. When I ask them if they do mechanical or civil engineering, then I usually get to say "ah, so you're a programmer, just like me".
One thing I did sometimes like about being in Austria is the obsession with academic titles they have there... if somebody is a qualified engineer, they invariably use "Ing" as a title in place of "Mr" or "Ms"; people with two doctorates (not uncommon) really do call themselves Dr Dr So-and-so.
Austrian here. "Ing" (Ingenieur, so basically Engineer) is a protected title (not an academic one, though) but to be honest, it does not signal status or qualification to me. The actual academic titles are worth more.
Still doesn’t really mean anything. Due to a loophole in major assignment I can legally call myself Ing., but what I’ve really studied is more or less computer science.
What is qualified in Austria? In the USA, in most states, you must be licensed to have engineering in your business name. Your name is usually suffixed with PE.
Yes. I don’t think there is even a PE licensing path for software, so even with a CS degree you could never become a licensed engineer. You have to work under a PE for 5 years, then take the PE exam, then CE credits.
I have an MSEE, but can’t put engineer in my title. Though Ing. sounds kind of cool.
It bothers me to no end when I hear programmers here in Ontario call themselves engineers when they neither graduated from an engineering program nor completed their PEO exam. It's very much illegal, and the PEO _will_ go after companies that allow their employees to title themselves as engineers.
And most states in the US too. Just rarely enforced. Only a licensed PE can do business as an “Engineer”. (For example have a company with the word Engineer or Engineering in the name).
I believe the city of Portland OR recently sued a Danish man for "practicing engineering without a license". They were upset at him for generating strong proof of shortened yellow light times at intersections with red light cameras. He had a degree (and I believe a certificate) from Europe, but it wasn't recognized by the Authorities.
Protected term in Canada as well, but I don't know the last time anyone that was used against in any level of court. During my education there was a case study of some guy digging basements calling himself an engineer when he wasn't but that's about it.
If you're a company the size of Cisco, and you have two widely publicised vulnerabilities like this, then this sort of failure is _always_ management's responsibility.
That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers, but it can take other shapes too. It might be poor oversight and QA processes, or a failure to hire and/or assign competent engineers for the task.
> That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers
This is the narrative that always shows up in HN. Big bad management vs the hero engineer. It makes us feel good about ourselves while also absolving us of all responsibility. Its a bullshit narrative.
You can always quit. Or force them to fire you for refusing to implement a non fix.
In reality though I doubt this narrative even occurred. Some incompetent engineer likely proposed this fix thinking that it was actually a fix.
Edit: I see I've been downvoted for this comment. If we were real engineers working on things like cars and bridges we'd actually be held accountable. Take some pride in your work people, this is one of the most in demand professions in today's economy.
At a large organisation I worked at years ago we used to call it Coder Shopping - when a manager would do the rounds trying to find someone who would say "yes" in a situation like this.
Invariably there would be someone who would, and if you were the person who had initially refused you probably wouldn't even be aware that it had happened until the bug reports came in...
If there were actual accountability, software engineers would have a much better lever against management. For some software this is case. If you write safety critical software you can be held personally responsible for accidents. In those industries engineer pushback is much more effective.
My background is safety-critical and -- speaking from first-hand experience -- it was truly incredible how much push-back we'd get when raising safety issues.
System tick timer rolled over after 2^32 1ms clock ticks (a little over seven weeks) and the software mishandled it by rebooting and losing control of the process. It wasn't until QA got involved (it turned out they saw it on a long-term test) that we were cleared to actually fix it. It was a two-character change in an inline function!
In some places "cover it up, don't discuss it in email" is the natural response. It really depends on the management and the organisational culture. You can have management who support the engineering team, or you can have management who micromanage and overrule and end up turning a two-day hotfix cycle into ten months of "try soldering a resistor to every single pin on the PCB".
You might be surprised, but if you move to Europe, it is actually so! Move to Berlin of Frankfurt. There is always work for Software Developer here, it is hard not to get one and you get spammed by recruiters all the time.
You don't have to take shit from management, you can just go elsewhere. Everything is also very near.
Well as a European I find the "uproot your life for a better job" mentality completely alien. Short of a war I'm not sure what would make me leave the place I call home.
I like your spirit, but there's another conclusion we can draw: very few of us outside of a few special areas in embedded are "engineers". We tinker with bits and bytes to make business systems. Sometimes we bump up against some general engineering fundamentals. But we're just technicians meeting business goals.
Yep, and the same can be said for “real engineers”. Most EEs, civil engineers, etc work on boring stuff as well with negligible risk of doing any harm.
You suggest it was just one lone engineer being incompetent. Then you respond to downvotes with the non-sequitur "take some pride in your work" - are you suggesting pride is a fix for incompetence? What?
It's a problem of someone coming from a very incomplete view of the situation, and then issuing judgments and advice without doing any research into the actual situation on the ground (armchair warriors), which is irresponsible.
Yes, there is a possibility that they are in fact right, but they'll probably be right for the wrong reasons. It's far more likely that they'll just be plain wrong (and arrogantly so). Even a stopped clock is right twice a day.
Let me know when pride is accepted as currency to pay bills or when insubordination due to personal pride is seen as a desirable quality in a candidate. At the very least, how much are you willing to pay a person for not implementing the fix you disagree with (including engaging in a legal contract to pay them if they lose their employment due to not implementing the fix). From my past experiences with others, moral high grounds rarely survive personal stake.
Pleas tell me in what country is it an issue for a software developer to make money?
I mean in this situation, if I were forced to implement such fix, I would do it, but would then quit the next month. Because if I were to stay, not only would I get worse professionally, but I would feel dirty for not doing my work properly. The companies that accept such fixes are more often than not like a factory and treat you as gear in a machine. And also have an ugly software which make good engineers quit.
USA. Not everyone has the economic mobility to just quit at any time. I know one such individual who I have been encouraging to get a new job because his current employment leaves him working a second job on weekends.
One could suggest to such people to display enough financial discipline to save up a rainy day fund, but this is ignoring all the procedures in place designed specifically for depressing wages.
1) is obviously false, since an engineer could be competent and evil. Or competent and under duress - needing to keep their job to keep their visa, family health insurance, etc. Or competent but misled - "Implement the suggested fix to show how easy it is to work around" "OK, here it is" "Now work on something else", while it ships.
2) Why should you? Unless you're willing to be homeless and leave society, you're not going to find a way to live in a capitalist society which is pure idealism and no compromises, and you can't effect change in a system from without, only from within. Quit, and someone else will implement this fix - you feel good, which is worthless, and the fix ships, which is bad, and Cisco doesn't change, which is bad. Pick your battles and you could use this result in future - to build other people's trust in your recommendations, to widen the review process before future fixes are agreed upon, to have a proven record of the cost of doing things twice.
Absolutely: Cisco made this happen, not Engineer Alice. If your boss asks you to do something within my ethical boundaries and you have 5 tired co-workers in favour of shipping it, eventually you'll break and say "fine". It doesn't make you responsible or the one that pulled the trigger.
The fact that this fix made it to prod tells you that something is deeply deeply broken with cisco’s processes that goes well beyond any individual’s responsibility.
Exactly. It's not like somebody just did this like in a three people startup. There are code reviews and quality management and many more, and this lousy "fix" has passed all those stages and instances. If nobody declined this, there's a serious problem within Cisco.
I honestly envy the sort of person who can post something like this. Not liking nodejs is fine, but to extrapolate from "I don't like nodejs" to "Thousands of engineers at companies like Google are wrong and I am right" must require such a level of myopic, ignorant self-belief that is completely alien to me. I just find myself respecting other people's work too much, even if I don't like it.
NPM is a trainwreck in many respects, and nodejs is terrible in many ways, but that doesn't mean developers can't use them both productively. It requires effort. You can't just push responsibility for what your app is doing under the hood to the ecosystem of library code you pull in.
However, this is true for literally every language we code in.
That's an appeal to authority. It's also worth noting that nodejs is not a Google project and it is barely used internally at all. Only for small toy projects. Google uses python for scripting, or java for real servers.
It is, but it's also shorthand for "people who are likely to be good based on the fact they've been through a rigorous hiring process designed to filter out the worst engineers". Sometimes, maybe, you should let the principle of charity win over the rigorous rhetorical logic you can use to dismiss an argument because it broke The Rules you learned on LessWrong.
It is bad management to hire incompetent engineers.
It is bad management to fire competent engineers that push back.
It is bad management to make employees feel as though they would be fired for standing up in any way to management.
Take away the rhetorical shortcuts, and responsibility still always falls at the feet of the ones that have the power to make the decision. As long as engineers are hierarchically bound to obey managers upon pain of sudden and unplanned unemployment, the managers are ultimately responsible for everything the company does. If the company does not have tenured engineers that have stop work authority, the managers are fully to blame. And if they do, management still might be partially at fault.
We have this headcanon where a competent engineer could stand up, sweep away the slide deck on screen with a wave of the hand, and offer ironclad mathematical proof that the quick fix is no fix at all, and that the long, painful, but correct fix is what is Right for the Company (tm). The manager would then do the slow clap and tell that engineer to fix it right this time. It is a fantasy that is a great comfort to those of us doing maintenance work on a massive crock of shit, making the payments on technical debt that we are expressly forbidden from making any attempts to retire. But it is a fantasy. A real manager would cut the engineer off in mid-proof and say that the quick fix is happening, and that would be the end of the story.
The only way for an engineer that is "at will" and without a contract to push back is by falling on their own sword. They can quit, at great cost to themselves, to create a very minor inconvenience to the company, who will end up doing what it wanted anyway.
Being in this situation myself, you're damned right I have no responsibility for the crap code that goes out, because I have almost no autonomy in the manner in which I do my job. I cannot work outside the scope of the maintenance tickets I am assigned. And my company rents me out by the hour to the customer, so it has negative incentive to allow me to build up infrastructural capital that would allow me to do the same work in less time. I will blame management all day long, and then a bit more on evenings and weekends.
There are plenty of poor engineers, but this “fix” has to go through a process defined and enforced by managers. In a good process, multiple engineers would review the code, and if the office culture is good, then objections will be brought up and listened to, and if a fix is insufficient, management will understand and assign resources to make it right. None of those things depend on any one engineer or manager. They’re a culture issue. But the only people with the power to fix a broken culture is management. Engineer Alice doesn’t have that power herself.
In large orgs there would also be some kind of security team which would have input on the CVEs. Cisco does have them. It sounds like they were not involved at all here for some reason.
The only way, and I mean the only way, this gets blocked is because a manager (not a dev) assesses the risk of improper fixing to be higher than the reward of not doing a shit ton of work justifies.
A significant portion of that is making sure the manager is fully cognisant of what the quick fix means. Alice's job is to ensure that happens. The managers job is to advocate to his bosses that this has to be done and is a net win to do it properly.
It is as likely manager person failed in advocacy as it is she failed in information transfer, but there are plenty of gradients here.
Yes, but this is software engineering. Where there is no such thing as malpractice, and where managers can not be prosecuted for criminal negligence even when their overriding of the engineers professional position gets dozens of people killed. In this delightful world, managers rule the roost and engineers do not have the ability to refuse to do anything.
If management let this important security fix get executed and deployed without a competent engineer in the loop, that's a horrible mistake in itself, but it's far less likely than management choosing this alternative based on accurate information from engineers.
I've been Engineer Alice before (not with a security issue, but a pretty bad systems issue). In case anyone is ever in that situation, here are some more productive replacements for that existential sigh (in that they sometimes work). Choose the one most suited to manager's biases. The cliches are important: think of it as giving manager an easy way to explain it to their manager.
> Yes, but that would only help with this specific case. The proper fix would ensure that we don't see a similar issue tomorrow. We don't want to prepare for yesterday's battle.
> Yes, but the extra regex rule would be new code, which would need its own maintenance in perpetuity. Fixing it properly would let us remove the existing code. Removing code is the most efficient thing an engineer can do, and will save man-hours in the long run.
> Yes, but the extra regex rule has worse runtime performance than the proper fix. [Performance is our big competitive advantage | We don't want to fall behind in performance | We are already behind in performance]. We don't want to create a customer-visible performance regression.
No! The answer is an unequivocal “No“, without any „but“s or anything like that. The fix does not fix the problem, it is not even a fix, just a wrong code change that sets out to do something but does not achieve it.
Answering „yes“ is a lie here. This is different from a fix that fixes the problem in an ugly way, where „yes, but...“ is applicable.
In this specific case (I have never encountered a proposal quite this bad), the correct answer is probably "Give me the proof of concept code and about ten minutes, and I'll give you a version that works despite the patch." You're right about that. "No" is likely to bog down the discussion with "but Bob says it will work", and it's better to skip to the end.
I'm envisioning more of a middle-ground case, where the code is still obviously wrong but it's harder to cleanly demonstrate why.
Ten minutes? How long does it take to type a user agent string into the curl command line?
In my case, a user agent string pretending to be Firefox is already part of my .curlrc, so it's no time at all, just grab the PoC code and fire away. (I've got commented out user-agent lines for a bunch of other browsers that I can uncomment as the need or mood strikes me.)
You must live in a very nice, ideal world, where simply saying "no, that's not the right way to do it" will convince managers to ignore the pressures placed on them to, at times, value speed over correctness.
What they are saying is that this isn't just "the wrong way to do it". It doesn't actually fix the problem it's supposed to fix, therefore it's actually wrong to say it's a functioning patch.
Apparently I do? Not sure what you want to hear here, but management does listen to me and my engineering peers. Sometimes things are a bit gray and fuzzy, but in this case here where they definitely are not, I am absolutely certain that there would be no overriding of our arguments.
That's awesome, actually. For me, it's not as bad as Cisco seems, but there are definitely times I get overruled and have to take shortcuts that I'm not always comfortable taking.
No, it won't actually fix anything, it'll only look like it did for most trivial cases, and 6 hours after releasing it our company will be on the frontpage of Reddit.
I actually said something like that during a meeting. "Yes, you can do it that way, but if someone finds out we will be in the news." This argument worked surprisingly well.
Or you can just do the minimum just like the manager and cruise on the ship until it sinks while surveying for another ship to jump on at the first opportunity.
> Manager: And we can just prevent it with an extra regex rule in the front end code?
> Engineer Alice: ... Yes. irrepressibly existential sigh
The answer is "no", though. This exchange sounds like the decision was between "fixing it cleanly" and "fixing it with a hack", which is roughly equivalent with fixing the problem at the expense of technical debt. But this change does not fix the problem at all.
Engineer Alice is the only person incompetent at their job in this conversation. If "irrepressibly existential sigh" is how you argue security with your bosses then maybe you're not senior enough to be in meetings like these.
The Manager and QA Engineers here depend on the expertise of the engineers. If the engineers fail to communicate key details of the situation, then that's on them. Sure, the boss is still at fault for depending on a shitty engineer for decision making, but that's all.
If you answer with the existential sigh, you’ve had at least 20 conversations with a similar outcome before. Nobody arrives at a new company that jaded.
I see your point and I suspect that that's what the author meant.
At the same time, I know lots of, well, arrogant nerds who get into "existential sigh" mode as soon as someone with less technical skill than them says something stupid. That's more how I read it.
> The Manager and QA Engineers here depend on the expertise of the engineers.
Both should be sufficiently competent and knowledgeable to know this is a terrible idea, especially in the long run. Otherwise they do not understand the task they have been given. So no, Alice is not the only incompetent one here.
1. Explain that the 'hack' does not actually fix the problem. This is absolutely the engineers responsibility. In your example Alice does not say that. This has always been sufficient.
2. I make a habit of sending emails or keeping minutes confirming such meetings and the details of who said what.
This is about all you can do. But you're right. You have to explain it and document it: "It's trivial to change User-Agent, and any decent hacker would probably not be using 'curl' as their User-Agent. We can look at logs from past events to validate this assumption."
Bob is probably an engineer with twenty years experience, who's been thoroughly ground down by management and peer power-plays and has realised that the best solution is to keep your head below the parapets!
Manager: Bob get to work on the quick fix that will mitigate some of the attacks based on the published exploit to go out within a day, Alice get to work on the correct fix to address the underlying issue to go out in a few weeks when it's fully tested and ready.
Manager: Alice, yeah, I know I said we'd carve out time this sprint for you to work on the long-term fix. But this other hot feature request came in, and there's potentially a lot of money riding on getting it done quickly. So can you put off the long-term fix, maybe a week? Two tops. You can just put the ticket.. yeah that one.. back on the backlog. There you go. Thanks so much!
Then someone realises one of their enterprise clients is using curl for a critical system, and said patch has completely broken it. Or more likely, they realise after customer/company calls complaining about how their systems are now busted...
i'll raise you a case where a team of very well paid senior engineers/architects and PMs dismissed a remote execution vulnerability down to a very low "some next release" priority on the grounds that "notepad.exe doesn't seem to do any damage" - as you may have guessed the vulnerability PoC used notepad.exe . After seeing that with my own eyes, this Cisco curl is just "meh" for me :)
I've had some visibility to big networking equipment manufacturers... the security or even stability on their web interfaces is terrible. It's like a strange rule of thumb for all the networking equipment manufacturers. If you have choice, disable the web interface.
I guess the biggest problem is that the mitigation isn't future-proof. Someday, there might be other command-line tools besides cURL that can be used for this kind of attack.
A better fix would be to just drop all incoming IPv4 packets that have the "evil" bit set. (https://www.ietf.org/rfc/rfc3514.txt) Extending this protection to IPv6 is left as an exercise for the reader.
Well if you’d read the RFC it’s obvious it’s your routers responsibility to flag those bits as evil. And if you’re using heuristic intrusion detection systems you need to do flag each bit as evil 50% of the time. It’s good we have such amazing protections in place.
Add an x-Lawful-Warrant parameter to provide a warrant. It can’t be read by anything but the best AI technology and is included in every HTTP request. No warrant, no response!
I couldn't agree more. The joke construction was a bit weak though, when you miss a good part of the audience. Add something like, "I mean, could you imagine the chaos it would cause if IE told websites it was really Mozilla?" and you demonstrate mastery of the subject matter, which should be enough to let other experts know you were facetious rather than ignorant. Unless you have timing issues... or need the comedian's plausible deniability.
I don't find that as funny, I think making the sarcasm more obvious makes the joke less funny as well. I prefer to take my chances than to go overboard and neuter the joke.
Yeah but it's not Reddit, it's HN. You have to know your audience, read the crowd. On Reddit it's 90% sarcasm so there's no fixing it. Here it's the reverse and people take things seriously without a tell. You just have to bury the tell in another joke or it will ruin the funny.
Yes, very well, as first action of the Joke Approval Committee, I propose that this august body instantiate the Recursed Approval Subcommittee, whose role will be to instantiate the Recursed Approval Subcommittee.
As soon as the subcommittees hit their break condition (0), they'll break for coffee and donuts, and I can submit my "joke" to the JAC. And I don't make a habit of repeatedly posting the same joke, but I cannot make the promise in this case.
I've never posted sarcasm online without at lease one person (that guy) who takes it seriously and decides to reply. But I think it's worth it because the people who get it will get it. If a small group doesn't then who cares (if everyone doesn't then you did a bad job constructing a joke).
I would call it hacking.. you sometimes run into those websites with 50 different opacity bits for all their web app things, and you don’t want to deal with all that rubbish so you set a “opacity: 1 !important” and the whole website looks off but it’s finally useable. I’d call that a hack.
Or also when you run into those websites designed on a $2000 screen that thinks it’s tres moderne to use 808080 text, so you “hack” it into readable text.
Actually you can use -A to change the user agent in Curl itself. For example:
curl -A "YourNewUserAgentString” http://url.com
Furthermore, there are free plugins or add-ons for both Chrome and Firefox that let you change the user agent and say you’re using IE, a mobile browser, a google bot, or anything you want the user agent string to say.
Most programmers should be able to write a small script using Python’s urllib or urllib2 to do a basic PUT or GET request with any User Agent. Heck, just type “change user agent python urllib” into google, click on the first stack overflow link, and copy/paste the answer. You can use Go, ruby, java, nodeJS, or PHP instead as well.
It does seem so obviously simple to sidestep that it makes me wonder if they pushed it because they had an active attack they believed it would mitigate. Were they concerned that the attack was already baked into some automated script in the wild and this could perhaps at least trip up the script long enough for them to engineer a real fix?
Scripted attacks can of course be modified too, but it still takes more than zero human effort to do it.
Curl can, easily. Curl is not just a client to get a file and pipe it to other programs, it's a diagnostic tool, and it has all sorts of parameters to help it simmulate all kinds of interaction with a webserver, including the -A parameter to change the user-agent, this is why this 'fix' is so stupid.
I am not laughing one bit let me tell you. Honesty in user agents is one of the last few defenses we have against hackers running rampant on the internet and turning it into some lawless wasteland where people don't even answer truthfully about their A/S/L.
How? Do you think developers would be so crazy as to add some sort of "--user-agent" option to software like curl that would cause arbitrary strings to be presented as the user agent? Why would somebody write software to do this, just go on the internet and tell lies?
I thought all this security nonsense ended in the 90s when they introduced the IS-MALICIOUS-REQUEST header. It must be, because I haven't seen any web traffic declaring itself malicious since then.
> Therefore, we obviously need to patch echo (and all echo shell builtins) to refuse to output strings containing "GET", "POST", "HTTP", or "Host:".
Unfortunately it is also possible to do this using file redirection or to write a new program that will make a TCP connection and send arbitrary data through it, making it necessary to do the same for all editors, compilers and interpreters.
That sounds like a lot of work though (there are many such programs), so maybe we ought to just patch the kernel to prohibit any of those strings from being written to a file descriptor. There can't be that many false positives.
But users have fingers, which can be used to disable or subvert said "features". Thus, those fingers are just gonna have to go. They're a security risk, and security risks are against the regulations.
Ooh boy I'm imagining some bureaucratic version of an operating system that uses this sort of security up and down the entire stack like some rube Goldbergian contraption to whack a mole a particular vulnerability. Like an operating system designed by doctor Seuss.
While that's probably technically feasible, that sounds like something only a terrorist would do anyway. All the more reason to ban the entire program, I say!
And it is also very good that regexes are simple, unambiguous, easy to debug and their engines are heavily scrutinized that no code execution/undefined behavior is possible.
This is a symptom of the rot in their management, and probably also a sign that they have hired too many incompetents.
It probably also is a sign of the current age. After the recovery from the IT-bubble programming got really hot. Thus: Too many of the new programmers wants to be programmers because it pays well - not because they love their craft. So therefore we have a bunch of well-paid but uninterested people seeking jobs at prestigious companies.
Culture matters. I want my socially maladapt terminal junkies back plz.
Some time ago I went through the list of all the major router manufacturers and rated them on 1) security, and 2) long term usability, and 3) culture.
My conclusion was that I would buy my infrastructure from Allied Telesis. It's pretty much a Japanese version of Cisco, but it's still healthy.
Ubiquity was number 2. I refrain from buying from them only because of their glossy UI.
Mikrotik was on that list. Until I saw how horrible their winbox protocol was. And their implementation of SMB.. I must assume there are still plenty of unknown RCEs there.
I had several Allied Telesis switches at a previous job. The hardware was fine but resetting the configuration password was a remarkably painful process.
They have had several remote code execution vulnerabilities lately (summer 2018). While they were very quick to patch them they did not notify their customers in any way. There was nothing on their website that said anything about the urgency.
Instead of reusing functionality that exists in the router already (ssh?), the authentication for winbox is something they built themselves. It was in the winbox auth that the main security flaw was. It just looked really bad to me.
The winbox client also downloads and runs any DLL that is sent by the winbox server. The winbox client has a windows certificate so all it's code is trusted. So own the router and you get the admins workstation too.
It just feels like maybe they hired some random guy without much appreciation for security for doing winbox.
The SMB server also had a rce a while ago.
That said, I guess that if you disable winbox and stuff that should not face the internet, you are probably safe?
WinBox is not that simple and cannot be replaced with SSH. WinBox works on Ethernet level, so one can connect to router by MAC address and recover when IP level configuration is invalid.
Culture matters. I want my socially maladapt terminal junkies back plz.
I know some fastidiously groomed VSCode front-end hipsters whose entire bodies involuntarily tense up if you suggest a lazy technical workaround to them. It's funny and makes me feel good about the future.
But I get you; I agree completely. You do need people who really care. They're just not as easy to stereotype any more (if they ever were.)
Your first point is just blabber. Cisco is a large company and this is a $100 product that they don't really care much about. It's the type of product that is produced at the cheapest possible cost, by interns. The way this product has been developed and managed is completely intentional and not a sign of mismanagement. Come back in 3 and then 6 months and let's see what effect this has had on their revenue and market cap, and then you can start talking about mismanagement.
As a sign of the quality of the current median developer, I think you are spot on. But I don't know that it's worth griping about. That is the world in which we live ... so live in it. In fact, all the better for those of us (if I may dare to put myself in a more elite group) that are highly skilled ... we don't have to work on some me-too unimportant SOHO router, at correspondingly low me-too unimportant wages.
This "fix" seriously hurts Cisco's credibility. How can you trust their products?
Perhaps they are thinking that noone gives a damn anyway after no less than five backdoors² were found in their products in 2018 alone? Just incredible.
Cisco have a loong history of things like the. I mean go back to 2005 when they actually sent people to a security conference to rip pages out of the handouts that mentioned a security vulnerability in their products (https://www.computerworld.com/article/2482483/cisco-s-blunde...)
That (and other bad security responses) have not really hurt them, so at this stage, this won't make a lot of difference, I'd expect.
Cisco is in the business of selling big, black, expensive boxes that have a lot of security badges and fancy icons. People who buy such boxes don't care if they actually work, they want a big box so that they can claim they "invested in security".
Actually, they want the Cisco box because it has those 'badges' and that then allows them to check off all the check-boxes on the semi-annual security compliance report paperwork they have to file with some other department.
But you are right, they don't care if the box is secure, they just care that they have a CYA that allows them to "check off their own boxes" on their reports.
Juniper has offices and partners all over EU, for example in Paris [0]. Similar to Extreme [1].
I don't know what problems you encountered in France, but I can assure you both of them are very well represented across the whole EU, even less developed parts of it.
The reason managers pick Cisco is mostly that "nobody gets fired for picking Cisco" imho.
I believe this change was rolled back, since my User Agent is "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.2 Safari/605.1.15".
They would be a lot more useful if they were more straighforward and honest.
Just give me "Chrome 68" instead of "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
So what. Removing one of the largest sources entropy available for fingerprinting users is important. We shouldn't maintain the terrible long-term effects[1] of tracking everything just to help you have an easier time debugging.
Does it really provide that much entropy? Maybe back in the days when browsers didn’t auto-update, but intuitively it should only provide about four bits at most (os: Android iOS macOS windows, browser: chrome safari edge ie ff) except for a few rare users who don’t stay up to date or choose a weird browser (and those folks probably are a little confused about how privacy works).
i ran this on my work laptop (windows 10 / firefox - mostly default settings). the "details" tab shows a full breakdown of each fingerprinting component with a "similarity ratio" (percentage of fingerprints that share the same value).
every entry is 20% or greater, most of them around 50% except for timezone (4%), canvas (1%), and user agent (0.12%).
so, yeah, user agents are still the largest source of entropy by a wide margin for fingerprinting scripts.
There are better solutions available than hanging functionality on unreliable vestigial bits and pieces that shouldn't be there anyway which, because others abuse the functionality, you can't trust to be correct for debugging purposes.
99% of the traffic has the correct user-agent which is useful in tracking down issues that are browser specific. The other 1% will just get ignored as noise. And it's not like we can't tell what type of browser they are using with browser specific objects that we can pull from javascript. We just can't infer the version of the browser which is critical for debugging issues.
I use a randomized user agent and every once in a while get rejected by a site that has decided my browser can't be supported. Usually when I'm on some Safari version. I doubt much effort is being done to identify fake user agents since there are better fruit to pick in the fingerprinting game.
I've had to debug around a short-lived bug in the Firefox rendering engine that only showed up in the FF rendering engine before. The site itself? Standards-compliant; there was flexibility in the standard (ironically, for flexboxes ;) ) about performance constraints and our site just fell down Gecko's slowness tree and hit every branch on the way.
Given the engines are allowed to vary, this experience is inevitable from time to time. The user-agent is really valuable in helping us figure out when that's what we're seeing.
> I said "mobile", not "cleaner". Sizes and aspect ratios are different on phones than PCs.
User agent doesn't tell you size or aspect ratio.
Even if you know I'm running Chrome on Android on a Galaxy Note 8, you don't know the size (physical or pixel) or aspect ratio of the viewport, because: (1) I may or may not be viewing in side by side windows, (2) the screen can be in any of three resolutions, (3) the screen can be portrait or landscape, (4) the browser may or may not be set to use the full space on the long dimension of the screen.
There is presumably some way, at least hypothetically, to fix the mess that User-Agent has become. (Everybody is 'Mozilla' now! And yet somehow a unique snowflake 'Mozilla' that lets you be tracked...)
But it's not quite that. It's not that a device is "mobile" instead of "stationary" that matters. It's maybe that it's a touch screen. It's maybe that it's a small screen. (With an especially uneven aspect ratio?) Maybe that it's handheld. It's maybe the combination of all of those things. But it's not really that it's "mobile" exactly. And there are, and possibly will be increasing proportions of, devices that have some but not all of these qualities we package together in our brains as 'typical mobile'
All of those small differences of mobile devices are better served by media queries I personally think. You could replace "mobile" in my example by "small-screen" if you prefer.
Agreed. Media queries need to be improved though. There are no good media queries to detect "touch screen" and/or "no keyboard or mouse/pointer, only touch screen", only JS hacks that attempt it. This can matter. But yes, improving media query (or input method query?) APIs for both CSS and JS is, I agree, the correct solution to User-Agent madness. It's just not entirely straightforward or obvious or easy, and as always getting all browsers on the same page is an issue.
For which screens ? Why would that be helpful in any case, given the browser window is not the size of any of the screens ? I'm not sure what you would gain from getting this information.
We sometimes render different HTML to handle browser and device behavioural variations, including FOUT-management variations in how they load assets that therefore can't be handled in JS even if we wanted to. Historically we also redirected to a pure HTML/CSS version of our tools when IE9 or earlier was detected.
Don't blame the manager, the PO, the CEO. This is ABSURD engineering incompetence. The fellow that did that _fix_ probably had no idea how to properly solve the issue.
If it is then it's management's responsibility for allowing that incompetence to exist. These sort of issues all come from the culture which is driven from the top.
I posted something related to this a couple weeks ago.
Chase banned my web client because its not Windows/OSX.
It works fine if I change my user agent...
FWIW the RV series is the ex-linksys stuff so it should all be thrown in the trash anyways.
Okay, so the fix is bad. Now, you have to wonder how this "fix" made it through code review, QA, release... You can blame the engineer. Perhaps they were rushed, inexperienced, or both, but this is a failure on all levels.
How do you know it wasn't fixed by the offshore maintenance team with an offshore manager controlled by a VP wanting a quick promotion for doing things fast? That would be where I work the likely scenario.
That's hilariously reasonable if someone didn't read the news latetly and suddenly got a 403 while executing this payload using curl would probably quit and be like "meh it got patched". Look on the bright side guys.
That's not source code, that's just a config file embedded into the firmware. For most firmware, it's a large binary file that often has different sections containing stuff like an OS kernel at one spot, a compressed archive somewhere else, etc. For the Cisco firmware in question in the middle of the firmware is a compressed CPIO archive (think of it like a zip file) that contains the root file system. The router is actually a small linux computer, inside that archive they have a config file for nginx (a web server) that tells nginx to return an error if it sees the user agent string for curl. There's no programming required here, just a trivial configuration change that masks the actual issue.
Without actually knowing the truth, my guess is outsourcing: Cisco thought it would be a great idea to save some cash on technical staff so they now do a lot of the grunt work through company X in country Y.
Source says they also did some input sanitizing along with blocking curl, and they had to make a new PoC to get around that. If I'm reading that right then this isn't really an issue, nothing wrong with defense in depth.
Edit:
>The update adds several filters to handle
single quotes in user input. However, these filters can be evaded by
specially crafted inputs. By providing the following string for the
certificate's common name, a "ping" command can be injected:
Title is misleading, implying the only patch was blacklisting curl.
and POST instead of GET (and kurl as the UA, of course).
Does their fix specifically check for injection starting with `a'` ? And only works for GET requests? Mind-boggling...
Edit: The new exploit also targets https instead of http. I would've said that surely that would not make a difference, but given what's already happened I'm not sure.
The real defense is the attacker needs to access and authenticate with the router's web interface. A more honest patch would be to legitimize the bug as a new feature since it must be too amateur-hour over there to actually address any webshit security issues. "Dear Admin, here's a textarea to run arbitrary commands as root, don't hurt yourself!"
Yes it can. If you see a product that claims to be secure, with multiple layers of security, certain exceptionally-fragile measures should decrease how much you believe that claim. Layers of security have to at least be mildly effective to count as layers.
If you're fighting an active attack you can stall by filtering on some arbitrary parameter unrelated to the actual problem. For anything that's supposed to last more than an hour, it's worse than useless. It makes your system more complex for no security benefit. An idea like that should never make it into a product release.
