
Huawei savaged by Brit code review board over poor dev practices - jslhs
https://www.theregister.co.uk/2019/03/28/hcsec_huawei_oversight_board_savaging_annual_report/
======
undecisive
The real point is at the end:

> Finding general issues is a good thing, but other vendors are not subject to
> this level of scrutiny. We have no real (at least not this in depth)
> assurance that products from rival vendors are more secure."

Pretty sure companies like Cisco (who have had recent high-profile security
fails) would not want the same code review.

Until they do, and someone can do a comparative analysis of the two sets of
codebases, there are no useful conclusions to be taken by decision makers.

The only value here is for firmware developers - learn from these mistakes.
Hopefully GCHQ is coming for your code next.

~~~
StudentStuff
Cisco has gone through in depth code review for many of their products, as
they are often the vendor of choice for security critical applications. Look
at the desk phones in the upper echelons of the federal gov't and note how the
"secure" systems are all Cisco with metal cages on the back to prevent signal
leakage/sidechannel attacks.

That being said, Cisco is a bag of dicks when it comes to their support
contracts, pricing, and any of their consumer rubbish.

Juniper and other 2nd tier hardware vendors are a bigger threat than Cisco
IMO, their gear is much more common, and much more vulnerable (JunOS is a pile
of security vulns). None of them take proper software development seriously,
and the audits they've undergone are often just to rubber stamp their
products.

Frankly, none of these companies should be rolling their own "firmware", as
they've proven that they won't reliably update as upstream (usually OpenWRT or
Debian) push out security critical updates. Downgrading them to a role whereby
their secret sauce is just an extra APT repo/extra metapackage from opkg is
more than sufficient for what these companies need, while preventing them from
accidentally or purposefully blocking updates.

~~~
pleasecalllater
Yea, an in depth code review. Like the last Cisco router bug fixed by banning
access if user-agent contains 'curl'?

~~~
StudentStuff
I referenced this in my prior comment, calling out their consumer rubbish
which they don't bother to maintain.

------
mikekchar
I once worked for a defence contractor (although I did not directly work on
any defence contracts). I didn't have a security clearance, but they wanted me
to do a code review of some software (which seemed to be some kind of
communications protocol). After reviewing it for some time with a colleague we
wrote our findings as "We don't understand what this code was supposed to do,
but we highly doubt that it works correctly" along with comments of serious
problems on almost every line. My colleague was Chinese and could only work on
his visa with that specific company. The manager threatened to fire him unless
he retracted the review. After that, they shipped the code. Apparently it
didn't work at all and they all got fired. I got a raise (but quit about 3
months afterwards as it was a horrible, horrible place to work).

I can well imagine what it's like to work on the code described in that
document ;-)

------
abraae
Quite the dog's breakfast:

> In the first version of the software, there were 70 full copies of 4
> different OpenSSL versions, ranging from 0.9.8 to 1.0.2k (including one from
> a vendor SDK) with partial copies of 14 versions, ranging from 0.9.7d to
> 1.0.2k, those partial copies numbering 304. Fragments of 10 versions,
> ranging from 0.9.6 to 1.0.2k, were also found across the codebase, with
> these normally being small sets of files that had been copied to import some
> particular functionality."

~~~
wjnc
Can someone explain to me how you can understand the culture in a team of
programmers that leads to this being acceptable? Just a few small guidelines
could help alleviate things like this, right? Modularization and a tiny
sprinkle of code review and this is solved. Perhaps some (big) initial effort
to clean up if you've started with code bases of different individuals with
the goal of integration.

~~~
PhantomGremlin
_the culture in a team of programmers_

The "culture" behind this is endemic to industry and isn't anything unique to
Huawei.

People have already commented on this not being a "team". Let me focus on
"programmers".

What probably happened is that many of Huawei's programmers are EEs who
happened to take a programming course or two in college.

But learning to think and behave like a good programmer is much more than
that. It requires a certain mindset. It may require a certain amount of
avocation and enthusiasm, beyond what is taught in an introductory programming
course. It requires one to grok good software practices.

Most non-CS majors in college (USA or China or whatever country) probably
don't get much of that cultural indoctrination during the few programming
courses they take. The engineer wants to pass/ace the programming course.
Learning the aesthetics or nuances isn't a priority.

Programming is just too easy. Type a few lines into a file, compile, debug
syntax errors, tweak until you get something that looks right. There's no
rigor involved. No style. No aesthetic.

Don Knuth called it "The Art of Computer Programming". I submit that many
engineers don't think of programming as an art. It's just a means to an end.

E.g. here's an example from a large codebase that was extensively reviewed.
The context was Toyota's unintended acceleration problem of about 15 years
ago. The design review found things like: _Other egregious deviations from
standard practice were the number of global variables in the system. (A
variable is a location in memory that has a number in it. A global variable is
any piece of software anywhere in the system can get to that number and read
it or write it.) The academic standard is zero. Toyota had more than 10,000
global variables._ [1]

None of this will ever change until and unless a company's top level
management understands the problem and mandates change.

Additionally, there's always an incredible push to bring new products to
market. That's 1000x more important than it is to follow best software
practices.

[1] [http://www.safetyresearch.net/blog/articles/toyota-
unintende...](http://www.safetyresearch.net/blog/articles/toyota-unintended-
acceleration-and-big-bowl-%E2%80%9Cspaghetti%E2%80%9D-code)

~~~
sannee
> What probably happened is that many of Huawei's programmers are EEs who
> happened to take a programming course or two in college.

Oh yes. Most of my classmates have picked EE over software engineering because
they "dislike programming". The fact that EE (and any other engineering field
really) is now mostly about programming anyway is lost on them.

People designing EE study programs are usually completely unaware that, well,
computers exist. I have literally heard "Want to learn programming? Uh, just
spend a weekend playing with Matlab or something and you will be set.".

------
dspillett
_> because it was developed by coders who learned by copying the crappy coding
practices they found in earlier versions of Unix/Linux, and who were highly
selected through programming tests which could only be passed by adhering to
these same practices_

This will be the same in a great many places, including western companies. It
is market forces in action, the race to the bottom, and a consequence of going
for the option that "looks the part and it cheap" for too long which makes the
supply-side players support being fast to market with the right buzzword
features cheaper than the other lot and the bare minimum of care for other
considerations.

The people making the buying decisions are at least vaguely aware of the
problem, but work under the hope that they'll be long gone (or the kit will
have been replaced by something else) before any excrement gets uncomfortably
close to the air conditioning system.

At some point things come to a head and people look more deeply into the
problem, hence:

 _> HCSEC threw a wobbly and told Huawei to sort itself out pronto_

What has brought things to a head this time is the question "are we sure that
can we trust Chinese companies?" combined with the question come out of the
various public surveillance/recording news stories "can we trust any
companies?!". People are being told to dig by the powers that be because said
powers don't want to be the ones to make a cock-up while there is some public
scrutiny going on.

 _> the Chinese company still came back with software containing "code that is
vulnerable to 10 publicly disclosed OpenSSL vulnerabilities some dating back
to 2006_

That though is particularly damning. Having been pointed to the problem and
told to sort it out, they seem to have done _less than_ the bare minimum and
hoped to get away with it. Just think how lazy they were being when not
actively being told to be more careful, and how lazy they might go back to
being if this all blows over before significant internal change is made.

~~~
helloindia
" Just think how lazy they were being when not actively being told to be more
careful, and how lazy they might go back to being if this all blows over
before significant internal change is made." \-->Chabuduo, the chinese word
for it. [https://aeon.co/essays/what-chinese-corner-cutting-
reveals-a...](https://aeon.co/essays/what-chinese-corner-cutting-reveals-
about-modernity)

------
dagw
Link to original article:
[https://www.theregister.co.uk/2019/03/28/hcsec_huawei_oversi...](https://www.theregister.co.uk/2019/03/28/hcsec_huawei_oversight_board_savaging_annual_report/)

Link to actual report:
[https://assets.publishing.service.gov.uk/government/uploads/...](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/790270/HCSEC_OversightBoardReport-2019.pdf)

------
anonymousDan
It strikes me all this wailing about code quality is simply a cover story to
allow UK public sector/companies to exclude Huawei without outright accusing
them of being a spying/intellectual property risk.

~~~
StudentStuff
If Huawei is incompetent at managing OpenSSL in their codebase to the point
that there are nearly 100 copies sprinkled through, most of which are
vulnerable to exploitation, that seems like a perfectly reasonable basis to
bar their products for a number of years, regardless their nationality or
other actions.

F5 Networks should have been branded a black sheep for its various TLS
implementation screw ups (they are the prime reason TLSv1 is considered
insecure), yet they came out mostly unscathed. We need to stop giving a free
pass to horrid development practices that create massively vulnerable
software.

~~~
BuildTheRobots
> F5 Networks should have been branded a black sheep for its various TLS
> implementation screw ups (they are the prime reason TLSv1 is considered
> insecure), yet they came out mostly unscathed.

I'm failing to find more information (googling for ssl issues on ssl
termination equipment doesn't work well,) though it sounds like it'd be a good
read - if you have any more details to share, please...

------
sschueller
Large customer base, big profits, clean code.

Pick 2.

~~~
zbentley
This goes a bit beyond what most people would consider "unclean code", I
think.

~~~
raxxorrax
I don't get the demands to cleanliness.

The memcpy shadowing is really, really, really bad. If I see something like
that in production code I would never touch it. Removing it could break code
from both, people who relied on a check and those who didn't.

The 4 versions of openssh? Happens if you are in business for a longer time in
my experience. We don't know on how many devices the software is run on.

Sure, it could probably be cleaner, but so could nearly every form of
production code I saw in the real world. I would surprise me to see a
different situation elsewhere.

------
xchip
They say this applies to every single vendor but they only mention( multiple
times) Huawei.

~~~
peteretep
The random newsgroup post this links to does, but the sources it relies on
don’t

------
drallison
I would favor requiring a serious third party code review and certification of
all infrastructure systems. I am fairly certain that most (Cisco included)
would be "savaged" by a serious review.

------
Chico11Kidlet
Does this mean now we should choose Huawei Infrastructure kit over any else's,
as they're probably all as bad as each-other but at least Huawei's have been
audited?

~~~
viraptor
No. It has been audited and shown insecure. There's still a chance that other
providers are not as bad. They can't be worse, unless they're actively
malicious.

~~~
cyberbanjo
Why can't they be worse?

~~~
viraptor
According to the review the software has been confirmed vulnerable to publicly
known issues via old public software. From here we could compare the number of
the severity of the issues... but we already know they don't care about fixing
those, so there will be more. What can be worse than this without being
malicious?

------
romanovcode
Just like every other code.

------
ArchD
They said:

"There were over 1400 direct invocations of 22 different safe strcpy()-like
functions and over 400 direct invocations of 9 different unsafe strcpy()-like
functions. Approximately 22% of the direct invocations of strcpy()-like
functions are to unsafe variants."

There's nothing provably wrong with using strcpy() instead of strncpy(). If
you are are careful to ensure certain initial conditions, there is nothing
wrong and on a device with limited resources, it may be important to reduce
unnecessary runtime safety checks and instead pay for error avoidance at
compile-time.

~~~
yjftsjthsd-h
> If you are are careful to ensure certain initial conditions

And you trust people to do this? Even competent programmers need safeguards.

~~~
ArchD
Black-and-white thinking and argument is your enemy. Your use case may call
for a lot of safety at the expense of performance, so you do run-time checks
like this and sacrifice performance, or it may require extreme performance and
you do static analysis instead of these wasteful run-time checks.

There's no such thing as 'trust people'. If you write the code yourself you
decide whether your own code is safe enough. If you are a manager, you look
for unit tests, static analysis or do a detailed code review.

------
wrong_variable
Alright if its so bad then why use it ?

Get your own army of coders to create your own 5G electronics.

Nobody is forcing the Brits to buy Huawei Gear - they did it because it was
cheaper and they wanted to save a buck.

Huawei will be profitable even if UK,USA,AUS,EU completely stops buying their
gear.

The scary part is not that their code is bad, but that they code at all.

30 years ago those hands were picking rice in rural china, and 30 years from
now who know what those hands will be doing, maybe writing code for their
moonbase.

Chinese gears is a boon to African, Latin American, South Asian countries.
Chinese Yaun puts less pressure on their FX reserves.

