Hacker News new | comments | ask | show | jobs | submit login
Huawei's attempts to copycat a T-Mobile robot read like a comical spy movie (businessinsider.com)
91 points by SirLJ 18 days ago | hide | past | web | favorite | 55 comments

(Disclosure: I make robots for mobile testing.)

I wish more companies realized all the spying and stealing of information would be unnecessary if they started collaborating on making open source hardware testing equipment. A robot that taps a screen shouldn't be considered a super important trade secret.

Nor especially difficult to recreate.

I don't get it, it should be so simple. There are three dimensions in which the tapper has to traverse, probably not changing the angle of it either. It probably has some squishiness in the tip, so it doesn't have to be super super precise.

This was in 2012, when there already where 3d-printers with much more harder requirements. Well, sure, this would run 24/7 likely so should have better components and be quite robust.

So, some development but not like inventing anything drastically new. What am I missing?

edit: ah, other comments raise some interesting points regarding detecting state of the phone and apps. I looked at it from mostly a hardware perspective since that's the impression I got from the article.

The article discusses the lengths Huawei went to in order to obtain hardware specifications in order to build a clone. However, Tappy appears to be an incrementally-evolved version of the open-source Tapster robot, making this akin to an auto manufacturer engaging in high-stakes corporate espionage in order to obtain the precise dimensions of a competitor’s steering wheel.

Quite frankly, you’ve got to wonder about Huawei’s engineering chops if they’re going all spy-vs-spy to steal something that’s 75% available on Github already.

Someone bought up this point in the other discussion but 3D printers is a really bad example for a gantry for production based systems. This is mainly because of the open loop design and the asynchronous motion profiles between the motors. There was a lot of work that went into GRBL to make the axis seemed sync. Also, getting into motion profiles is super tricky as well (which is trapezoidal preset in GRBL). The thing with these devices is that it has to be working 100% of the time with a feedback making sure its doing what you expect.

That being said, besides the kinematics, instrumentation for hardware is the tricky part. Getting repeatable and consistent measurement for large production quantities is not an straight forward task. Processing and digesting the results is also another hurdle.

I'm curious whether there's a better reference for gantry robotics? It looks like the hard parts here are non-mechanical, but I'd be interested to know more generally.

Automated brick-layers come to mind as another 3D positioning task with no pitch or roll, but I'm not familiar enough with the general space to know if they're at all representative. Among other things, I seem to remember they can't corner properly, which suggests they might be operating on a linear axis and a width rather than a proper 3D workspace.

> if they started collaborating

Note possible anti-trust red flag

> on making open source hardware testing equipment

Anti-trust red flag? That would be a real hard sell to even the most anti-corporate type.

Is it an anti-trust red flag if web browser vendors started collaborating on testing tools, too?

There was a civil lawsuit and Huawei won.

This is probably just part of the general witch hunt the US is undertaking against Huawei.

Actually, Huawei did not win.

The jury found for t-mobile.

See, e.g., https://www.geekwire.com/2017/jury-sides-with-t-mobile-in-ca...

They then settled the case.

Civil suits and criminal suits are separate. I'm not sure if settling a civil suit can be seen as exonerating for a criminal suit. Standard of proof is higher for a criminal suit, and civil suits can be settled easily with money.

I don't think witch hunt is the right term, as there was obvious unethical acts (possibly criminal). Although, it is definitely an escalation and I do wish there was more communicating through diplomats than through legal systems.

A classic example: OJ Simpson was found not guilty in his criminal trial, but was found liable in his civil trial.

It's abundantly clear from the emails that Huawei was maliciously trying to steal IP, to the point wherein they snuck people into a facility to carry out industrial espionage.

There are numerous reasons why a civil suit might have failed even when the aforementioned is true.

Moreover, this seems to be not an isolated case, but a systematic behaviour instituted within the organization, and others, including the state.

However ridiculous it is for people to be trying to copy the 'tappy robot' (!!) - this is a serious issue that needs to be addressed.

My point is that one way to address it is to not consider a screen tapping device a trade secret and make these devices open source. But yes, all the stealing is wrong and should be dealt with, too.

If one is in the business of making such robots, then it would definitely be a good idea to protect one's IP.

We don't get to automatically assume the position of one player in the value chain (say mobile manufacturer) and assume it would be 'beneficial' to just open source the IP of some supplier.

Every 'buyer' everywhere would want all the 'supplies' as cheaply and freely as possible (!) but it can't usually work that way.

In the article, there's an email referring one person to the 'manufacturer of the robot and not Tmobile' - which I guess indicates that the thing is not made by Tmo but by some 3rd party supplier.

Should that 3rd party who ostensibly invested in designing that thing just 'open source' their products? How would they benefit from that? It's possible they may, but doubtful.

I guess because this isn't exactly 'advanced robotic's it all seems a little clownish, but most of what we make isn't super hight tech, yet laden with specific know how and knowledge.

Witch hunt? Are you seriously suggesting that Huawei _doesn’t_ engage in corporate espionage?

If this is factual, it shows a HQ-coordinated push to steal secrets about a single modified industrial robot. I don't understand why they put up so much effort for this one machine with questionable impact on their bottom line?

Most likely because it is institutionalized to steal IP and stolen IP is seen as more valuable than homegrown IP. The actors are financially incentivized by the company for any stolen tech (per the indictment). Add Chinese culture of pleasing the boss and you have the perfect recipe for this kind of behaviour.

This is just the one they got caught for, probably.

Right. I've heard from friends in telco that Huawei techs would routinely open server racks during 3am service calls and photograph everything. This was years ago, but I'm guessing a company wide incentive program wouldn't get igored by many Huawei techs.

Never underestimate desperation. If they have a mandate to get a phone approved for reasons, I am sure the personal repercussions for failure are grave.

This is ridiculous. It’s a SCARA arm with a touch probe, camera, etc. I have personally developed such systems to test touch screens for applications ranging from commercial to aerospace.

As a side thought, this is the kind of thing that is wrong with our patent system:

Implementation is confused with invention. The USPTO grants patents for things that any reasonable engineering team would implement given a problem to solve.

Engineering is about problem solving. Not all solutions are inventions. In fact, I would argue that there are very few true inventions these days and that the general rate of true inventions should naturally decline over time.

In other words, the threshold for what should constitute a true patentable invention, given accumulated knowledge, should increase with time.

This is China's modus operandi, aquire other people's tech and possibly improve. They have a ton of brilliant engineers and great manufacturing, however it seems like creativity is a bit lacking.

a few (and i mean very few) examples. Aircraft carrier bought from russia to copy, J-20 stole (allegedly) F-22 designs from the US and modified, segway - ninebot, J-11B fighter they copied from the su-27. The list goes on and on. Here is a link with greate examples and Pictures (https://news.usni.org/2015/10/27/chinas-military-built-with-...).

Again this is not knocking the engineers in China, their reverse engineering is second to none. Reminds me of the "Pakled" on Star Trek TNG, they just aquire all technology from other races and make it work.

I posted this on the other discussion but the actuator and gantry is usually the easy part. "The instrumentation and processing side is never straightforward, especially in production. I'm guessing the trade secret is probably some sort of measurement device or some special pressure balance system on the actuators"

Side note, trade secrets != patents. The whole idea between the two is fundamentally different.

I think its human nature for things to seem 'obvious' after the fact, but creating something novel doesn't require it to be technically challenging.

I had this reaction to a number of graduate computing courses, the feeling that given five minutes on the problem (parallelism comes to mind) anyone would have thought of that approach!

In retrospect, it's clear how much that mentality undervalues what was being taught in the course. Yeah, the initial approach might be obvious, but so are plenty more initial approaches. What made it through to the course and was put down in the textbook are the few initial approaches that actually worked.

/Textbooks should have a long list of attempted approaches that seemed great and weren't, but no one publishes failures.

I'm baffled as well. They literally took an old Epson pick and place machine and just created a custom attachment for it. Unless there are some massive gotchas in the problem, I would think this would be the kind of thing that would pass as a middle of the road senior project for an ME at a good engineering school. Surely Huawei has good enough engineers to re-implement this themselves? Why would they go to such lengths to try to steal it?

> Implementation is confused with invention.

I look at the system as an admission that it's impossible to know, post-hoc, whether something was copied or independently developed.

We wouldn't have a problem with no patents, if we could only determine that.

Agree that it seems like there should be more of an obviousness floor (e.g. "Can a reasonably trained engineer build this in less than a week?").

This particular case doesn't seem to be about patents, rather design and know-how, i.e. the kind of IP not very well protected by law.

It is a little ridiculous as you say, but the fact that a company was obviously involved in attempts at industrial espionage makes it kind of serious.

This raises so many questions:

1. How many pictures do you need? It seems they keep sending out pictures and then they send more people to get more pictures. Is there a lot of additional value in additional pictures?

2. How hard is it to reverse engineer tappy? Once you know how it works, I just assumed that Huawei has more than enough engineering talent to put together a tappy.

3. Was there a particular component of tappy that was hard to copy?

4. If I had to guess, the software that recognizes if the phone screen is on and working is the secret sauce. Not the mechanical tapping which, although sophisticated, is presumably copyable.

In answer to #1, I'd assume because Chinese PMs are much the same as PMs elsewhere.

PM: "Why aren't we done yet?"

Dev: "Umm, we need more photos."

{Cue never ending stream of request for photos}

I tried to build something very much like "Tappy" as an internal testing tool, and with the limited time and effort I put in, #4 was definitely what prompted me to give up and go for a "screen capture over ADB" instead of a camera.

There was glare, all kinds of distortion, noise etc, that while of course possible to overcome, was way out of my league to deal with. I guess one could have thrown machine learning in the mix too... :-P

Physical things are hard, there are a million little issues.

Sometimes 'seeing' a piece or component can reveal a lot as it represents the solution to a problem that is not otherwise evident.

The smallest details like the size/type of motors, power ratings, how they are positioned, what kinds of sensors they use can reveal a lot.

To boot, you're not dealing with a toy that can fall apart after 10 uses - this is gear that ostensibly has to be fairly robust, which introduces a whole other set of quality challenges beyond 'just making it work'.

Example: making a little 'toy car' that ran across the floor, we ran into all sorts of control issues with the motors as the car encountered surfaces of different types, and also issues with getting caught on carpet etc.. Upon seeing how our partners made a previous device, it became pretty clear how they solved the problem. Instead of testing various different configurations and trying to find the 'best fit' - we just used their example.

I raised this in the other thread... that article suggested this incident had already been litigated in civil courts with Huawei being found guilty.

What is different now and why wasn’t it a bigger deal before?

Googling says they weren't found to be "willful and malicious" and only "breach of contract." I guess this new evidence suggest it was willful and malicious.

Also note that this story doesn't cover the other charges of wire fraud brought against them.

Also, IIRC, the new charges are criminal. The previous case was a civil one.


Why would T-Mobile still give one seat to Huawei US engineer for testing after knowing the company is trying to steal the trade secret...?

People elsewhere in the the comments pointed out that Tappy seems to be a derivative of the open source Tapster robot. So I suppose it's possible that T-Mobile (or at least the people overseeing the Tappy lab) simply didn't care very much. They might have objected to photos and long lists of questions to avoid letting Huawei crib all the challenging parts of their upgrades, but not have been particularly concerned with whether people saw the basic functioning of the thing, even knowing they wanted to copy it.

Now, a very cynical idea is that T-Mobile figured Tappy wasn't expensive to replicate regardless, so the maximum profit lay in indulging Huawei's decision to commit a trade secrets violation. That would explain giving Huawei continued access while refusing to answer questions and installing good-faith safeguards in the lab. There's some circumstantial evidence which could back this: T-Mobile apparently considered licensing out Tappy, which is what started the lab-access program at issue here. And TMO did sue Huawei for "reasonable royalties" on Tappy plus double that in punitive damages, though all they got were breach of contract damages. If they suspected that Huawei wouldn't buy an expensive license and could replicate Tappy without espionage cheaply, then a shot at triple royalties might have sounded like a winning outcome.

I don't believe that's true here. It'd require probably-illegal coordination to get T-Mobile staff on the same page, all in the interest of fighting a lengthy trial to maybe receive damages which would be tough to sustain on appeal and collect. But laches defenses exist for a reason, and a strategic delay to collect larger damages is at least an interesting possibility to consider.

My bet: good old one-to-one professional relationships

Did they not have access to youtube?


Yep - in China they do not have access to youtube. https://en.wikipedia.org/wiki/Websites_blocked_in_mainland_C...

Even so, I'm sure Huawei provides an uncensored VPN connection to its engineers to bypass such blocks.

What kind of security systems do they have in place where someone can carry this thing out of the building without being noticed?

This reads like Benny Hill was in charge of the operation.

“China asked A.X for detailed measurements of Tappy's robot arm. On May 29, A.X allegedly walked into T-Mobile's lab, broke off Tappy's arm, and put it into his bag. When discovered again by T-Mobile, he gave the arm back, but the damage was done.“

I can hear the theme music.

Reading this makes me want to play Shadowrun. :)

This whole thing seems so tacky and cheap. It reminds me of a rich old person holding up the line so he can get his Senior discount and save $1.47. Is Huawei so poor and T-Mobile charging so much to use the robot?

Yea this sounds like some random people back in huawei China were super curious about this robot and wanted to copy it so they didn’t need to buy from some company rather than some kind of broad company effort to steal super high tech stuff. I hope if governments are going to go after huawei they have more than this pretty lame story..

Probably lots of firms have committed similar goofiness at some point. Huawei is criticized for this particular episode because TPTB have approved Huawei as an acceptable target for our Two Minutes Hate.

How hard could it be to build? Actually the whole articulated arm thing seems quite unnecessary to me since you’re dealing with interactions on a flat plane.

A simple XYZ platform driven by steppers would probably be far easier and more reliable/reproducible. Throw some OpenCV in with feature tracking and a database of phone screenshots and some shitty internal web app and...lol

These robot arms are off the shelf and very repeatable and reliable. They're used all over the place in automation and assembly every day. What's hard about using this either the controllers are very good and can be easily scripted for the whole sequence of test touches T Mobile's engineers wanted. They're also very fast moving so they can rapidly test across different parts of the screen.

If they are so easy why was Huawei trying so persistently to steal the details? This is more irrefutable than the usual info. H might have done a great job developing their own tech in some areas totally on their own, but that's not what was happening here.

> If they are so easy why was Huawei trying so persistently to steal the details?

As someone mentioned upthread, it could just be company culture around how they value stolen trade secrets vs internally developed know-how. Sort of like, once upon a time, "nobody ever got fired for choosing IBM."

It's not okay to try to steal things (as we probably don't even need to mention). It's doesn't matter if that is the accepted way to do things in a company, you are making a moral mistake. I don't want to tar everyone in the company as being that way, but it would be a serious thing to consider if someone in Huawei in my town (they have offices in the Seattle area) was looking for a job.

If you work in a company that has standard cultural practices of stealing things, then you have to not do it. Of course many companies try to be aggressive and push the illegal envelope (like say Uber) but you have to be moral and honest, it's a crucial aspect of the professionalism that we should have. That's a difficult line that I feel fortunate never to have faced in my software engineering career. Think about those engineers and leaders at VW who lied about the 'defeat device'. I would guess they were in some kind of automotive or mechanical or other professional organization that had a code of ethics that you couldn't do those things.

I was responding to iamleppert's assertion that it'd be easier to use a gantry based system versus a SCARA arm like T Mobile used. Either method runs into most of the same issues including the end effector (pokey-bit that touches the screen like a finger) which seems like the bit Huawei was having trouble with given that's the bit they stole.

Ultimately the real trouble probably isn't the actual arm or gantry but all the software around it to determine what's on the screen and if the phone is functioning properly with respect to the expected test results and the response times which seems what they were interested in. Maybe that's just my bias though and there may be some major hurdles with the function of the fingertip.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact