I worked on the censorship and government reporting (sending all logs) infrastructure for Akamai China CDN. I'm glad to see it get shut down. Happy to answer questions.
Thanks for the disclosure. Don’t feel too bad, those stuff you helped built may not cause much damage as you feared.
CDN used to be at the front line of blocking content and surveillance citizens, nowadays that happens mostly thru social networks.
Unlike other countries, Chinese citizens are ultra online, and mostly concentrated on two platforms Webo and Wechat. Most other online services all require authentication via either of these two, plus cellphone number. So for govt it’s very simple to block anything or see any identity, CDN hardly play any major role anymore.
1. How was working with China requests and logging, differing from working with other nation states?
2. Was there full services brought up only for China specific needs? What would they take care or?
3. How would any blocks work? allowlist or denylist? Was takedown immediate, or was it working with the customer/client and getting them to take it down within SLA?
1. At the time it was the only nation state that had specialty infrastructure except for maybe the US.
2. There were specific infrastructure changes made for blocking and sending logs inside mainland china.
3. The CDN node would deny access to specific urls uploaded by the Chinese partner company. I don't remember the SLA. The SLA for reporting visited URLs was 15m IIRC
When I was at Akamai about 5 years ago, I was involved in building the system for making their CDN compliant in China. There were two main features, and they were activated on all servers running inside mainland china (not HK, macau or Taiwan)
1. Logs of the CDN were sent in real time to the ministry of technology -- there was about a 15 minute delay if I remember correctly, and they could impose fines if they were delayed. The log included the url visited, the IP address of the visitor, and a few other things. Perhaps the user agent? I forget.
2. The ministry of technology had a special API to block URLs on the CDN. Basically, they provided a list of URLs that would return a 451, and of course those logs also went to the government.
No other country had this kind of access at the time, but it was considered critical for the business to continue to operate in China. As I understand it, these are required to comply with chinese government regulations, and other CDNs like Cloudflare and Cloudfront have also built similar capabilities. Perhaps jgrahamc can comment on what cloudflare did?
I feel quite guilty about being involved with that project, but the business was set on building it, so I did what I could to limit the blast radius. I would not be surprised if someone got arrested or was killed because of it.
Glad you regret it. Not trying to rub it in as I don't think anything productive will come from self-flagellation, but this is truly awful and I think the US should have laws that make it a crime for any US corporation to participate in this sort of thing.
I was powerless to stop it. I was just a junior engineer, and it was decided by the CEO to do the project. So, actually, I feel I made the right choice -- I participated in the project but worked hard on making sure it was as limited as possible. I successfully advocated for several categories of logs to not be sent because they were not required by law.
So, yes, I regret I couldn't do more, but I don't regret the choices I made with the information I had and the position I was in.
You couldn't have stopped someone from building it. But you could have refused to work on it on principle, or even have become a whistleblower.
Yes, doing so might have been infeasible for you, particularly if you couldn't risk a temporary loss of income. But your involvement was, nevertheless, a choice, and it's important to acknowledge that.
edit: If it was exactly 5 years ago, you may recall that, when you were working on this, China was starting to round up Uighurs to send them to concentration camps. Nobody should take working on this sort of thing lightly.
If they refused the work and let someone else who cared less about limiting the amount of data, things would be worse.
Also what would whistleblowing do? A lot of companies were operating in China and followed similarly privacy-hostile regulation.
Also to bring up Uighurs in this is ridiculous. Logging ips and urls has no direct correlation with being able to round people up in concentration camps. It has nothing to do with what the Uighurs ideologies were, it has something to do with who they are and the cultural differences they had with mainland China.
To try and look down your nose at an engineer who did the best they could with the position they were in with the belief that there was more that could be done is just naive.
You may recall that, when someone blew the whistle on Google Dragonfly, a censored search engine intended for use in China, the public outrage was enough to bring the project to a halt. The same might well have happened to Akamai.
Regardless: this whole attitude strikes me as an overly utilitarian outlook. Yes, if someone else handled the development, the consequences might well have been worse. But it is still wrong to participate in an injustice when you have the opportunity not to do so. "I was just following orders" is a pretty weak excuse.
Again, if the commenter had no other options because they couldn't risk the loss of income, that would be a good reason, but it isn't clear that that was the case.
I think it's pretty naive to assume that this project wouldn't have been used against the Uighur population, given how China has used extensive surveillance against them.
I applaud you. I absolutely did not intend to rub it in or make you feel worse than you might already (which it sounds like you have little reason to). I admire you sharing your experience and being honest about it without beating yourself up too much. The world is made better by little decisions for the good like the ones you made. Thank you.
Thanks for sharing your story. I am curious what you would do next time if you found yourself asked to build something that you found unconscionable? Would you refuse to work on the project?
Other engineering disciplines have a strong focus on 'engineering ethics' and it may be more acceptable in different branches of engineering to refuse to build something that you consider unethical. I do not know if there are any professional bodies or laws which protect the employment rights of individual engineers who refuse certain work on ethical bases. But I feel that software engineers should be able to exercise their conscience, reference a standard of professional ethical principles, and refuse to work on such projects.
Akamai has very tight relations with the US Government. So what was probably happening is that the USG was fine with Akamai treating it's users like this because it was getting a copy of those URL filters and access logs too. Don't know if that should make you more in or less sorry for being involved.
While I wouldn't put any authoritarian moves beyond China's reach, the ICP recordal mechanism already requires government approval.
In that case, isn't it better for user privacy (not that anyone cares about it in China) to receive an ICP recordal but then wait for an actual request from law enforcement to turn over the logs?
Also, while you wouldn't see anyone from Amazon or Cloudflare comment on your thread, both have the ability to stream logs to a destination, and that is also exposed to customers, so I don't think they needed to build anything else.
All of the sites served had an ICP license. This is separate, and the CDNs in China have regulations specific to CDNs they need to comply with.
At the time, Akamai also had the capability to stream logs, but the ministry of technology required a specific, custom interface to receive them, which required engineering work, especially to do it for an entire country without the customers configuring it themselves. I would be extremely surprised if it required no engineering work at Amazon or Cloudflare to deliver the logs in the way they requested.
There was a person in Beijing that was arrested for a post on Twitter. Government surveillance was able to track him down in real life, which I think is deeply troubling
This type of China vs US narrative where existential stakes are implied is just dumb. The technological Cold War we’re in is not going to result in the collapse of either nation.
I think you may have misinterpreted the parent comment, (or perhaps I did). China doesn't have a deep libertarian tradition, so overbearing state surveillance, while odious, isn't a strange concept in Chinese culture. Western governments are/will be unable to resits the urge to deploy such measures because they clearly work so well for authoritarian states, but will fatally undermine their own cultural foundation by doing so. It wasn't so long ago that regimes like East Germany were considered the epitome of totalitarianism, with the Stasi (domestic intelligence service) coopting large numbers of informers, listening in on the citizenry etc. Yet today many of us are subject to far more intrusive surveillance, it's just got super-friendly branding and some people are making money out of it so 'freedom' has nominally prevailed.
That’s the thing though, it doesn’t work that well. China is a huge place that seemingly has many challenges that are underreported in the global landscape. The trade offs the Chinese government makes to better control the people limits their ability to draw from the global talent pool and restricts free thinkers and innovation.
In the first novel written by Ernesto Sabato, The tunnel [1], in one of the novel's tangents, there is a twist on the common fairy tale of the knight, the princess, and the dragon. For Sabato, the moment of clarity, of truth, is to realize the princess is the dragon.
The monster, the technological golem foreseen in the works of Lev Shestov, Martin Heidegger, Jacques Ellul and more, is that because our tool-making inclination is both our salvation from nature's whim and our damnation (environmental mayhem and others), the princess is the dragon, we are not able to purify the technological pursuit in order build a bridge between intelligence (how to achieve a goal) and wisdom (why to pursue a goal).
A historical factoid: John von Neumann, whose ideas we exploit every day by turning these very devices on, has a quote "If you say why not bomb [the Soviets] tomorrow, I say, why not today? If you say today at five o'clock, I say why not one o'clock?" [2] Apparently, von Neumann wanted to nuke the Soviets in 1945 or soon after, before they had the nuke themselves, in order to establish dominance. The Soviets collapsed themselves, half a century later, in the meanwhile and in the aftermath torturing and maiming the lifepaths of hundreds of millions of people. Who's to say what course would have been better.
[1] Sabato wrote only two more novels, On Heroes and Tombs and Abaddón, The Exterminator, all three are masterpieces of universal literature.
[2] "The Passing of a Great Mind" by Clay Blair, Jr., in LIFE Magazine (25 February 1957), p. 96.
The parent comment said "[t]he west created the monster". The point is, since our technology is only a means of enhancing intelligence, not wisdom, we are unable to perceive the end-leaf monster of each decision tree.
But sure, we can imagine a political-fiction novel à la Philip K. Dick where Oppenheimer, von Neumann, and Feynman team up and after nuking Hiroshima and Nagasaki, they nuke Moscow and Washington D.C., and form a world government ruled by scientists, a techno-solar punk utopia based roughly on Herman Hesse's Castalia from The Glass Bead Game [1].
Previous discussion about it: https://news.ycombinator.com/item?id=33678019
reply