Hacker News new | past | comments | ask | show | jobs | submit login
U.S. v. Ross Ulbricht: Declaration of Joshua J. Horowitz [pdf] (antilop.cc)
79 points by cyphunk on Oct 2, 2014 | hide | past | favorite | 70 comments

Quite an interesting read. Summary? Silk Road used a pair of nginx servers, one as front-end, one as back-end. The server images (captured forensically) show that the then-in-place configuration made it impossible for the FBI agent to do what he claims, that is, connect to the back-end by using the server's IP address (the allow/deny settings forbade this). The log files on the servers lack entries to support FBI claims.

Furthermore, FBI agents describe using other techniques to obtain information, e.g., fuzzing and packet sniffing. Not only do server logs NOT contain any evidence of this, but the agents did not preserve any evidence of their packet sniffing activities, despite having training in forensic investigation techniques and claiming expertise in these areas.

While it is possible that there were bugs in either the Ubuntu 12.04 hosts or the nginx servers themselves that would have allowed these activities to occur, unlogged, I suspect it would now be up to the prosecution to establish reasonable grounds for believing this to be the case. Likewise, while it is possible that the handful of lines of log pertaining to FBI activities might have been removed from the several million (yes, you read that correctly) lines of log later captured in forensic image, that does strain credulity; again I suspect the prosecution would have to establish that, somehow.

I've no particular interest in this case or in Silk Road, but I cannot help but conclude that the lawyer filing this brief just seriously schooled the FBI.

> Without identification by the Government, it was impossible to pinpoint the 19 lines in the access logs showing the date and time of law enforcement access to the .49 server.

Unless I missed something, this is the only point in the document where the defense suggests that they couldn't find the log lines provided by the FBI. It seems like a weak claim, since they could have said "could not locate" or "could not find", and suggests that there could have been multiple non-.onion or non-frontend accesses to the frontend and backend servers respectively. Surely the expert would know to 'grep -v', and that should make it easy to determine any access that came from other than the prescribed addresses. This suggests that the logs contradict the claims of inaccessibility.

The defence will do well leaning more on the incompetence of the prosecution and holding their cards close. In general the system should work where the burden of proof is on the state. What they have done now is show that the configuration of the server as recorded by the FBI makes the access description of the FBI impossible. Yes they could go further to grep -v out localhost and the front-end's IP addr but they can save this to further humiliate the FBI when convenient. Imagine the prosecution now counters with some other detail they forgot to mention but the proof the defence held back humiliates them again. Do this several times and the judge will, rightfully, be disturbed at the state not disclosing the full detail of their case.

...with the implication being that the U.S. Government discovered the identity of the operator of Silk Road through means that would either not be admissible in court or that would anger the public. They then engaged in acts of "parallel construction" to contrive lawful means that they discovered his identity using the foreknowledge of who he was via unlawful means.


The nginx conf for the .49 server[1] has a bug. The location entries are exclusive[2].

[1] http://krebsonsecurity.com/wp-content/uploads/2014/10/70-6.p... [2] http://nginx.org/en/docs/http/request_processing.html#simple...

Well, fuck. Regardless of the rest, if the government's story changed in a factual way before vs. after the government acquired the Silk Road server ... well, further confirmation that parallel construction is in use and the "foreign tools" are being used in domestic cases.

"the account by former Special Agent Tarbell in his Declaration differs in important respects from the government’s June 12, 2013, letter to Icelandic authorities. For example, that letter (which is Exhibit A to the government’s opposition papers) suggests the possibility of an alternative method for the government’s identifying and locating the Silk Road Server; "


"The Government’s response to Mr. Ulbricht’s omnibus motion filed September 5, 2014, contains a Declaration from former FBI Special Agent Christopher Tarbell, attached hereto as Exhibit 2 (Dkt #57). The Declaration contains a vague explanation of how the IP address of the Silk Road server was initially discovered. For instance, former SA Tarbell asserts that, “[w]hen I typed the Subject IP Address into an ordinary (non-Tor) web browser, a part of the Silk Road login screen (the CAPTCHA prompt) appeared.” Tarbell Decl. at ¶ 8. As explained below, based upon the Nginx server configuration files provided in discovery, that was not possible. "

Agreed regarding the usage of parallel construction-- I am sure that somewhere in the government's internal guidelines on when to use parallel construction there is a catchall clause providing exemption for "exigent circumstances" which allows it to be abused domestically for any purpose.

This case now reeks of heavy NSA involvement merely passed to the FBI for the actual enforcement side of things.

I'm not sure if there's actually an angle here, but it would be extremely interesting if the defendants were able to impeach the FBI's evidence to such a degree so as to force an admittance of parallel construction. It's a no brainer that something fishy happened here.

They'd drop the case before they admitted parallel construction. The old Silk Road is gone and if Ross Ulbricht isn't innocent he won't ever be trying anything like this again knowing that the NSA and the FBI are going to be surveilling him to the best of their ability so they only thing they lose is locking him up in prison.

> I'm not sure if there's actually an angle here, but it would be extremely interesting if the defendants were able to impeach the FBI's evidence to such a degree so as to force an admittance of parallel construction. It's a no brainer that something fishy happened here.

Tbh, I think that is basically their defense if you read between the lines. This defense reads as basically:

"The FBI's explanation isn't technically possible for how they found the IP address therefore this case should be thrown out as they are lying."

The FBI will never admit to parallel construction. They get enough leeway in their testimony, etc. that they can just say "We agree to disagree on what happened" and no one would question them seriously, from a justice system perspective.

Why would they incur a massive scandal for one guy they can surveil forever?

If enough doubt is cast, they'll drop this.

> "[...]For instance, former SA Tarbell asserts that, “[w]hen I typed the Subject IP Address into an ordinary (non-Tor) web browser, a part of the Silk Road login screen (the CAPTCHA prompt) appeared.” Tarbell Decl. at ¶ 8. As explained below, based upon the Nginx server configuration files provided in discovery, that was not possible. "

I've not been following this closely, so the following question is based just on the above. How do we know that the configuration files that we have now (the ones that would make it impossible to show the login screen to an ordinary browser) were the ones in place at the time Tarbell claims to have seen the login screen via an ordinary browser?

The mtime attribute on the configuration files from the server image as explained in the brief.

"The mtime for the live-ssl configuration file provided in Item 1 of discovery is June 7, 2013, and the phpmyadmin configuration is July 6, 2013"

That wouldn't always work on my server. When testing a new config, I often mv the old config out of the way, then put in the new one. If the new one doesn't work, I mv the old one back.

I'm not consistent on this. Sometimes I just make a backup with cp, cp the new one in, and cp the backup back if the new one doesn't work.

I honestly have no idea why I mv sometimes and cp sometimes.

A tarball is not a disk image and - in my view - is seriously shaky as evidence in a criminal trial.

I had always imagined - apparently incorrectly - that evidence-gathering requirements in this area would have been more along the lines of imaging the disk bit-for-bit in a controlled, well documented procedure onto another disk which is immediately made read-only in hardware before being placed in the chain of custody. Copies provided in legal proceedings should be verifiably identical.

EDIT: Also not only is it insane that they didn't save the sniffed packets but for a one-shot, unreproducible event like the one the FBI describes they really should have preserved the local environment too. Any number of unknown browser/extension/proxy/system behaviours could have caused that captcha to appear on the screen once. Hell, if they only saw it in the viewport and not, say, the DOM source, it could even be a bug in the system graphics renderer - massively unlikely, yes, but I've seen weirder.

All computer evidence is basically Law Enforcement's word that it is real. It is really scary because of how simple planting things are in the context of a tarball or a log file.

That was my concern as well. First thing I learned early on was that when you suspect that a server is compromised, or you you just deleted something super critical, is to dd it to an image and then mount it ro for forensics.

It was pretty obvious that the FBI was lying (or misrepresenting or whatever weasel word you want to use) in their original claims. The real question in my mind, which still remains unanswered, is why? The most optimistic explanation is that they just botched the investigation and then spectacularly confused the prosecution as they were preparing their claims. I'd rather not think too hard about the most pessimistic explanation.

I find it plausible that the NSA, DHS may have assisted. During the same time of the rise of Silkroad we have the NSA in full cowboy mode. A reading of the xkeyscorerules100.txt[1] leak shows this, among other things. Looking to Tor as a potential tool for terrorism, if not now than in the future, they would turn to Silkroad as the only interesting target for them to train their teeth on. I just know that if I were in the shoes of a NSA "hacker" dealing with Tor I wouldn't be satisfied with playing war games against internal teams. Hackers are addicts that get their kick best when its public. I would certainly have argued for "better cooperation" with the FBI and DHS for that reason alone.

Anything the NSA does is "secret" so the holes in the FBI arguments, in this scenario, would make sense. An interesting result in this case would be if at any point evidence is not permitted for publication. I wonder if the defence could construct a canary for this possible circumstance.

1: http://daserste.ndr.de/panorama/xkeyscorerules100.txt (note how certain civil liberties are dictated by Geo IP location alone -- that is what I mean by "cowboy mode")

It's called "Parallel construction"... http://en.wikipedia.org/wiki/Parallel_construction

Most likely, the NSA got the information around May, tipped off the FBI and the FBI started working on their "PC" story. Unfortunately, Ross was moving his servers and reconfiguring them during this time, making the PC story a bit more difficult to get.

Isn't it obvious? They wanted him gone. This is just like what happened to Kim Dotcom - the US government/elite wanted him gone, so they used whatever methods they wanted, executed by the Law Enforcement, to try and remove him. Sure, they might lose in court now, but most of the damage is done - Megavideo is gone, Silk Road is gone - just what they wanted.

EDIT: I guess the biggest problem here is that the people responsible for actually perpetrating the crimes (e.g. the police officers lying in court) will most likely not be held responsible.


Also, isn't there a new Silk Road already?

There is a new Silk Road and a plethora of successful alternatives. Far from shutting down darknet markets they really just gave them more publicity and now have 20 sites to go after instead of one.

And a new DPR to go along with it

They changed their story because the truth is that they violated the 4th amendement rights of anyone who browsed Silk Road for a period of time. They executed drive-by Javascript code on the computers of hundreds of unknowing silk road users, without a warrant. At worst, that's extremely illegal, and at best, it's in a definite gray area. I don't blame the FBI for trying to cover it up.

(Citation -- wired article I read at some point; maybe someone else can find it.)

You are confusing this with Freedom Hosting.


Ah, my mistake. You're right. Although I would not be surprised if they used a similar technique once they had access to SR servers.

Someone admitted I'm right in HN. I should go out and buy a lottery ticket. ;)

We all are wrong sometimes. Shit happens. :)

Some researchers at Yale have been working on a project [1], due to be presented this weekend, that disables the FBI from gaining any meaningful information via Javascript exploits. The idea is to contain each "pseudonym" in its own virtual "nymbox", and only that one box. Read the paper for more, it's interesting.

[1] http://arxiv.org/pdf/1312.3665.pdf

I also have an idea to stop the FBI from gaining any meaningful information via Javascript exploits: PGP or Ed25519 sign all .js files and have a browser that only executes signed code.

Not only do you need to obtain access to the server, you also need the private key of its operator. (Which should be kept offline for signing.) If they don't surrender the key, then you cannot compromise their visitors.

This can also be a mitigation for drive-by malware exploits and whatnot.

(I emailed this to LiberationTech last year, no one took any interest in the idea.)

It's an interesting idea, but currently quite impractical as there's no infrastructure for it.

The most secure way right now is to isloate your Tor browsing activities to a virtual machine which is only able to access the internet via Tor.

Create a VM to act as a middle node with 2 NICs, the first of which will connect to the internet and the second of which will connect to the other VM. Disable any unnecessary services, expose only Tor's proxy to the second NIC.

Create the other VM with 1 NIC in the virtual network with the Second NIC on the other VM.

This enforces Tor-only access for the second VM. Any activity must go over Tor or it may not access the internet.

Snapshot the VMs. Always restore the snapshots when beginning a new session.

Setting this up takes some time, but nearly guarantees an exploit cannot escape.

I also recommend disabling unnecessary services which may expose the host to attack (ie: file sharing and video and mouse acceleration) through the second VM. It may be more secure to use a low-level emulation (QEmu, VirtualBox or VMWare with hardware acceleration disabled) rather than a hypervisor-based VM solution for this.

> It's an interesting idea, but currently quite impractical as there's no infrastructure for it.

If TOR and Namecoin hooked up, Namecoin could provide a list of page hashes for a given onion address. Better still, interrogate Namecoin's DNS system into TOR and then make Hidden Services accessible via human readable domain names.

You mean some people use Tor differently than as you suggest?


I had assumed that everyone would be either on the 'net with their 'net computer or on tor with their tor computer.

Doing both with one computer obviously defeats the purpose, doesn't it?

...Also, the chances of breaking out of a VM into the host are non-zero. Just sayin'.

That's a cool idea, but it comes with a lot of overhead. Javascript files are one of the most heavily-cached on the web, and by signing them with PGP, you cripple your caching ability.

Or am I missing something?

The original idea I had was to have an index (think hybrid cache.manifest and Content-Security-Policy header) of relative URLs and their SHA3 hashes, then that file was signed. If anything fails to validate, it doesn't run.

Signing just prevents you from altering the files so you can cache them just fine.

You could also just sign the hash.

How does the browser get the public key in order to verify the signature? And how is this any different from just serving scripts over HTTPS?

You could just make it an HTML meta tag which specifies trusted keys. And add an attribute to the script tag which contains the signature.

Might need to disable inline JavaScript, though. And double-check that meta tags are only meaningful when located in their proper place in the head tag.

I don't like the HTTP header idea, 'cause it precludes static content unless you modify the web server. And I don't like external files which are automatically read by the browser because then you run into namespace problems. Keeping everything in the HTML document provides tidy isolation.

If you're pulling the public key from the document to use it to verify the document, you've already lost.

You're using it to verify JavaScript code, not the document. Presuming you also disabled inline JavaScript, you've just shutdown cross-site scripting (XSS) attacks.

Not OP, but I think there's a big difference. HTTPS protects code in transit, but if you can hack the server, you can change the JS code. This proposal would cover that gap.

That's exactly the issue. This is designed to work in conjunction with HTTPS/Tor such that if the server is compromised, unsigned JS payloads cannot compromise the user.

None of the vendors have javascript enabled though, how to disable .js was in the vendor guide and all over their forums as a warning.

This isn't just a "fight the man" anti-FBI idea, it's meant to stop the spread of drive-by malware on legitimate sites too. :)

So you're going to fork the web?

No more than NoScript forks the web.

So if I hack a website what's stopping me from serving javascript with no signature or my own signature?

This was solved for the case of http/https with the "Strict-Transport-Security" header which says that all future communication (for a certain time) will always be over HTTPS. The browser could easily respond to similar "Strict-Signature-Security" header requiring all files to come with a Signature header in the future.

Ah, key management, the bane of any asymmetric crypto system!

My draft specified TOFU; if a site was previously signed and no signature is attached, don't allow any JS. If the signature doesn't match the key cached in the browser, go full noscript.

Wouldn't I typically want my Tor Browser to not record any details about sites I have previously visited? Seems like a lot of work for modestly more security.

Disclaimer: This was an idea that nobody ever showed interest in and therefore I never put forth the effort to refine and flesh out. Feel free to take it and run with it if you have better ideas.

I'm not a lawyer, but AFAIU the defense has an affirmative burden to prove impropriety. Simply casting doubt isn't enough. They're unlikely to ever get this evidence suppressed on 4th Amendment grounds, so at best it's a tactical maneuver for negotiating a plea deal.

In fact, it's probably not even enough if they prove something as shocking as the NSA helping them. The NSA could always say that they stumbled upon Silk Road in the course of their anti-terrorism operations. And then instructed the FBI to hide the origins of the tip. We know for a fact that this happens regularly, and AFAIK nobody has ever been punished.

A court could technically toss the evidence as a way to punish the government for not playing fair, even if the law doesn't require it to be suppressed. That provides leverage for bargaining with the government. But evidence as central as this, in such a high-profile case where the defendant was clearly guilty, means that's unlikely. It's more likely for a smaller case where a court wants to rap the government on the knuckles without letting a really bad guy (from their perspective) go free.

At this point I feel I'm missing something as we now have two legal documents going "Nuh uh!" - "Yuh huh!"

FBI's Explanation[1] states (Page 4, Footnote #5) that the admin himself kept logs explaining that there were frequent IP leaks due to misconfiguration of the web server.

At this point aren't we lead to believe that he showed multiple cases of mismanagement. From this can we not call bullshit on the very definitive declaration by the defense that the webserver was explicitly configured to deny external connections?

    [1] https://www.scribd.com/doc/238844570/FBI-Explanation-of-Silk-Road-vulnerability

Well, that's the thing - they are not providing packet traces even though they were doing a forensic investigation, putting their methodology into doubt. Also, there are no logs indicating that they were successful in what they were trying to do. Even if they were successful, they said that "headers of some of the packets"..." as the source" contained the ip of the server. If it was coming from the TOR network, that would not be the case. Furthermore, since HTTP is a request and response protocol, a request would first have to have been made to the host. A reply would not come without first making the request. That being said, supposedly it was the captcha box. However, that server was only setup to reply to and the backend server. There's no way they would have gotten access to it from the regular web without going through tor, by the methods they describe. You can see the last date the files were modified, and since the "investigator" told what time they accessed the server, we can see that the server was configured not to allow access except through tor. Since he had made a log of ip leaks, I'm pretty sure the guy was on top of security issues, as this is not something he'd overlook.

Considering the FBI's explanation didn't make any sense to begin with this isn't that surprising. How were they getting non-Tor traffic out of the server at all? The IP leaks were never a result of Tor misconfiguration but were instead due to things like exploits allowing examination of PHP variables storing internal information. There is no reason to believe the FBI could have gotten the IP of any of the servers by looking at packet headers as they've claimed.

I have limited experience with nginx, but I believe points 12+13 are technically incorrect -- sites/virtual hosts can be configured in the /etc/nginx/conf.d directory or directly in the main nginx.conf file, not only in the sites-available/sites-enabled directories, as is implied. This makes the conclusions of point 19 and others incorrect.

I don't think it necessarily disproves the conclusions of the document, but it calls into question if the author knows nginx as well as he claims to. Id be happy to be corrected if I'm wrong.

This is in response to various comments to the effect of "I can't believe they didn't log, image, etc properly".

Every organization is made out of people. Each person grew up somewhere, had interests, went to school, etc before joining the organization.

Some organizations attract some types of people more than other types of people.

Do you think the FBI attracts the sort of people who stayed home in front of their computers on most Friday nights during the best years of their lives?


Said differently: you're better at computers than you realize. Shh.

From a legal perspective, why does it matter how the FBI got access to the server or determined it was a Silk Road server? I assume they got a warrant for the server itself, and therefore the evidence found on the server is a candidate for inclusion in the trial.

Yes, there was a warrant, but that warrant must be obtained based on legally obtained information, or else the warrant is invalid as is all information gathered through the execution of that warrant.

Here's an example: Suppose you have drugs in your house, and no one knows that but you. The police may suspect that that's the case, but they need some kind of information that provides probable cause in order to obtain a warrant to search your house.

There are two general categories of ways they could then obtain a warrant: either 1.) they happen to see you carrying drugs or someone else does and reports it or 2.) they break into your house, find drugs, and then request a warrant after the fact to do so.

Now typically option 2 isn't so blatant. More likely is that they break in, find drugs, and then make up some other way that they determined you had drugs. This made up story is called "parallel construction", and it's generally a way to lie and cover up the act of illegally obtained evidence.

If the FBI found the Silk Road IP through an example of scenario 1, then everything's legal. But the FBI's story provides fairly strong indication that scenario 2 is far more likely. If that is the case, then the contents of the server are not admissible in court, nor is anything obtained further down the line from that information.

The result: if scenario 2 is indeed what happened, it doesn't really matter the legality of the actions committed by the defendant were under US law as the authorities have no legal knowledge of said activities. Therefore there is no case, and the defendant must be acquitted.

Thanks to you and the others that replied.

So basically the prosecutors have to explain probable cause to justify the warrant, and that explanation cannot include information that could only be found in the evidence captured as a result of the warrant.

Is there no legal process to ensure that the original warrant request (probable cause justification) is not changed once the warrant is issued? The warrant itself does not have to be public immediately but some aspect of it, like a case# and hash of the document could be preserved as part of the warrant (or embedded in the warrant). I assume these documents form a critical part of the due process.

Parallel construction doesn't usually work that way. In 50+ years of heightened warrant requirements, the police and DoJ have learned lots of tricks. Helped along the way by various legal loopholes crafted by conservative Justices.

A better example (albeit more brazen than usual) would be the Federal government illegally breaking into your home. They find the cocaine. That means they can at least tell the local cops that they wouldn't be wasting their time by investigating you.

Perhaps they stumble upon a note on your kitchen table with the name John Garcia. The Feds might also tip off the police that they have reason to believe you may have working with a John Garcia. Maybe the look at the cellular location logs of your cellphone and John Garcia's. They notice that you're both in an abandoned warehouse on the first Sunday of each month. So they tip off the police that they might want to double their previously non-existent patrols in that area on Sundays.

When you eventually get popped, all the evidence presented at trial looks to be on the up-and-up. An officer patrolling the warehouse district noticed two suspicious men meeting regularly and exchanging boxes. When they ran the license plates, they found that your name came up in the files of some other drug investigation. They had no reason to suspect you personally at the time, but with the suspicious rendezvous they decided to put a tail on you, which doesn't even require probable cause. Eventually, given your pattern of behavior, colored by their expert experience chasing bad guys, they can convince a judge to issue a search warrant.

If their local judges aren't sufficiently pliable, maybe they pop a known drug user who may have crossed your path. They ask him if he knows you. They make sure to tell him that they know you're a drug dealer (the scope of the lies cops can tell is for all practical purposes limitless), and if he tells them about you, they'll be lenient with him. So he lies and says he knows you're a drug dealer. So the cops go back to the judge for a search warrant, telling the judge that an informant has fingered you.

_That_ is parallel construction. The government has very limited resources. It's immensely useful even when all the FBI or NSA does is point the cops in the right direction. It can be so incredibly subtle and indirect that many judges, convinced it happened, still might not consider it to violate 4th Amendment warrant restrictions.

And it's plausible that this is what happened here. The NSA decided to point the FBI in the right direction. They could have even done it through an "informant"--e.g. tipped off a hacker working with the FBI. There are myriad ways to do it. And because the defense has very limited resources, it's almost always impossible to trace the origins of tips all the way back to their source. Eventually the paper trail ends at the footsteps of somebody with a fuzzy memory. What can you do?

Because you can't get a warrant without probable cause. That's why it's illegal for the police to just arbitrarily decide to get a warrant and search your house for drugs because you look like the kind of guy who might have drugs.

And the police can't say "Well, we didn't have a warrant, but we were right so who cares" either.

If they never had probable cause to get images of this server, what gave them the right to a warrant for it?

More importantly, the world needs to know how the heck they did know, so that we all know what the government is capable of.

Ulbricht is still an idiot, and it's not surprise he got caught eventually- but we can't let the police bullshit their way into convicting anyone, even idiots.

If the investigation is not performed in a forensically valid way, then there will always be a suspicion of evidence tampering. The law is like that for a reason. You have to prove that you captured the evidence exactly as it was, and not only you, but everyone who touched it made sure of that fact too. If you can't prove that, the evidence could be tampered with. Planting evidence, or even changing the circumstances under which existing evidence is seen can have a major impact on a trial.

Since a warrant would otherwise involve violating the rights of the person being investigated you typically need to have some evidence that a crime is being committed. The ends do not justify the means in a court of law.

Say the FBI used a power drill to put holes in people until they got the information they want.

Yes, origins of evidence matter.

If they hacked the servers, they've tainted the evidence.

Section IV.A.18 shows an extremely technically versed lawyer that I hope one day becomes a judge. Joshua Horowitz was able to find a contradiction in the SAs testimony based on nginx's configuration.

the FBI lied? no way.

according to the defendant.

It's very important to realize that declarations are almost always written by an attorney, even if they purport to be written by an expert.

This doesn't even purport to be written by an expert. It's literally by the guys defense attorney. They didn't even bother getting an expert to sign off on it.

It's very common to have two highly qualified expert declarations saying the exact opposite of each other.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact