
CPU cache-based communication between two co-located virtual machines - j_s
https://github.com/IAIK/CJAG
======
brendangregg
I noticed the paper tested an Xeon E5-2670 on EC2, which is old (Sandy Bridge;
eg, m2's) and pre-dates Intel Cache Allocation Technology (IntelCAT):

[https://software.intel.com/en-us/articles/introduction-to-
ca...](https://software.intel.com/en-us/articles/introduction-to-cache-
allocation-technology)

Support for IntelCAT was added to Xen two years ago. Eg:

    
    
        [root@vmm-psr vmm]# xl psr-hwinfo --cat
        Cache Allocation Technology (CAT):
        Socket ID       : 0
        L3 Cache        : 12288KB
        Maximum COS     : 15
        CBM length      : 12
        Default CBM     : 0xfff
    

And to Linux 4.10 a couple of months ago:
[https://lwn.net/Articles/694800/](https://lwn.net/Articles/694800/)

While I suspect IntelCAT could disable this side channel, I'd personally
hesitate before turning it on for everyone on the cloud. Does it out weigh the
performance cost? ... Can customers opt-in to this?

~~~
j_s
Yes it was specifically called out in someone's tweet I'm trying to find but I
believe this was mentioned in the paper.

Edit: My mix-up on the paper!

 _Malware Guard Extension: Using SGX to Conceal Cache Attacks (Extended
Version)_ |
[https://arxiv.org/pdf/1702.08719.pdf](https://arxiv.org/pdf/1702.08719.pdf)
(2017)

 _Recently, Intel introduced an instruction set extension called CAT (cache
allocation technology) [51]. With Intel CAT it is possible to restrict CPU
cores to one of the slices of the last-level cache and even to pin cache
lines. Liu et al. [83] proposed a system that uses CAT to protect general
purpose software and cryptographic algorithms. Their approach can be directly
applied to protect against a malicious enclave. However, this approach also
does not allow to protect enclaves from an outside attacker.

[...]

Instead of using Intel CAT on the operating level it could also be used to
protect enclaves on the hardware level. By changing the eenter instruction in
a way that it implicitly activates CAT for this core, any cache sharing
between SGX enclaves and the outside as well as co-located enclaves could be
eliminated. Thus, SGX enclaves would be protected from outside attackers.
Furthermore, it would protect co-located enclaves as well as the operating
system and user programs against malicious enclaves._

[83]
[https://www.researchgate.net/profile/Yuval_Yarom/publication...](https://www.researchgate.net/profile/Yuval_Yarom/publication/291830462_CATalyst_Defeating_Last-
Level_Cache_Side_Channel_Attacks_in_Cloud_Computing/links/56a6b0d408aeded22e3544ff.pdf)

~~~
brendangregg
this paper?
[https://cmaurice.fr/pdf/ndss17_maurice.pdf](https://cmaurice.fr/pdf/ndss17_maurice.pdf)
where is IntelCAT mentioned?

------
spullara
One totally practical use case for this is to ensure that none of the virtual
machines that you have allocated on your cloud provider are on the same
physical hardware. This would be extremely useful for reliability purposes. I
wonder how hard it would be to build something like that on top of this
library?

~~~
j_s
With virtual machines I control, I don't have to use covert channels?

[https://cseweb.ucsd.edu/~hovav/dist/cloudsec.pdf](https://cseweb.ucsd.edu/~hovav/dist/cloudsec.pdf)
(2009)

Edit: _we describe an empirical mea-surement study focused on understanding VM
placement in the EC2 system and achieving co-resident placement for an
adversary. To do this, we make use of network probing both to identify public
services hosted on EC2 and to provide evidence of co-residence (that two
instances share the same physical server). In particular, we utilize nmap,
hping, and wget to perform network probes to determine liveness of EC2
instances_

 _The hard disk contention channel was used in Section 6 for establishing co-
residence of instances._

~~~
Florin_Andrei
That assumes the provider will disclose hardware info to you, which is rarely
the case.

~~~
j_s
(I apologize in advance if I've missed recognition of your authority on this
topic; as an outsider it appears you are wrong.)

Are you speaking generally or specifically addressing a shortcoming of the
linked paper which demonstrates a methodology to determine whether or not two
AWS VMs are on the same physical host?

After a brief review I still don't see any mention of "hardware info" you
mention as assumed being available.

Edit: If something has come up since 2009 with this specific paper, I'd
appreciate the heads-up!

------
scott_s
They also have an Network and Distributed System Security Symposium (NDSS)
paper with a title that brought a smile to my face:
[https://cmaurice.fr/pdf/ndss17_maurice.pdf](https://cmaurice.fr/pdf/ndss17_maurice.pdf)

~~~
redrabbyte
I suspect this might also make you smile then ;)
[https://www.youtube.com/watch?v=yPZmiRi_c-o&feature=youtu.be](https://www.youtube.com/watch?v=yPZmiRi_c-o&feature=youtu.be)
, if you haven't seen it already

------
darkengine
Reminds me a lot of this DEF CON talk:
[https://www.youtube.com/watch?v=7X772EBdvnM](https://www.youtube.com/watch?v=7X772EBdvnM)

He managed to stream video across VMs by doing a similar cache timing trick,
which brought a round of applause from the crowd. Very cool stuff.

~~~
redrabbyte
I guess you could call it an evolution, because memory deduplication is not
always enabled anymore (research like this might have played a role here ;)

he talks about that @35:27 in the video

------
jwilk
[https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wron...](https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wrong_with_insert_x86_instruction_here)

------
amenghra
Does this have any practical applications? You might get a perf win, but the
likelihood of being collocated is small.

~~~
cowboysauce
I think at this point it's mostly of interest to security researchers and
malware writers. It's not hard to imagine malware using this technique to
transfer data out of an infected VM in a difficult to detect manner. If you
can infect enough VMs it might be worth it. Also, this type of technique could
be interesting to government employed malware writers.

~~~
sillysaurus3
Another interesting way to exfiltrate data is via DNS. Few networks block DNS,
so you can set up a protocol on your own DNS server to transfer data.

The bandwidth is pretty decent, too.

~~~
cmdrfred
I'm still surprised how well this classic attack works. I've seen $100k IDS
and firewall packages not detect this.

I use this personally. Works behind captive portals sometimes too:
[http://code.kryo.se/iodine/](http://code.kryo.se/iodine/)

------
vgddvh
Explanation anyone?

~~~
wolf550e
Watch this:
[https://www.youtube.com/watch?v=IT6PFSRrvsU](https://www.youtube.com/watch?v=IT6PFSRrvsU)

