Virtually insecure
Discuss this article in the ZeroFlaws forums
Virtualisation is a fascinating subject. Simply by installing a piece of software like VMWare or Virtual PC you can instantly host multiple virtual computers on your one, single physical computer. To the vast majority of software and operating systems a virtual PC is indistinguishable from a physical one. But because the entire virtual PC is stored in files on disk it makes backing up, rolling back and swapping between installations as simple as clicking the mouse. This makes virtualisation perfect for home users who want to try a new o/s or application without risking damage to their physical PC's configuration, or for businesses who want to build large test environments without tying up lots of servers.
VMWare is the biggest player in the virtualisation game, and they host a mean party to prove it. A couple of weeks ago, at the end of February 2008, the VMWorld event took place in Cannes, France, home of the famous movie festival. VMWare hired the bulk of the Palais des Festivals, a massive convention center on the Cannes marina, and filled it with 5 floors of technology, resellers, demo suites and lecture halls devoted to virtualisation. The event lasted 2 nights and 3 days; on the first night they held a smaller party for around 600 attendees at a local casino. The event was fully catered with free food, drink, a mediocre DJ, some free-play arcade machines, some fußball tables and a bunch of Nintendo Wiis hooked up to large plasmas.
That was a pretty decent get-together, but VMWare pulled out all the stops the following night for the main event party. Spread out over two floors and several thousand square feet, the event was split into four themed zones (fire, water, earth and air). With more food and drink than the 4,000 attendees could handle, more arcade machines and - you guessed it - more Wiis, it was nothing short of incredible. The attention to detail, from the ice sculptures and chocolate fountains to the live entertainment on 4 stages, was fantastic. Apparently it was almost as good as the Las Vegas VMWorld that takes place every Autumn. Oh, did I forget to mention this was the Europe-only event and they do it all again in the USA in September?
At this point you might be wondering why this has anything to do with Zero Flaws. This event cost VMWare £millions. Virtualisation is big business. Server hardware and data center space are very, very costly: when most estimates put the average utilisation of servers at around 25%, the idea that a company can consolidate down their hardware and get greater return from fewer servers is an attractive one. Virtualisation isn't just for demo labs, training courses and home users. It's being used as a core component of large, critical, commercial and government IT infrastructures, and when a solution is used in a live environment security becomes of paramount concern. Of course VMWare realise this and ran a security track within their "break out" (presentations, to you and I) sessions. I was an avid follower of this track because I have one fundamental concern about virtualisation: it causes security risk to grow silently and exponentially.
The reason for this isn't immediately apparent, and to understand it we need to look at how virtualisation actually works. Virtualisation software like VMWare adds another layer within your computer's stack of hardware and software. At the very bottom of the stack we have the physical hardware of your PC; the processor, the memory, the hard disk and so on. Above that we have the BIOS, the small piece of code that lets your PC boot up and communicate internally. Layered on top of that we have the operating system such as Windows or Linux. This is what begins to provide user-friendly functionality and a framework for applications to run on. Above Windows, right at the top of the stack, are those applications you use every day - Microsoft Word, Mozilla Firefox, iTunes etc.
VMWare is just like any other software application. It sits on top of Windows or Linux but, using a bit of pixie dust and clever coding known as a hypervisor, creates a whole brand new stack of virtual hardware. On top of VMWare you can imagine a brand new processor, a BIOS, some memory and everything else that's also in your physical PC. Of course none of these devices actually exist; they're part of VMWare's cleverly created abstraction and emulation layer. That doesn't matter to your favourite operating system and applications though. As far as they are concerned you've installed them on some hardware that works just fine. They don't even know VMWare is there.
Because VMWare runs as an application and creates virtual hardware, you can get it to make as many virtual machines as your physical hardware can support. It's perfectly possible to have 3 or 4 virtual computers running on your home PC. On professional server hardware it's not unheard of to run 15 to 20 virtual computers on one physical server. The operation, access and resources of these virtual machines are all managed by the VMWare hypervisor.
If you're new to virtualisation all this might seem a bit like heresy and black magic, but it does work and it's easier than it sounds. If you'd like to learn more there are some references at the bottom of this article.
So for now we've established that virtualisation can improve the utilization of your hardware, is robust enough to work in a corporate environment and will let you run multiple virtual machines on one physical computer. Sounds good so far, so what's the problem?
Think about that last feature for a moment. Using virtualisation you can run multiple virtual machines on one physical computer. Each virtual machine can run the same software and, if properly designed, provide the same performance levels as its physical equivalent. Operationally each of these virtual machines are as good as a physical machine.
Except they aren't individual. They all run on the same physical hardware and are managed by one single piece of software, be it VMWare, Virtual PC or something else. It's just a clever software trick that makes it look as though you have multiple virtual PCs running on your one physical PC. From a security point of view this is bad news. Not only are the individual virtual machines at risk of compromise (as any individual, physical computer would be), but the underlying physical machine is also a potential target.
"Guest escape" is the generic term for an attack on a virtualised system. It refers to an attack, executed within a virtual machine, that breaks out of the virtual "container" and compromises the hypervisor and managing software itself. If you compromise the security of a physical server, at worst you lose one server. With virtualisation, if you compromise the virtualisation software or the operating system it runs on you will compromise 5, 10, 15 machines at once. With a physical server you have one operating system, one set of applications, one set of drivers and hardware to attack and compromise. If you fail (because of security configurations, patches or any other mechanism), that's it. With a computer hosting 15 virtual machines you get 15 different operating systems and application sets to attack, as well as the underlying physical hardware and o/s installation. Only one of those 15 targets needs to be vulnerable and you have a platform for launching the guest escape attack.
Aside from the increase in risk, there's an integrity issue to consider. Because virtual machines are, well, virtual, any communication between them takes place in a virtual manner as well. In VMWare terms, a virtual switch is used to route network data between virtual machines hosted on the same physical server. Until recently there was no way to inspect the data crossing this virtual switch. That meant one compromised virtual machine could attack another across the virtual switch and there would be no real way for a security team to spot it. If we were only using physical servers the data would have to cross a physical switch and could be picked up by security monitoring devices such as an intrusion detection system.
VMWare realised this was a rather glaring problem, not least because auditors were regularly raising concerns about the lack of transparency of inter-VM communications. To address this they enabled a way to attach a security monitoring system to the virtual switch. This was one of the main topics in the TA23 Security Essentials track.
Unfortunately there's a serious problem. The monitor has to run on a virtual machine - you guessed it - hosted on the same physical VMWare installation as the machines it monitors. It's not possible to route virtual switch traffic out to a separate, secure monitoring device. Much like the issue explained in the One and one factor security article, this is a feature that looks, smells and feels like a security mechanism, but isn't. If the underlying hardware and VMWare instance is compromised and the sole security mechanism runs on a virtual machine managed by that same VMWare installation, you cannot trust anything it reports.
During a Q&A session I made this point to the VMWare presenter. There was a general murmur of agreement from the rest of the audience, some of whom complained their auditors and security teams were saying the same thing. The answer I received from Nand Mulchandani, the senior director of Product Management and Marketing at VMWare, was:
There are no exploits for the VMWare hypervisor, and we're confident of our technology
I pointed out that if that was indeed true (and not knowing what 0day exploits are under development right now I can't argue it), it's a dangerous assertion. VMWare is saying their products will always be secure simply because they say so. Bear in mind other hypervisors on the market have been proven vulnerable to attack, including closed platforms like the XBox 360 [click for advisory].
Of course VMWare know they have a serious worry here. I'm pretty sure that there's a team somewhere in VMWare Towers quietly soiling their trousers whilst worrying about an attack on the VMWare hypervisor. It will only take one successful exploit to seriously damage VMWare's reputation, because the outcome will be a lack of trust in VMWare due to a lack of security manageability. If a virus that successfully exploited the VMWare hypervisor was released today there would be a mass panic amongst security professionals.
Why? Simple. A compromise of the VMWare hypervisor and the underlying machine would give an attacker complete control over every virtual machine hosted on that physical hardware. It would be undetectable from within the virtual machines themselves, meaning that VMWare's suggested security monitoring solution is useless. It's a security specialist's nightmare, and until VMWare come up with a solution that allows monitoring from somewhere other than another virtual machine, it won't go away.
Virtualisation is an amazing technology that brings significant benefits, but caution and common sense must be used. Maintaining integrity through physical separation of security mechanisms is essential.
Technical note: Although this article has focused mainly on virtualisation solutions that run within a host operating system, for example VMWare GSX running on Windows Server 2003, "bare metal" virtualisation also exists. Solutions such as VMWare ESX Server don't require a host operating system and can run directly on the hardware itself. In this case the scope of risk is reduced as the option of attacking the host operating system has been removed. That said, VMWare ESX is still just a piece of software - and an extremely complex one - like any other. It's not automatically immune to security flaws. Theoretically if VMWare ESX was compromised it may well be completely undetectable: whereas you can shut down and analyse virtualisation software that runs in a host operating system, you can't do the same with ESX where the software is the o/s.
Although relatively minor vulnerabilities (in that they can only be exploited under certain conditions), there have been a number of full-compromise exploits created for various VMWare products already [click for examples].
I've focused on VMWare in this article, but the principles apply to all virtualisation systems. VMWare wins pole position because of their somewhat cavalier public attitude.
Further reading on virtualisation:
