I ran into an odd issue today on one of my colocated servers. The server is a Core 2 Duo with 8 GB of RAM, running a thinned down version of CentOS. The host is configured to fit all virtual machine memory into RAM, and the host s able to reserve 6400 MB of RAM.
There are 3 VMs on there; VM1 has 3072MB of RAM allocated to it, VM2 and VM3 each had 1536MB allocated. I was adding some memory-intensive software to VM3 so I upped it's RAM allocation to an even 2GB (2048MB).
Shortly after that change the load on the host jumped from ~0.30 to ~1.8. A handful of services on VM3 stopped starting and when I opened the Task Manager (it's a Windows server) the task manager process was pegging the CPU. If you search for "taskmgr.exe 100% cpu" or "taskmgr.exe peg cpu" you'll find that most of the cases are spyware-related. In this case it turns out that the host was starved for memory. 'top' on the host showed that I wasn't swapping any RAM to speak of, but I only had ~256MB free.
Tasks: 89 total, 1 running, 88 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.8%us, 58.5%sy, 0.0%ni, 39.5%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 7457468k total, 7194396k used, 263072k free, 24652k buffers
Swap: 2048184k total, 120k used, 2048064k free, 6716580k cached
I reduced VM3's RAM back down to 1.5GB and magically all of my problems went away.
Tasks: 95 total, 1 running, 94 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.0%us, 5.1%sy, 0.0%ni, 92.5%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 7457468k total, 7194588k used, 262880k free, 22956k buffers
Swap: 2048184k total, 124k used, 2048060k free, 6687792k cached