What have you guys used? What's your experience? We've recently started looking into virtualization, mostly for server consolidation. Probably going the VMWare route. However, looking between Server (GSX), Foundation, standard, and Enterprise, I don't really see compelling reasons to go with anything other than Server or Enterprise. Their 'in between options' seem to really lack features and cost justification. Their HA system is pretty miuch a joke and their resource pooling is WEAK at best.

The main thing people talk about ESX vs. Server is performance, but they are talking 2-4 VMs per core on server and 4-8 on ESX (for lightweight applications)

Anyone have thoughts?

I've used GSX. It's pretty sweet. We're going to replace that with xen running on Red Hat 'cause of the cost (none). But, for ease of use, it's pretty hard to beat VMWare. Assuming, of course, you have the budget to throw around.

we use ESX round here. :|

The only one I've used is VMware, and I know a bunch of people who work there. It's great for running something in an isolated sandbox, either to avoid an installation/deinstallation round trip or to address robustness/security concerns. There are many development organizations that probably couldn't function without it, because it's so handy for testing their products.

As far as using it to slice and dice M physical servers into N virtual ones, though, it's a more complicated picture. Let's say that the most cost-effective physical servers have a "capacity rating" of 100. Users/applications often only need 30-70. Virtualization then lets you buy approximately half as many physical servers, in return for a modest increase in complexity (there is some increase no matter what vendors say) and that's cool. Literally, in the sense of thermal issues. Then the real benefits kick in. You can add and remove virtual hosts dynamically, literally with the click of a button. You can move them, possibly overloading some physical servers and degrading performance but not suffering any total-system downtime after a hardware failure. That's all good. In some situations, especially those where an application is already clustered (think database servers), I think smaller servers that consume less power would be a better solution than virtualization, but virtualization suffices and there's that reality of buying the hardware that vendors sell. Hardly anybody's selling server-grade gear at less than 80% of that day's top-of-the-line performance, and I think that's a shame.

Perhaps I should add a bit of perspective on that. The company I work for makes its business out of boxes that follow the "school of piranhas" model rather than the "big bad shark" model. Our individual nodes are not actually all that fast - six cores, 500MHz, single issue. They're a little beefier in the floating-point area because that's what our customers care about, but they're actually rather crappy for the sort of integer computation that an OS guy like me cares about. The trick is that the whole thing - including memory (dual controllers on die), PCI Express and Ethernet controllers, and a connection to our own fast communications fabric - is 14W. Because of that, we can pack a lot more of those into a box and the box draws a lot less power (thus also producing less heat) than anything else that might be considered comparable. Our smallest box is the size of a PC and contains 12 processors (72 cores); our biggest is about a double rack and contains 972 (5832). Getting back to virtualization, what you need when you have a system like that is not so much a way to slice one of those nodes into multiple virtualized pieces but a way to combine sets of them into application-specific groups.

I think that's the way the whole industry will be headed in a few years. Intel's already making noise about 80 cores per chip. If you want to use that much power effectively, why not run separate OS images (perhaps even different OSes) on different sets of cores? What you'll want is not virtualization but partitioning. They present much the same interface to the user, and there are areas where they overlap, but there are also areas where very different technology is involved. In fact, I'll bet some of the companies selling virtualization right now are already thinking about how to do partitioning instead without losing their customers. Virtualization as it currently exists is an artifact of the fact that current servers are bigger than they really need to be, and that's only temporary. As power and heat continue to be limiting factors, the shift will be more and more toward the more/smaller model.

I think that's the way the whole industry will be headed in a few years.

Me too. That's why I took a job that doesn't pay real well, but will be focusing increasingly on virtualization.

Me too. That's why I took a job that doesn't pay real well, but will be focusing increasingly on virtualization.

Are you talking about processor virtualization, or system virtualization? For the reasons I just mentioned, I think processor virtualization is a technological dead end as more physical processors become the norm. OTOH, system virtualization - e.g. making network or storage systems play nice with "multiple personality" hosts whether they're composed of a dozen real cores or a dozen virtual ones - is an area that vendors are just beginning to address effectively and there's plenty of room for growth there.

System virtualization. Setting up the multiple personality hosts, and making the network and storage play nice with them. I think we're going to see that evolving quite a bit, and I want to already be there, not trying to play catch up. The savings in space & power by packing multiple virtual hosts onto a stack of boxes or blades makes it well worth while for an awful lot of companies -- especially savings in power. It can cost an awful lot to refit a building when you're at maximum capacity and need to grow.