A completely (local) diskless datacenter with iSCSI

Being able to boot a machine from SAN isn’t exactly a new concept.  Instead of having local hard drives in thousands of machines, each machine logs into the fabric and boots the O/S from a LUN exported via fiber on the SAN.  This requires a little bit of configuration on the Fiber HBA, but it has the advantage of no longer dealing with local disk failure.

In OpenSolaris Navada build 104 on x86 platforms, iSCSI boot was incorporated.

If you have a capable NIC, you can achieve the same results of “boot from SAN” as fiber, but without the additional costs of an expensive fiber SAN network.  Think of the possibilities here —

Implement a new AmberRoad Sun Storage 7000 series NAS device like the 7410 exporting hundreds iSCSI targets for each of your machines, implement ZFS Volumes on the backend, and leverage the capability of ZFS snapshots, clones, etc with your iSCSI root file system volumes for your machines.  Even if your “client” machine mounts a UFS root filesystem over iSCSI, the backend would be a ZFS volume.
Want to provision 1000 machines in a day?  Build one box, ZFS snapshot/clone the volume, and create 1000 iSCSI targets.  Now the only work comes in configuring the OpenSolaris iSNS server with initiator/target parings.   Instant O/S provisioning from a centrally managed location.

Implement two Sun Storage 7410 with clustering, and now you have a HA solution to all O/Ses running in your datacenter.

This is some pretty cool technology.  Now, you have only one machine to replace disk failures at, instead of thousands, at a fraction of the cost it would take to implement this on Fabric!  Once this technology works out the kinks and becomes stable, this could be the future of server provisioning and management.

3 Comments

Matt Simmons  on December 31st, 2008

I think you’re right, this very well could be the future of datacenters. I’d be surprised if, in a few years, some of the large co-location companies didn’t start offering this as a service from their SANs (many already offer SAN backups). If you went with a colocation provider who offered this, you’d never have to worry about a failed disk again (assuming the colo company was up to snuff).

Hans van der Made  on January 2nd, 2009

This sounds great, but I am wondering how many zones could effectively be deployed in the suggested setup before the (highly random) I/O load would bring this nice Amber Road filer to its knees, even with the SSDs. It would all be dependant on load obviously, but still. Have you heard of real world setups like this yet?

You may want to have a look at Ben Rockwood’s keynote at the Open Storage Summit a few months ago: Storage In The Cloud. Around 27:25 he mentions centralized storage in virtualized environments, iSCSI enters at around 30:45. You may have seen it already, but I thought I’d mention it anyway :)

Best,

Hans
NL

mike  on January 2nd, 2009

Hey Hans

Thats a good question. From experience, not really much I/O happens because of the O/S itself. 99% of the I/O occuring during normal operation is application driven, not really O/S driven.

Not only that, but the file system cache is going to store much of the frequently accessed data.

You do bring up a going point through. Where is that limit here?

You would most defiantly want to separate your O/S iSCSI LUNs from application storage. If using iSCSI for everything, I would put the application data on a separate subnet, VLAN, etc.

Leave a Comment