A month and a half ago, I discussed setting up a homelab for the purpose of running Openstack for all of my readers that actually have an interest in doing so. So here’s my formal Blog post on doing so.
For what it’s worth, it took me a few months to get it together (being a daddy full time and holding down a job is difficult).. OK.. The reality is it was boating season and I spent every weekend out on the boat with my daughter. It comes with living in Florida.. lol.
Well, maybe the question is really why build a lab and why should Openstack be a part of it? The answer is simple. If you want to learn, you might as well learn something that will actually help you gain employment. Remember when I wrote a book about helping the unemployed? Well, this is considered my way of helping those of you that are still unemployed (or just wanting to actually get a decent wage that will allow you to put food on the table without stressing out about making ends meet!) Single parents, take note!
So what is Openstack? It’s not a virtualization platform like Xen, VMWare, KVM, or Hyper-V. It’s a Control fabric platform that allows you to automate (provide) the many services associated with a virtualization platform. Ever hear about XaaS (Where “X” is insert whatever technology you want to name)? Openstack is IaaS or simply, “Infrastructure As A Service”. It allows you to put together a fully-blown automated VM provisioning platform that will orchestrate all the bits of setting up a truly self-provisioning Virtualization Environment. Why is this important? Two reasons.
1. It allows your users to actually instantiate (big word.. also means create) a virtualized server
2. It keeps the end-users happy by allowing them to avoid bothering the IT people to “create a server.”
Virtualization without a proper control fabric is like ordering a grilled cheese sandwich and having to grill it yourself. Openstack is not an easy technology to learn. In fact, it’s a royal pain in the ass. I guess that’s why people like me are sought after to do this for companies.
So here’s the deal. I know Openstack, but I noticed many of you don’t. I’ve decided I wanted to build a new (improved) home lab and I’m going to do lots of Openstack based research and experimentation on it. It seemed like an appropriate opportunity to write about and also help a bunch of you as well 🙂
Selecting the Hardware / Sizing your Lab
So I decided I needed / wanted to install Openstack. Why? Because the more you work with something, the better you get. I decided I wanted to do some “super cool stuff” and test some ideas I had with Openstack. So an Openstack installation was what I was looking to size the hardware for.
A basic modern Openstack architecture requires a minimum of 3 machines:
- 1 Controller
- 1 Network Node
- 1 Hypervisor Node
What can you realistically do with something like this? Uhm… pretty much nothing. Yes, I can instantiate a VM, but beyond that, it’s pretty much useless. I added another Hypervisor Node for the purpose of managing a multi-node configuration. This would allow me to test a very basic Openstack installation while being able to test VM recovery and migration practices. While not completely dysfunctional and useless, it allowed me to test one of my theories (which I will not divulge right now).
OK.. So this makes the installation a bit more useful, but it’s still relatively useless in my book. What to do? Double it. Add another four machines and now I can have two Openstack clusters each with dual Hypervisors. A bit more useful. It will also allow me to do fun stuff like test WAN technologies and recovery practices.
So the reality to me? I needed 8 nodes for two Openstack installations to allow myself to do anything even slightly resembling an Enterprise installation (and allowing me to develop and test methodologies for recovery of Openstack across a Geo Distributed “enterprise”).
OK.. So I’m almost there, but not quite. I wanted to implement my lab to be as close to what an Enterprise might have. I’m not talking about an enterprise on a budget, but a TRUE enterprise installation. Let’s add a storage / management network. As anyone who has been in the enterprise space can attest, Gigabit ethernet is never sufficient on the storage network. Let’s do 10 Gbe! OK.. One problem; it’s expensive and I’m already kissing 8,000.00 for my initial setup. I’m not adding a bunch of network gear that will more than double the cost. What to do? Enter Infiniband. IPoIB or Infiniband over IP allows me to get the sweetness of high bandwidth, low latency links while using the traditional IP stack!
Used Infiniband Gear can be had on the used market (see eBay ppl) and the cards themselves will look like a traditional ethernet interface to Linux (and Windows with a bit more prodding). So Infiniband it was / is for the management network. Interestingly enough, Infiniband also allows me to do some other cool stuff like add high bandwidth interconnects to *uhm* a SAN? Kinda neat, huh?
Putting it all together, I started looking for Hardware. Here’s what I came up with:
2 x Supermicro 4 Node 2U servers with Dual Redundant Power supplies.
Each node consisted of a Xeon Processor with 32 GB of RDIMM ram, 3 Hard Drive bays currently populated with 1 SSD 250 GB drive and the other two empty (for now).
Resulting in: 16 Cores or 32 Threads per chassis.
The Supermicros all have built in IPMI controllers, one PCIe Slot, 32 GB RAM, 2 Gigabit Ethernet and VGA support. Not too bad for a little lab. Effectively, each chassis will be what I will call an Openstack Micro Cluster.
For Infiniband, I bought a Used Cisco Topspin chassis with 16 ports. This effectively gives me plenty of expansion and allows me to plug in up to 4 Micro Clusters should I ever grow to that size. (with me, you never know).
So off to eBay to purchase my equipment!
The long road
I get the chassis ordered, found some Xeon Processors and even some RAM! Also picked up the Infiniband hardware and some cables. Life is good right? Wrong.
Nothing’s ever easy if you’re in tech. The more Enterprise your solution, the more complicated it gets. I purchased Dual Ranked RDIMMs only to find out the particular brand of RDIMM I purchased doesn’t work with the X8SIT-F boards I got. Upon speaking to the ebay reseller whom I acquired the chassis from, that particular board is super picky about Manufacturer RAM and type. You’d think Quad Rank RDIMMs would work based on what the manual stated right? Nope.. Not going to happen. Ugh. 3 days / 24 hours of research and calling around later and even a ticket to Supermicro support led to very little information.
I finally gave up and did a search for “X8SIT-F memory” and came up with a few options. Unfortunately, they were very expensive.
I ended up buying: Supermicro RAM for X8SIT-F for those of you that need it. It’s not cheap, but at least I know it will work.
Also picked up a Topspin switch and some Cisco Infiniband cards. I’ll probably replace them with Mellanox Connect-X cards later on down the road, but for now, they work.
Picked up some Sandisk 250 GB SSD Drives for each of the 4 nodes right now. because the price is right and I wanted enough IOPs to make the systems relatively quick. After all, time is money right? I’ll add a few more SSDs when I get it all working perfectly or I’ll just build convert one of the 4 nodes into a more robust SAN once I get it all working well. That means 1 TB SSDs for the whole kit and kaboodle.
Now.. It would seem everything is in place right? Just slap it all together and start installing “stuff” right? Wrong.. As luck would have it, the Supermicro Hot Swap caddies are 3.5 inch and the drives are 2.5 inch. Clearly, I needed an adapter that would work well and still line up the connectors. Luckily, I knew this was going to be an issue and I’ve addressed this problem at a prior customer. Please note, not every drive adapter will work. In fact, only one that I know of actually does. Wanna guess which one? lol.. I don’t expect you to. They’re not cheap though.
The only adapter I’ve ever found that will position a 2.5″ drive properly in a 3.5 inch caddy is the Icy Dock EZ Convert caddy and they’re not cheap. They’re about $13.00 each! A far cry from the cheap $2.00 adapter from China. It’s cheap when you buy one or two.. It’s another thing when you have to buy 24 of them. Guess how many I bought? lol.
Now, it would seem I’ve cleared the hurdles of getting the machines up and running right? Uhm.. no.
I put the kit all together and the machine starts up. I look at my DHCP leases and figure out the proper IPMI IP address. Connect with the default User / Pass combination (ADMIN / ADMIN). Guess what? Passwords are changed. Yay! More BS to deal with! lol. What good is an IPMI if I can’t access it right?
No problem.. Just plug in a keyboard and monitor and reset it right? Nope. 1. I don’t have a USB keyboard lying around (I threw them all out) and 2. Can’t reset the IPMI password from BIOS! lol. Does it get any better? haha.. of course it does!
So anyways, I ordered a keyboard / mouse from Amazon and 15 min. after I order it, I decide that waiting for it to come overnight is not something I want to do. So off to Tiger Direct (the retail store) and I pick one up for $40.00
1 hour later, I’m back with keyboard, but still no IPMI password. After some research, the only way to reset the IPMI is with a utility and a bootdisk CD. Well, as you know, the Supermicro chassis don’t have CD-ROMs and I don’t want to go back out. Let’s just make a boot USB with all the utilities on it.
Now, as you all know, I’m lazy. I want this to be quick. So I just created a cute little USB Boot Image that automatically resets to default the IPMI password and account without user intervention. In other words, insert USB and Boot, then power-cycle when it tells you to. (My old DOS skills from 20 years ago actually came in handy lol)
I’ll write a separate post for that later with a link to the USB image I created so all of you out there with IPMIs you need defaulted can save yourselves the pain I experienced.
2 Hours later, I’ve got a base Ubuntu install on all 4 nodes and the servers humming along.
Openstack configuration is next 🙂 That will be the next part of this series.. I’ll try to write that up in the next few days, but for what it’s worth, I think I’ve made a serious amount of progress!
If you want to follow along, you can buy the same Supermicro Chassis I bought. They’re about $250.00 each and just add CPU, RAM, and HDD.