It’s been a few days since I wrote the first article on the Home Lab that will be hosting an Enterprise Lab Cluster. Work schedule hasn’t prevented me from moving forward on it, my time has been well spent. Here’s where I am at this point. In my infinite wisdom (and my innate inability to just keep things small and compartmentalized for my personal projects of love), I started thinking about how I would be using Openstack once this series was complete. I realized I was going to need more hard drive space (for block storage) and rather than build a small cluster up front only to redo my work, I decided to up the ante and deploy with 2 TB of MIRRORED storage in my initial configuration.
This poses the interesting question every enterprise architect goes through when faced with deploying SSD vs. Mechanical storage? In a perfect world where money is no object (and it rarely is), I thought about how I would deploy this. There’s no doubt that 2 TB is plenty of space for my personal cluster, but it was a question of performance. In an ideal world, I would go with SSD and put all of my fears to rest, but the price of SSDs even at ~$1/GB is still a bit on the high side for me. Mechanical storage when striped will give me the performance I need, but it’s not the performance I want.
I decided to conduct a bit of research into what would be acceptable. A few hours of reading about the more technical aspects of mechanical hard drive performance and how they would potentially perform in my environment led me to the conclusion that risking a couple of hundred dollars on Hybrid SSD drives would be worth taking. Why Hybrid drives? They offer more performance than a mechanical drive with some of the benefits of an SSD based on their caching algorithms.
Why Hybrid Cinder Storage
I’m not an expert on HDD seek algorithms, nor am I an expert on HDD firmware design, but I am smart enough to know how a seek algorithm can dramatically affect storage performance. Hybrid drives are supposed to cache / pre-fetch information that is most commonly accessed for serving up via NAND. That’s actually a very desirable thing in block storage. Consider a VM that mounts a cinder volume and runs a database. The database blocks will be accessed more often than most other blocks on the storage volume. The hard drive algorithms should be smart enough to prefetch those specific blocks of information giving me near SSD performance. Well, that’s the concept anyway. It was worth investing in 4 x 1 TB drives for the investment.
My thinking behind deploying hybrid drives is that the drives will adjust continuously to provide performance to the VMs that need it most at any given time. Whether or not this is true will be an interesting experiment. If it doesn’t, I’ll just pull the drives, replace them with SSDs and build a small 3 TB RAID-5 NAS as a backup appliance. So no love lost and the hardware won’t be wasted.
After doing my research, I decided to purchase 4 Seagate Hybrid drives. They have 64 MB of cache and I have more confidence in the Seagate firmware than other manufacturers.
To UPS or not to UPS?
The lab is relatively self-contained. It’s a 2U chassis, a switch, and a few cables. As far as labs and clusters go, it’s relatively compact. The only issue; It’s Supermicro. As anyone who is familiar with SuperMicro gear can attest, it’s like buying a jet engine. No such thing as silent for these beasts. In other words, I’m not exactly going to put this in an area I frequent. Where to put a small Jet Engine in your house? Where anyone would put a jet engine; your garage.
A few months back, I installed a nice shelf with concrete anchors into my garage for the purpose of hanging “heavy stuff” one day. It would seem that a jet engine would be a perfect item to hang on the wall in the garage. It can be as loud as it wants, it’ll be out of the way, and it’ll be accessible for those moments when I’ll feel the need to tear into something.
There are two issues with servers in garages; connectivity and power quality. (3 if you count cooling, but I don’t care about that and I’ll explain why.) Let’s address connectivity first. I don’t need 10 Gbps to the rest of the house where I’ll be living. I only need Gigabit and Internet. Now, even if I had Google Fiber (and I don’t), I would still only need gigabit access. The storage backbone is relatively self-contained. It’s only going to communicate with the other nodes (all located in the garage and connected via Infiniband). So in other words, I could uplink to my switch in the office via 1 Gbps (or even less) and I’d be more than fine. I chose to go the route of Powerline Ethernet and after much review and testing of various manufacturers and technologies, I settled on TP-LINK TL-PA6010KIT AV600 units. They work well and will easily traverse different legs of a breaker box. (I currently have these deployed throughout the house). I’m pushing about 200 Mbps through them so that’s fine for my needs now. I just ordered another set of these little buggers and I’ll just join the existing network I currently have configured.
This leaves the problem of Power. I’m a fan of Dual Conversion UPS units. I don’t trust line-interactive at all. I also don’t trust surge protectors as they’re pretty much a massive waste of time and money. If you’ve lived in Florida, you know that power quality is complete crap. I didn’t want to drop the cash for a Dual Conversion UPS, but I didn’t want to go so cheap as to buy a surge protector; I settled on Line conditioners.
The Supermicro chassis has dual redundant Power Supplies and they’re 932 Watts each. I purchased two APC Line-R LE1200 line regulators to give me the happy medium of quality power conditioning at a decent price. I’ll be using one unit for each of the power supplies and hooking up the ancillary equipment to them as well. The benefit is they’re really small as well, do the job, and are very affordable.
Cooling the Home Lab and Why I don’t care about it
With such a large investment and build-out, why wouldn’t I cool it? Simple; economics. It’s more expensive for me to cool the garage over 3 months than it would cost for me to simply replace the components that fail. I bought Supermicro remember? They’re cooled by 80000 horsepower fans! lol.. you get the idea. There’s plenty of ventilation in the chassis, they’re connected to IPMI units that will allow me to turn them on remotely when I need / want to use them and for the most part, they’re going to be powered down 20 hours a day.
What about backups? When it’s time to run the backup job, I’ll just remotely turn them on (automation yay!), run the backup job, then power them off again ;P I’m an enterprise guy remember? There’s a thing called orchestration (and we’ll discuss this later in another series).
If a component fails due to heat, it’s cheaper to replace it (gogo ebay) than it is to pay for cooling 24/7/365. The most expensive part in the whole set-up (besides my time) is the RAM (which has a lifetime warranty). The drives are cheap ($94.00 each at the time of this writing), the CPUs (Xeon X3440) are only $75.00, and the power supplies for the chassis cost me $50.00. No matter how I do the numbers, the cost of replacing one component will be significantly lower than the cost of cooling my garage.
Remember what I said about planning? This is why you think things through before you make your purchases. Selecting Supermicro was not a mistake; it was planned for a couple of reasons:
- Remote management including power on/off
- Great Ventilation (lots of fans)
- Redundant Power Supplies
- High Density (4 Nodes per 2U Chassis)
- Rock Solid Hardware
Every one of the points I just stated played a crucial role in my initial planning process. It enables me to be cautious of my power bill, cooling becomes a non-concern, access to hardware is easy, it’s friendly to confined spaces, and it will last me long enough to get to the next major upgrade.
As you can see, planning this project has been a major ordeal typical of what a planning exercise for an enterprise deployment would look like. (Although, there would be no garages and Powerline Networking involved). Next steps? I’m waiting for the equipment to be delivered. It should be in by end of week and I’ll be moving on to actual Infrastructure deployment and configuration in my next post. See you then!