My Home Lab

I’ve always had this itch to set up a few of my own servers at home (A.K.A a home lab) so I can run a few random side projects and services without constantly worrying about cloud costs. While I still pay for cloud compute for things that need more consistent uptime, there’s something nice about having computers at home where I can experiment and make mistakes without worrying about run-away costs. Plus, consumer-level computers are surprisingly powerful, and cloud VMs can get expensive if you need a VM with a little more oomph. I’ve never found the space in previous apartments to stuff a bunch of computers in a corner. However, after my recent move granted me a dedicated office, I decided it was finally the opportune time to build my home lab! I think reading this Home Lab Beginners guide around the same time finally convinced me to put something together.

When I finally decided I was going to put together a home lab, I knew I wanted to run not just one or two servers but a cluster of them. I wanted to learn more about container orchestration and the nitty-gritty details of keeping a bunch of computers happily buzzing along. Each individual server didn’t need to be extraordinarily powerful, just powerful enough to run a handful of docker containers.

On the other hand, I wanted to ensure the home lab was practical and supported the development and hosting of my side projects. The goal was to mimic the deployment experience typically provided to me by cloud offerings (like GitHub Actions or other DevOps/GitOps products), where I could push some code to a repo and have it built and deployed to my cluster. Despite the upfront cost of purchasing the equipment for the lab, it gave me peace of mind that I could easily spin up a dozen services without worrying about increased cloud costs. Not worrying about costs enables me to deploy as many services as I need constantly and frequently. When it came do developing personal apps and projects, deploying a whole new copy of my stack - including a frontend, api, and database - on each pull request was no longer a problem.

I had a few tertiary considerations as well when putting everything together. I knew the cluster would sit next to me as I worked from my desk. For this reason, I had 2 more requirements:

• Keep the entire cluster quiet and cool - I need to be able to sit next to it without getting distracted or needing to turn up the A/C
• Have the home lab look good - it’s sitting in my office, so I need to not hate it.

What does it look like now?

I currently run a cluster of 7 computers in my home lab! It is still a work in progress, but I’m pretty happy with it so far. I’m storing everything in a 6u studio rack, which is pretty compact.

Ok, it’s not exactly pretty, but it’s all contained.

The top shelf consists of the 6 intel nucs, all powered together with a meanwell psu and networked together with a single 8-port switch. There’s enough physical space for 2 more nucs if needed, but only one remaining ethernet port. I haven’t figured out if I want to add more nodes just yet anyways.

The second shelf below it currently only holds a Dell Optiplex Micro.

I’ve got two Noctua fans in the back to keep the air flowing through it, but honestly, it doesn’t get hot enough to warrant it. Even with all the fans off, the rack stays pretty cool. I’ve installed a patch panel + brush panel on the back to keep any cables coming in and out of the rack organized.

Hardware

Most hardware was bought used whenever there was a good deal on eBay or r/homelabsales. Here’s a quick list of the major components.

• Studio Rack
• 6 Intel NUC NUC5PPYH equipped with 8GB ram and 2.5" misc ssd - This came off a crazy ebay sale where I was able to get a bunch of these and more for $250 shipped.
• Netgear 8-Port unmanaged ethernet switch
• 1 Dell Optiplex 3080 Micro (i3-10100T, 16gb ram, 512 nvme ssd + 4tb 2.5" hdd)
• TP Link AX1800 Wi-Fi 6 router
• StarTech 8 Outlet 1U Rack Mount Power Strip
• 2x 10" Rack Shelfs
• 2x 1U Vented Rack Blanks
• 4U Vented Rack Blank - Placed on the back of the rack for fan mounting
• 1U Brush Panel
• 24 Blank Keystone Patch Panel
• Coax Keystone Jack Inserts
• Cat6 Keystone Jacks - Don’t recommend these, they were too wide to fit in the patch panel next to each other
• Rack Rail Set - I wanted to add a second set of rails to the back
• PC fan power supply
• Noctua PC fan controller
• 2x Noctua NF-A9x14 HS-PWM Slim Fan
• A bunch of Monoprice Slim Ethernet cables for small spaces
• Short Extension Cords
• Misc modem I had laying around

Software

All computers are running Ubuntu Server. I’m currently using Docker swarm mode as my container orchestrator and running the following services:

• Portainer - Provides a UI to manage my docker swarm. An alternative to just using the docker stack deploy and a nice way to see which nodes are running what containers
• AdGuard Home - Provides ad blocking to any devices that use it as a DNS server. I avoid using it as a DNS server to my router and just have it as opt in on a device-by-device basis
• BOINC - I contribute excess compute to several research projects:
  • Rosetta at home - Contributing compute to help design new proteins or predict protein’s tertiary shape.
  • Einstein at home - Contributing compute to help discover new neutrino stars from telescope data.
  • World Community Grid - A collection of different research projects including cancer marker and TB research.
  • Here are my latest contributions:

  

• Drone CI - Self-hosted CI I use to deploy projects that are pushed to my self-hosted gitea instance.
• Gitea - A self-hosted github alternative
• Docker Registry - A private container registry to push containers to
• Photostructure - A google photos alternative and a way for me to manage my humongous image library (over 2 terabytes)
• Speedtest Tracker - Automatically run speed tests every hour to make sure we’re getting the speed we’re paying for and to track how unreliable our service actually is.
• Personal projects and other things (Mere Medical, Hapi FHIR Server)

To expose some of my self-hosted projects to the internet, I’m using a $4 a month Digital Ocean droplet, TailScale to enable the droplet to act as a VPN gateway to my home lab, and NGINX to securely forward requests from the droplet to my home lab.

Update 11/26/22

After load testing some of my self-hosted projects using this setup via wrk and loader.io with my Digital Ocean droplet acting as a VPN gateway, I was really surprised to see that the total amount of requests per second (rps) handled by one of my projects was really low. For context, this was a simple static website being served via NGINX, so it should be pretty easy to hit a high rps.

The command I primarily used to benchmark was

wrk2 -t12 -c250 -d60s -R250 --latency <URL>

When running a benchmark with wrk, my homelab would struggle to serve this static site with more than 12 rps through the VPN gateway. However, running the same benchmark on my local network (bypassing the VPN gateway) brought the rps back to 250.

After some investigation, I found one possible bottleneck. While I’m not confident this was the only problem, it seemed like TailScale can be too heavy a process to run on the anemic 1 vcpu given on a $4 Digital Ocean droplet I was using for my VPN gateway. Temporarily upgrading the droplet to a dedicated 4cpu droplet seemed to fix the issue (was able to hit 250rps easily with no timeouts over a minute). However, I didn’t feel like paying >$40 a month just to expose my side projects to the internet. Instead I migrated to Hetzner, where I was able to get their CPX11 2vcore and 2gb ram vm for ~$5. This let me easily hit my expected mark of 250 rps.

However, 250 rps is half my (admittedly arbitrary) target goal of 500rps. While I don’t really need crazy performance, I figured 500 rps would be enough throughput in case I ever needed to handle a random burst of traffic.

After moving to Hetzner, I played around with scaling the VM further to see if more compute would further fix the bottleneck. Alas, it seems like this was not the case, as further scaling the VM didn’t get me any appreciable increase in rps over 250. This wasn’t too surprising, I already knew 250 was the best I had done, even on my internal network, For further improvement, I’ll need to look at my homelab itself. That is a investigation for another day.

Update 11/28/22

Turns out, I just needed to update my NGINX config. It wasn’t a CPU bottleneck, I didn’t need to update my ulimit, or play around with sysctl. I just needed to bump up the number of worker_connections in my NGINX config. With this, I managed to breach >1000rps on my local network (Could be way more, I stopped testing after that.)

This was the change in my /etc/nginx/nginx.conf

events {
    worker_connections  4096;
}

It seems like the worst bottleneck is my upload bandwidth on my home network which is around 10 MBps. The difference in rps between stress tests on my local network and through the VPN gateway is several orders of magnitude different.