Four nodes, five old GPUs, and an AI with root.
A small home lab: one custom-built server and three retired office desktops running Proxmox, a storage server full of secondhand disks, a local AI stack on 2016-vintage workstation cards, and an AI assistant that does most of the administration over SSH from a ThinkPad running Omarchy. Nothing here is enterprise gear. Everything is real, measured, and still running.
Details are kept general on purpose: no addresses, hostnames, account names, or anything that would help someone poke at the network. No links out, either.
The room: no rack, no enterprise gear
The heart of the cluster is a custom-built tower: a 16-core Ryzen 9, 64 GB of RAM, and a SAS host bus adapter feeding six spinning disks to the storage server. Around it sit three retired office desktops that were headed for recycling. Each became a Proxmox node; two carry a pair of workstation GPUs each.
Proxmox lets one physical computer host many virtual machines and lightweight containers. If one node needs maintenance the others keep going, and every guest is snapshotted before a change and rolled back if the change goes wrong. Everything is managed from one ThinkPad, described below; it is swapped daily with a work machine, so two docks, not the laptop, own the monitors.
Also in the room
- Two spare iPads on the wall run kiosk dashboards: cluster health on one, guest-network activity on the other. The older one can't run a modern browser, so a container renders the dashboards to PNG and serves a plain auto-refresh page.
- One 600 W consumer UPS carries all four nodes, the router, and the satellite dish. Monitors and docks sit on the surge-only side so they don't eat runtime.
- Five Quadro M2000s. A 2016 Maxwell card: 4 GB of VRAM, 75 W, single slot, no power connector. Cheap, quiet, and still enough for an 8-billion-parameter language model or CUDA speech-to-text. Persistence mode keeps an idle card at about 48 W instead of 110 W.
The admin laptop runs Omarchy
A ThinkPad T14 (8-core Ryzen 7 PRO, 14 GB) running Omarchy, an opinionated Arch Linux desktop built around the Hyprland tiling window manager. It is the one machine the human and the AI assistant share: the assistant's command-line tool lives in a terminal on the laptop panel, while Proxmox, the dashboards, and the wiki spread across a 3 × 2 wall of six screens driven by two DisplayLink docks.
- Keyboard-driven tiling. Hyprland with Omarchy's Lua configuration, monitors matched by panel serial so the six-screen layout survives a dock replug, a software cursor to keep DisplayLink from flickering. Stock Kanagawa theme; the only additions are a VPN bar widget and a low-battery sound.
- Built for the cluster. A text-mode Proxmox client, a SPICE console viewer, wake-on-LAN, a cluster monitor, push-to-talk dictation, fingerprint login.
- Locked down and rebuildable. Host firewall, keys-only SSH, every secret in a GPG-encrypted password store the assistant reads by name, mesh VPN for the road, battery capped at 80%. Every change since the fresh install is an 18-step rebuild checklist in the wiki.
- Sudo stays with the human. The assistant has root on the servers and never elevated rights on the laptop. That password is the one thing it is not given.
The nodes and what runs on them
One flat gigabit LAN, no shared storage, nightly backups to the NAS. Almost everything is an LXC container; only the storage server and a retired laptop are full VMs. Allocations are what each guest is given, not what it uses.
| Node | Hardware | Runs |
|---|---|---|
| Node 1 custom build | Ryzen 9 5950X, 16 cores 64 GB · 500 GB NVMe 1 × M2000 SAS HBA | TrueNAS VM 4 vCPU · 16 GB, six disks passed through. Jellyfin 8 vCPU · 4 GB, the household media server: GPU transcoding at about 20× real time, 17 free live channels, intro skipping. Workflow automation 4 GB: a daily pricing refresh for a hobby site, an AI-read inbox that turns email into records, chat-bot intake, a GPU-deal watcher that pushes alerts to a phone. Git server for infrastructure scripts, site code, this page, and the encrypted secret store. Metrics and dashboards for every node; also renders the wall iPad pages. Cloud-drive mirror, nightly, with 180 days of history. Retired laptop VM 6 vCPU · 8 GB: a cracked-screen laptop imaged physical-to-virtual so everything on it lives on. |
| Node 2 office desktop | Core i5-8400, 6 cores 16 GB · 240 GB SSD | Media automation 4 vCPU · 4 GB: a Docker stack for requests, subtitles, VPN-only downloads with a kill switch, and a worker re-encoding the library to 1080p one title at a time. Runbook wiki: 10 books and about 80 pages on how every piece works and how to fix it, written mostly by the AI. Web analytics for the public sites. Push notifications to phones from any script. Household budgeting. A second CMS for a not-yet-public road-safety site. |
| Node 3 office desktop | Core i5-6500, 4 cores 32 GB · 120 GB SSD 2 × M2000 | Local language model 4 vCPU · 12 GB: Qwen3 8B split across both cards, so private email and meeting transcripts never leave the house. WordPress 16 GB, the CMS behind the public sites. Never exposed: a crawler exports it to static HTML and a deploy tool pushes that to the CDN. Also an upload inbox, a chat bot, and a small ledger. |
| Node 4 office desktop | Core i5-6500, 4 cores 16 GB · 120 GB SSD 2 × M2000 | Speech-to-text 3 vCPU · 6 GB: whisper.cpp with CUDA behind a small HTTP API, for meeting notes and subtitles. Price API: a home-grown Python service that identifies collectible items and pulls market prices from four providers, for a hobby site. Tunnel connector, the outbound-only link to the CDN. Honeypot, the decoy from stop four. |
Three more containers mined a CPU-friendly coin on idle cores. They have been off since the UPS swap; the thermal guard still knows how to pause them.
A day in the life
- Backups, then the mirrorEvery container and VM to the NAS, then the one-way cloud-drive sync.
- Price refreshFresh market data for every item in the CMS, a static re-export, a deploy, and an emailed report.
- WatchingThermal guard every 20 s, metrics every 15 s, the honeypot always, the AI-read inbox whenever mail arrives.
- Caps liftDaytime download caps come off so bulk transfers run while nobody is on a video call.
Storage: five disks from 2012 and one from 2023
The NAS is a TrueNAS VM on node 1 with its disks passed through individually, so the VM owns them but the host still reads their SMART health. The main pool is RAIDZ1 across five 2 TB enterprise SAS drives, all built in early 2012 and now at 45,000 to 59,000 power-on hours: five to seven years of continuous spinning, each.
The first-ever long SMART self-tests ran in October 2026 and came back clean, but one drive had been growing defects. A spare 4 TB SAS drive from 2023 was reformatted from 520- to 512-byte sectors, hot-plugged into the VM, and swapped in with the pool live. The resilver moved 6.5 TB in about six hours at roughly 200 MB/s with zero errors. Capacity stays at 2 TB per member until all five are 4 TB. Nothing is offsite yet, which is the honest weak spot and the top of the roadmap.
What lives on the pool
- Nightly backups of every guest (vzdump, zstd), kept 7 daily / 4 weekly / 3 monthly. Restore proven end to end.
- A 1.25 TB archive of open-weight AI models, checksum-verified and pinned to exact commits. More in stop five.
- The media library: a few hundred films and about 1,750 TV episodes.
- A nightly mirror of a cloud drive with 180 days of dated history.
Lesson learned the hard way. The first vzdump --all tried to back up the NAS VM's six passthrough disks (10 TiB) and deadlocked the guest. PCIe passthrough is excluded from backups automatically; raw disk passthrough is not. Every passthrough disk now carries backup=0, and the NAS backs up as a 64 GiB boot disk in 29 seconds.
Network, power, and the things that keep it safe
Satellite in, nothing out
Internet arrives by satellite, behind carrier-grade NAT, through a small OpenWrt-based router. Midday it does 175–215 Mbit down; evenings can sag to 15–45. Upload is the real constraint at 10–14 Mbit, so the router shapes it with cake in five tiers: laptops first, phones, the cluster, everything else, the media stack last as bulk. Shaping download made things worse on this link, so download is capped at the application layer instead: 1.5 MB/s by day, unlimited overnight.
There are no port forwards, and under CGNAT there couldn't be. The few things that must be reachable from outside, an upload endpoint for a hobby site and an analytics collector, ride an outbound tunnel to a CDN. The public sites are static files at the CDN's edge, so the cluster being off never takes a site down. Administration from outside the house goes over a mesh VPN, with the router as a subnet router.
Power
The UPS is small for the load, so the cluster idles around 60% of its capacity. If the mains fail, the UPS daemon waits 90 seconds and shuts the whole cluster down in order: the three desktops in parallel, then node 1's containers, the laptop VM, the NAS, and node 1 itself. A thermal guard on every node pauses heavy work at 82 °C, stops the hottest guest at 90 °C, and powers the node off at 95 °C. The storage VM is never auto-stopped.
Defence in depth
- Cluster firewall with default-drop inbound; only an allow-list of admin addresses reaches the Proxmox web UI, SSH, and console ports.
- SSH keys only on all four nodes and the router. Every new container gets the admin key before anything else.
- Real TLS certificates on every node, issued by DNS challenge so no node needs a public address.
- A honeypot: a decoy container with fake login services nothing legitimate touches. Any connection raises an email alert.
- Isolated guest Wi-Fi for TVs, iPads, and IoT, with per-device exceptions for the two dashboard iPads.
- Secrets in a GPG password store, versioned in the private Git server. The AI reads a secret by name; the human never pastes one into chat.
The AI stack
Local language model
Node 3's two M2000s (8 GB VRAM combined) run Qwen3 8B split across both cards at about 7 tokens per second, and Qwen3 4B on one card at about 11. Context is capped at 8k tokens. Maxwell is too old for modern inference engines, so it is llama.cpp and GGUF only. Enough for the jobs it has: parsing email into structured data, summarising meetings, smoke-testing workflows.
Speech-to-text
Node 4 runs whisper.cpp built from source for sm_52 with CUDA 12.8 (a 32-minute compile), medium model quantised to 5 bits, at five to seven times real time. The obvious choice, faster-whisper, ships no Maxwell kernels; a prebuilt Docker image was 30 GB and also failed. Build it yourself, it turns out.
Meeting recorder
One hotkey on the laptop starts a recording; pressing it again sends the audio to the whisper service, chunks the transcript through the local model, and writes a Markdown note with summary, decisions, actions, and open questions. A 40-minute meeting takes about 5.5 minutes to transcribe and 10 to summarise. After two runaway overnight recordings, it stops itself at 75 minutes.
Vision in the loop
For a hobby site, phone photos of small collectible items are uploaded through the tunnel. OpenCV finds every item and matches fronts to backs by grid cell; each goes to a cloud vision model, a cheaper one first and a stronger one only when the first is unsure. Prices are fetched, draft entries written, and a human approves the batch from a chat command. The first batch was 66 items from a handful of photos.
The weights archive
Sixteen open-weight model repositories, 1.25 TB, sit on the NAS: embedding models, a large speech model, and current open LLMs from 7B dense up to 120B mixture-of-experts, every file SHA-256 verified and pinned to a commit. The downloader resumed across power cuts and a satellite link that caps near 17 MB/s; the set took about eleven days. None of the large models fit the current GPUs. The archive exists so that when bigger cards arrive, the models are already on the shelf.
What's not here. This page only describes what is running today. The GPU node under construction, the storage rebuild, and the rest of the hardware plan are on the roadmap.