It has been a while since my Postmortem & Homelab Updates post, which was the sequel to Homelab 2024. So I figured I would write up some of the changes.
The biggest change is due to moving from NY. Because of that, none of it runs in my basement anymore. Instead, the servers now live in colocation at 24shells in Newark, New Jersey.


It is still mostly the same shape, but I'm running a lot more random stuff. There is also a new machine (not pictured above).
Platform
Every machine has a base set of infrastructure which serves as my base platform for operating.
For networking, Tailscale runs on every host. I don't use it much, but for a few use cases, it is great to have. The majority of my connections come through SSH, using Mist as a jump box. I also run fail2ban on Mist for blocking all the bad scrapers.
For a very long time I had no monitoring of my stuff. Observability is my day job, why mess with it for fun? Because I am a sadist? Maybe. Anyways, Grafana Alloy runs on every host and ships logs and metrics to Loki and Prometheus on Rope, with Grafana and Alertmanager on top. I don't really use Grafana much, instead mostly relying on alerts and AI to debug.
For external monitoring, I use the free tier of UptimeRobot which watches the public sites and endpoints.
For my backups, every host runs Borg to Brook, and Brook copies the Borg repository to Google Cloud Storage daily.
Most of my services that need data use either Postgres or Redis on Rope.
Finally, the newest addition: A mail server (docker-mailserver) on Mist, plus Parse DMARC for DMARC reports.
Storage
Brook and Gallon are mostly just disk arrays. But Brook does run Maestral to keep a copy of my Dropbox account.
Media (Rope)
Rope serves most of my media archive. You can think of it as the librarian to my storage boxes. It runs the following software:
- RomM for game ROMs.
- slskd for Soulseek.
- spotify-to-plex to sync playlists into Plex.
- GYB to back up Gmail. I should move this to Brook.
- harvestarr and letterboxd-list-radarr, two small tools for feeding the Servarr stack.
- FlareSolverr for Prowlarr. This doesn't really work anymore, I should take it down.
Plus all of the *arr software mentioned in previous posts.
Websites and apps (Mist)
The following are open source apps I have built and currently run.
- Etu, a journaling app: web and backend (plus sync and tag generation workers) at etu.timeclimbers.com.
- art is a small app for auditing my email and my calendar.
- artgrabber is a small discord bot for inspiring me.
- cambridge, an analog style world clock for my house.
- code, images, videos, linkbot, and roderick.
- gotak, lilguys.love, and robot.villas.
- go.icco.me, which I wrote about in Hosted Go Urls (config).
- acntng, my personal accounting tool.
- A Borg status page and a devcloud.org redirect, both just compose entries in icco/icco.me.
AI (Kiln)
Kiln is a new box with a Tesla P40. By "new", I mean new to me. Like almost everything else I run, it's built using used and recycled parts I get from Craigslist, eBay, Facebook Marketplace, friends, etc. Most computer stuff, especially datacenter equipment, is sold and gotten rid of while it is still in great shape. So I'm happy to refurbish or replace a few pieces to get something greatly discounted.
Anyways, Kiln runs LightLLM, Ollama, OpenCode and OpenChamber (packaged as opencode-server). It gets the same Alloy, Borg, and cron setup as the other hosts, plus nvidia_gpu_exporter so I can watch the P40 in Grafana. I use a mixture of open source models, Claude, Gemini and Jev to serve opencode and any of my sites that might need a little bit of AI.
I hope you found that useful. Or at least interesting.
/Nat