Node - Athena

For the full cluster architecture, see the Home Server Infrastructure Overview.

At a Glance

One-line summary: Athena is the compute-only node that runs per-project VMs and GPU workloads side by side on a single machine.

What Runs Here

Athena’s base unit is “one VM per project → dependency isolation with Docker inside the VM.” It is designed so a package conflict or runtime problem in one project never affects another.

Past project VMs

Deployment pipeline (for service VMs)

A GitHub Actions self-hosted runner sits inside an Athena VM, so commit → build → container swap happens automatically without opening any build port externally.

Storage usage

• VM OS disks: local to Athena
• Project data / models / datasets: mounted from the Hades NAS (over WireGuard)
• Container data: regenerable data is excluded from backup; only long-term data is copied to the NAS backup path

Key Decisions

- Why XCP-ng (Xen)

Athena is a 20-core machine, which exceeds the free ESXi license’s 8-vCPU limit. I evaluated Proxmox and XCP-ng as alternatives and, since Hades already ran ESXi, chose XCP-ng because its management model is closer to ESXi. Keeping both nodes’ operations (snapshots, console access, VM migration) similar reduces operational overhead.

• Trade-off considered: Proxmox is KVM-based with strong ZFS integration and plenty of Korean resources, but I preferred unifying on XCP-ng over paying the cognitive cost of moving away from the ESXi-style workflow.

- Why Docker on top of VMs (instead of Docker alone)

Docker alone already isolates dependencies. The reason for wrapping it in a VM as well:

• Guaranteed resource limits: vCPU/RAM/disk caps are set at the VM level so one project’s OOM or runaway disk usage cannot affect the whole host.
• GPU passthrough: dedicating a Tesla P100 per VM (rather than container sharing) is easier for managing driver/CUDA version conflicts.

In return, I accept some inefficiency from double virtualization (hypervisor + Docker). As services grow, moving to a single plane such as VM consolidation / Kubernetes is planned as the next step.

- Why JupyterHub

Analysis/experiment workflows are often notebook-based, and I did not want to copy notebooks back and forth between laptop and server. JupyterHub lets Athena’s GPU resources be used directly from the notebook environment — that is the core reason.

Operations & Incidents

Routine operations

• Take per-VM snapshots manually, before and after major changes
• Keep project data on the NAS mount path, included in Hades’s daily backup
• WireGuard keys are separated per node — Athena exposes only 1 external WireGuard port
• The VPN VM monitors UPS (APC SMC1000) status and, on power loss, propagates the alert to the other VMs

Power protection (UPS)

• Device: APC UPS SMC1000, 1000 VA output
• Load basis: the Lenovo X3650’s hardware can draw up to ~1.8 kW at peak, but a UPS that sustains that level is too expensive to be practical. Instead I sized for the real-world peak of a normal workload, ~400 W, rather than the extreme of GPUs at full load. The UPS’s goal is not uninterrupted operation but securing enough time to shut down safely.
• Target protection window: configured to hold 10+ minutes at 400 W → a window to save in-progress work and shut down safely on power loss (the window shrinks under full GPU load, so on an outage signal, training jobs are stopped first)
• Detection & propagation: the UPS is connected over USB to Athena’s WireGuard / VPN VM. When the UPS raises an external-power-cut alert, the VPN VM receives it and propagates it to Hades, Daedalus, and the other nodes → the whole cluster enters a coordinated safe-shutdown

Incident — forced shutdown from an outage & GPU kernel corruption (the reason the UPS was added)

Situation: a power outage in the area around Athena forced a hard shutdown. External power was restored about 45 minutes later.

How I noticed: the boot after power was restored looked fine, but as a GPU workload came up, nvidia-smi failed to recognize the device and CUDA calls failed. Tracing the kernel log, I found the NVIDIA driver’s kernel module was corrupted.

Response:

1. Removed and reinstalled the NVIDIA driver at the same version, rebuilt the kernel module, and GPU workloads recovered.
2. Concluding that the forced shutdown itself had to be prevented, I added a UPS — installing the APC SMC1000 (1000 VA).
3. Attached the UPS to the VPN VM over USB and built the outage → VPN-VM detection → propagation to other nodes structure.
4. Organized per-VM shutdown commands to gracefully stop GPU workloads and training jobs on outage detection.

Lesson: power is the essential foundation that makes the whole system operable. Separately from data backup, a “sudden outage” itself can leave state damage on accelerators like GPUs. And a single node’s outage response is only meaningful when it propagates across the whole cluster.

Result: the same failure has not recurred since the UPS was added, and the node has run without a reboot for ~254 days (~8 months) since the 2025-10-04 boot. There were a few brief outages in that span, but the UPS absorbed them and the system stayed up. (Based on the current uptime reading, with no time-series logging.)

Incident — environment drift

Over long-running operation, unused Docker images and old VM snapshots accumulated and the disk filled up. After that I added “clean up once every two months” to the operational routine.

Limitations & Next Steps

What’s missing now

• No GPU scheduler, so allocation relies on manual agreement
• Weak self-hosted-runner isolation — worked around by a policy of not accepting external PR builds
• No monitoring/alerting → VM state is checked by a person directly

What I’ll change next

• Visualize GPU/memory usage with Prometheus + node_exporter + nvidia_gpu_exporter
• Strengthen isolation by running the self-hosted runner as a single-use container
• Evaluate a lightweight scheduler (Slurm or Ray, etc.) for queuing heavy training jobs

Photo

Previous Next

Korean version: Node - Athena