remote-gpu-trainer
An Agent Skill for the whole life of a DL experiment — RUN a job (on a GPU you own or rent), VERIFY that the number is real, and DELIVER it as organized, single-source, reproducible figures and tables. Its deepest part is still remote-GPU operations (AutoDL, RunPod, vast.ai, Lambda, Paperspace, the Chinese platforms 恒源云 / 矩池云 / Featurize / 揽睿星舟, bare SSH, Slurm, Kubernetes — one instance or a fan-out of many), now wrapped in the full run → verify → deliver arc.
What this is, and what it isn't. "AutoDL" here means the autodl.com GPU-rental platform, not AutoML or NAS. And this is an Agent Skill — a
SKILL.mdwith reference docs and script templates, not a CLI or an SDK. It sits on top of each platform's API and the DL workflow, and captures the operational and judgment knowledge they leave out.
One skill, three phases of one real workflow:
- RUN — get a long GPU job to start, survive, and finish, then get the result off the box. On a
machine you own there is no meter; on a rented one the core insight is that you are a short-term
tenant on someone else's machine — so the job is to detach the work, make the result outlive the
instance, and stop the meter safely. The things that genuinely vary between platforms (stop-vs-destroy
billing, machine-locked volumes, whether
/rootsurvives a power-off, acceleration proxy vs HF mirror, spot grace windows) are pushed down into one profile per platform. - VERIFY — is this number a bug, a real effect, or noise? A surprising result is a hypothesis, not a fact to report. Platform-agnostic.
- DELIVER — organize the result so every shipped number / figure / table is a deterministic function of one immutable evidence layer; provenance and cross-document consistency are locked by mechanism, not by a human remembering to update three documents. Platform-agnostic.
Two stances run through VERIFY and DELIVER: user sovereignty (the science — seed count, which samples,
whether an aux channel exists — is your call; the skill organizes and discloses a tradeoff once, then
stops nagging) and audit → disclose, not enforce (an integrity issue must surface with the conclusion
it affects, but the skill never blocks you from shipping). Mantra: disclose it, or don't claim it.
flowchart TD
TASK(["A DL experiment on a GPU you own or rent"])
TASK --> MATCH{"description keywords<br/>match the task?"}
MATCH -->|skill activates| HUB
HUB["<b>SKILL.md</b> — the always-loaded hub<br/>RUN / VERIFY / DELIVER router + the operating spine"]
HUB --> RUN["<b>RUN</b>"]
HUB --> VER["<b>VERIFY</b><br/>is the number real?"]
HUB --> DEL["<b>DELIVER</b><br/>single-source, reproducible"]
RUN --> RL["references/run-local/ + profiles/local.md<br/>a box you own — no meter"]
RUN --> RR["references/run-remote/ + profiles/×7<br/>a box you rent — billing, spot, teardown"]
RUN --> TR["references/training/ ×8<br/>when the run breaks: OOM · hang · NaN · convergence"]
VER --> VM["references/verifying/<br/>14-probe methodology · collapse · smoke"]
DEL --> DM["references/delivering/<br/>EVIDENCE.json · append-only runs · reconcile"]
Contents
Why this exists · How it differs · Architecture and layout · Install and deploy · What's inside · Scope · Verification status · Disclaimer · 中文简介 · Contributing · License · Citing
Why this exists
Renting the GPU is the easy part; so is starting a training run. The costly surprises are everything
around the job — and they don't stop when the job finishes. A box you "stopped" that keeps billing. A
checkpoint that printed synced but never actually wrote, because the disk ran out of inodes rather than
space. A download that hangs behind the wrong mirror. A terminate that takes the only copy of a week's
training with it. Then, after the run: a 1.2 % "win" that is really a misconfigured baseline; a leaked
test split that survives every shallow check; a headline number left stale in the slides because the fix
only landed in the paper. None of this is in any platform's API docs, and you usually learn it after you
have already paid — in money, or in a claim you have to retract.
This skill puts that knowledge where an agent can use it: operating principles that say why each step matters, a six-phase lifecycle where every phase ends in a check you can run, one profile per platform with the exact commands, a probe-by-probe methodology for deciding whether a number is real, and an evidence-layer architecture that makes a stale or hand-typed number physically impossible to ship. It spends its attention on the things that cost money, data, or a retraction.
How it differs
The infrastructure orchestrators (SkyPilot, dstack, Modal) own or abstract the box and price-shop across Western clouds. They are good at that, this skill does not compete with them, and the two go together — let SkyPilot or dstack move the box, then use this skill to make your code resume correctly so recovery continues the run instead of restarting it. Two things set this apart:
- It works on the raw rented instance you already have, including the platforms they skip. AutoDL and the Chinese platforms, and cheap bare-SSH rentals, where the day-to-day work is disk budgeting, inode caps, mirror stalls, cgroup OOM, spot grace windows, and teardown you cannot take back — none of which the orchestrators model.
- It does not stop at "the job ran." They hand you a finished run; this skill continues into is the number real (VERIFY) and is the deliverable reproducible from one evidence layer (DELIVER) — the part of the experiment lifecycle the infra tools do not enter at all. That is the half most likely to cost you a correction after submission, and it is the half they leave entirely to you.
Architecture and layout
The layout uses the Agent Skills idea of progressive disclosure: a small hub that is always loaded, with the deeper material read in only when a phase needs it. RUN is platform-specific at the edges (one profile per platform owns every concrete path, proxy, billing verb, and spot rule) and invariant at the core; VERIFY and DELIVER are platform-agnostic and run the same whether the job trained locally or on a rental.
The remote-RUN third has a six-phase operational spine; each phase delegates its platform details to the active profile and ends in a check you can run:
flowchart LR
P0["0 · audit<br/>df -i · cgroup · GPU"] --> P1["1 · ssh + creds"]
P1 --> P2["2 · CPU smoke<br/><i>before you rent</i>"]
P2 --> P3["3 · detached launch"]
P3 --> P4["4 · durable monitor<br/>(four-layer)"]
P4 --> P5["5 · verify + teardown<br/><b>Iron Law</b>"]
The folders map onto RUN → VERIFY → DELIVER:
remote-gpu-trainer/
├── SKILL.md # the hub: RUN/VERIFY/DELIVER router + the operating spine
├── references/ # platform-agnostic knowledge, loaded on demand
│ ├── run-local/ # RUN, a box you own: env-hygiene · launch · multi-gpu · local-oom
│ ├── run-remote/ # RUN, a box you rent: principles · 6-phase checklist · monitoring ·
│ │ # ssh · spot-resilience · china-network · parallel-ablation ·
│ │ # multinode · gotchas_universal (U1–U43)
│ ├── verifying/ # VERIFY: methodology (14 probes) · representation-collapse ·
│ │ # smoke-hidden-failures
│ ├── delivering/ # DELIVER: principles · data-architecture (EVIDENCE.json) ·
│ │ # evidence-manifest-schema · figures · delivery-gate
│ ├── training/ # the DL-training debug layer (8 files; local/remote-agnostic):
│ │ # OOM · NCCL-hang · NaN · throughput · ckpt · domain · convergence · data
│ ├── companions.md # optional companion skills + the no-companion fallback
│ └── self-improvement.md # how the skill captures new gotchas without corrupting itself
├── profiles/ # one file per platform — the only place concrete specifics live
│ ├── _schema.md # the shared 8-field contract every profile fills
│ ├── local.md # a box you own (no meter, no teardown clock)
│ ├── autodl.md # deepest, battle-tested
│ ├── runpod.md vastai.md lambda.md paperspace.md
│ ├── china.md # 恒源云 / 矩池云 / Featurize / 揽睿星舟
│ └── generic-ssh.md # bare SSH / Slurm / K8s / Colab-Kaggle
├── scripts/ # parameterized, runnable templates
│ ├── run_one.sh.template run_queue.sh.template health_patrol.sh.template
│ ├── mem_monitor.sh gpu_health.sh reap_vram_zombies.sh
│ ├── aggregate_to_fs.sh download_loop.sh setup-china-mirrors.sh
│ ├── verify_local.py # load-and-verify each artifact before any teardown
│ ├── reconcile.py