Spark Memory Thermal Ops
Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Spark Memory Thermal Ops when an AI agent needs a reusable SKILL.md workflow for this job: Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.
When not to use it
Skip Spark Memory Thermal Ops when the task is outside the writing category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/spark-memory-thermal-ops # Save SKILL.md to ~/.claude/skills/spark-memory-thermal-ops/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/spark-memory-thermal-ops/SKILL.md. For a shared project, commit the folder at .claude/skills/spark-memory-thermal-ops/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Spark Memory Thermal Ops skill for this task: Sizing a training run against the 128GB pool. Confirm any required inputs or files before making changes.
What supports this recommendation?
84/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 92/100
- Practical detail
- 92/100
- Source evidence
- 63/100
- Source signals
- 80/100
Model compatibility evidence
- claudeuntested
No model-specific signal or recorded compatibility test was found.
- chatgptuntested
No model-specific signal or recorded compatibility test was found.
- geminiuntested
No model-specific signal or recorded compatibility test was found.
- copilotuntested
No model-specific signal or recorded compatibility test was found.
- llamainferred
The skill text mentions Llama or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.
- perplexityuntested
No model-specific signal or recorded compatibility test was found.
- mistraluntested
No model-specific signal or recorded compatibility test was found.
- grokuntested
No model-specific signal or recorded compatibility test was found.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# Spark Memory & Thermal Ops
DGX Spark's GB10 chip has one 128GB unified memory (UMA) pool shared by CPU and GPU, and a sustained power ceiling well below its rated figure. Both break discrete-GPU assumptions: headroom isn't what `nvidia-smi` reports, and a run that starts fast will slow down mid-job with nothing misconfigured. This skill covers planning memory headroom, working an actual OOM, and watching thermals across a long job. For launch-time failure modes (ABI mismatches, flash-attn, playbook breakage), see `spark-training-gotchas` — this skill assumes the job starts.
## Common Issues Quick Reference
| Situation | Do this | |---|---| | Planning headroom before launch | Budget against `free -g`, not `nvidia-smi` — see UMA Memory Model | | Job OOMs on unified memory | Work the OOM Ladder in order: flush, then batch/pack, then method downgrade | | Throughput drops mid-run | Check the power/temp log before assuming a config bug — see Thermal Monitoring | | Trainer + inference server both wanted | Run one at a time — see Concurrent Workloads |
## When to Use This Skill
- Sizing a training run against the 128GB pool before launch — will this model, method, and batch/pack combination fit. - A run OOMs mid-load or mid-step and the remediation order matters — what to try first, second, third. - Watching temperature and power during a multi-hour job, deciding whether a slowdown is thermal throttling or something else. - Planning to run a trainer alongside an inference server (vLLM, Ollama) on the same box.
## UMA Memory Model
Spark has no separate GPU VRAM — the GPU and CPU share one 128GB pool. Two consequences:
- **`nvidia-smi` and `cudaMemGetInfo` underreport pressure — or report nothing at all.** Both report CUDA-allocator-visible memory, not the pool's actual state — a box can show headroom in `nvidia-smi` and still OOM, because page-cache and mmap'd pages the allocator doesn't see consume the same pool. On some driver/setups, the memory query returns `[N/A], [N/A]` outright instead of a number — a script grepping for a numeric value there gets nothing, not a misleading undercount (see `spark-training-gotchas` gotcha G3).
- **Model load is a transient peak, not the steady state.** Loading safetensors weights mmaps the file, then copies into CUDA tensors — for a window during load, both the mmap'd pages and the CUDA copy count against the pool at once. A model that fits while training can still OOM during load if headroom was sized for the post-load footprint instead of this doubled transient.
Plan and diagnose with `free -g`, not `nvidia-smi`:
```bash free -g | awk 'NR==2 {print "free:", $4, "GB"}' ```
Rule of thumb: take that free figure, subtract a few GB for OS/driver overhead, and budget against the result — not the 128GB spec number. The worksheet in `references/uma-accounting.md` accepts parameter count, dtype, and method as input, and returns a memory estimate to compare against known anchors.
### Planning Sequence
Before launch, work through these in order:
1. Read `free -g`; subtract OS/driver overhead for the budget. 2. Estimate weights + optimizer + gradients + activations from `references/uma-accounting.md`. 3. Compare against the closest anchor (70B QLoRA, 27B LoRA, 9B full FT), not the estimate alone. 4. If the estimate is close to the budget, start with shorter packing or a smaller batch — cheaper than hitting the OOM Ladder mid-run.
### Example: Sizing a 70B QLoRA Run
A sanity check of the worksheet formula against the ≈40GB anchor:
```python params = 70e9 weights_gb = params * 0.5 / 1e9 # NF4, step 1 adapter_gb = 0.5 # step 5, negligible total_gb = weights_gb + adapter_gb # + activations print(f"{total_gb:.0f}GB before activations") ```
Weights alone land near the ≈40GB anchor — a plan estimating far above that for the same model class is a signal to recheck dtype and method.
## The OOM Ladder
When a job OOMs on unified memory, work this ladder in order. Each step is more disruptive than the last — don't skip ahead: **reducing batch size is never step 1.**
1. **Flush the buffer cache.** Page cache from a previous run or a large dataset read often accounts for GB of the "missing" headroom. This costs nothing but a rerun and doesn't touch the job's configuration:
```bash sync; echo 3 > /proc/sys/vm/drop_caches ```
Needs root; a between-run reset, not a mid-training step. See `spark-training-gotchas` (gotcha G3) for the full diagnostic behind this step.
2. **Reduce batch size or packing length.** Only after a flush fails to free enough headroom, cut batch size or packing length — the first step that changes what the run does. Prefer packing length first; it drives activation footprint more directly at long context.
3. **Downgrade the method: bf16 LoRA before QLoRA.** If flushing and shrinking batch/pack still OOM, drop the method a tier — bf16 LoRA is next, not the reverse. QLoRA's bitsandbytes dequantization buffers are transient CUDA-side allocations that can OOM before an equivalent bf16 LoRA run would, even though QLoRA's steady-state footprint is smaller. A QLoRA OOM is not proof the model doesn't fit.
Fall back further (smaller model, multi-Spark) only after all three steps and the job still won't fit.
## Thermal Monitoring
Multi-hour runs push into Spark's sustained power ceiling, well under the rated figure — expected platform behavior, not a symptom to explain away:
- Sample temperature and power alongside the training logs, not after a slowdown is noticed — every 30-60 seconds correlates a throughput drop with a thermal event. Keep the CSV output format `assets/thermal-sample.sh` writes, so timestamps line up against the log:
```bash bash assets/thermal-sample.sh 30 thermal.log ```
- **A sustained ~100W power draw is the platform cap, not a configuration bug.** Don't re-tune batch size or precision to "fix" a plateau that's the box behaving normally under load. If temperature climbs while power stays flat under the rated 240W figure, that's the signature to recognize.
- Log throttle events explicitly instead of letting a run silently slow down unrecorded. A run whose per-step time doubles two hours in should show that in the log, correlated against the thermal sample at that timestamp. Full throttling diagnostics: `spark-training-gotchas` (gotcha G4).
## Concurrent Workloads
Because the 128GB pool is global, eviction happens without either process's logs showing an OOM:
- The one-heavy-job rule applies to **uncapped or near-capacity** workloads — an uncapped trainer and inference server (vLLM, Ollama) compete for the same pool. A small, capped workload doesn't: a <4GB LoRA fine-tune coexists fine alongside vLLM capped at `gpu-memory-utilization<=0.5` — check the other process's cap, not just its presence, before stopping it.
- Inference servers evict trainer pages silently under uncapped/near-capacity contention, and vice versa — neither logs an error, so a slow run or lost KV cache is a contention symptom to check for. Stop unrelated *uncapped* servers before a long or full-pool run.
Check for GPU-resident processes first:
```bash ps aux | grep -E 'vllm|ollama|trl|axolotl' | grep -v grep ```
This procedure complements `spark-training-gotchas` (gotchas G3, G4, G6) — that skill covers launch-time failures; this one, the running job.
Memory math worksheets: `references/uma-accounting.md`.
Intended uses
- Sizing a training run against the 128GB pool
- A run OOMs mid-load or mid-step and the
- Watching temperature and power during a
- Planning to run a trainer alongside an
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.