Cicd
CI/CD reference for NeMo AutoModel — pipeline structure, commit and PR workflow, CI failure investigation, and common failure patterns.
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Cicd when an AI agent needs a reusable SKILL.md workflow for this job: CI/CD reference for NeMo AutoModel — pipeline structure, commit and PR workflow, CI failure investigation, and common failure patterns.
When not to use it
Skip Cicd when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/cicd # Save SKILL.md to ~/.claude/skills/cicd/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/cicd/SKILL.md. For a shared project, commit the folder at .claude/skills/cicd/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Cicd skill for this task: Use Cicd when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
69/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 74/100
- Practical detail
- 68/100
- Source evidence
- 63/100
- Source signals
- 70/100
Model compatibility evidence
- claudeuntested
No model-specific signal or recorded compatibility test was found.
- chatgptuntested
No model-specific signal or recorded compatibility test was found.
- geminiuntested
No model-specific signal or recorded compatibility test was found.
- copilotuntested
No model-specific signal or recorded compatibility test was found.
- llamauntested
No model-specific signal or recorded compatibility test was found.
- perplexityuntested
No model-specific signal or recorded compatibility test was found.
- mistraluntested
No model-specific signal or recorded compatibility test was found.
- grokuntested
No model-specific signal or recorded compatibility test was found.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# CI/CD
## Commit and PR Workflow
### Commits
All commits require DCO sign-off:
```bash git commit -s -m "feat: add new recipe for Qwen2" ```
If sign-off is missing on a recent commit, amend it:
```bash git commit --amend -s ```
### Pull Requests
Follow the PR template in `.github/PULL_REQUEST_TEMPLATE.md`. Every PR should include:
1. **What**: concise description of the change. 2. **Changelog**: bullet list of user-visible changes. 3. **Pre-checks**: confirm linting, tests, and sign-off.
PR title format: `[{areas}] {type}: {description}` (e.g., `[model] feat: Add Qwen3 VLM support`)
See `CONTRIBUTING.md` for the full PR workflow, area/type labels, and DCO requirements.
### Branch Naming
Use descriptive branch names prefixed with your username or a category:
``` username/feat_add_qwen2_recipe fix/gradient_clip_nan ```
## How CI Is Triggered
CI is triggered on `push` — **not** on `pull_request`. A bot called `copy-pr-bot` controls when CI runs.
**Mechanism:** 1. When a PR is opened, `copy-pr-bot` watches for a trust signal. 2. Trust is established in one of two ways: - All commits on the PR branch are **GPG-signed** by a verified NVIDIA contributor → bot triggers automatically. - An NVIDIAN posts `/ok to test <commit-sha>` as a PR comment → bot triggers manually for that SHA. 3. Once trusted, `copy-pr-bot` copies the PR's code into the remote branch `pull-request/<number>` and pushes it, which fires CI.
**Consequences:** - CI never runs on untrusted pushes — external contributors always need `/ok to test`. - The running workflow branch is `pull-request/<number>`, not the author's feature branch. - Pushing a new commit does **not** automatically re-trigger CI unless the commit is signed or `/ok to test <new-sha>` is posted.
## Pipeline Structure
``` lint-check └── cicd-container-build ├── unit-tests-core ├── unit-tests-diffusion └── functional-tests (L0 always; L1 with needs-more-tests label; L2 on schedule) ```
CI test scripts live in `tests/ci_tests/`. These are executed in the CI pipeline and should not be run locally unless reproducing a CI failure.
## CI Failure Investigation
### Locating the PR from a CI Branch
```bash # Extract PR number from branch name (e.g. pull-request/1234) PR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?<=pull-request/)\d+')
gh pr view "$PR_NUMBER" --repo NVIDIA-NeMo/Automodel gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Automodel --name-only gh pr checks "$PR_NUMBER" --repo NVIDIA-NeMo/Automodel ```
### Investigating a Failing Job
1. **Review the changeset**: `gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Automodel` 2. **Identify the failing job** from `gh pr checks` output. 3. **Fetch job logs**: ```bash gh run list --repo NVIDIA-NeMo/Automodel --branch "pull-request/$PR_NUMBER" gh run view <run_id> --repo NVIDIA-NeMo/Automodel --log-failed > run.log ``` 4. **Scan logs in chunks** — log files can be large, never load them whole: ```bash wc -l run.log tail -200 run.log sed -n '1,200p' run.log ``` 5. **Cross-reference the changeset** against the failing step.
## Common Failure Patterns
| Symptom | Likely Cause | Action | |---|---|---| | CI never started | Commits not GPG-signed and no `/ok to test` | Post `/ok to test <full-sha>` on the PR | | Lint job fails | `ruff` violation | Run `ruff check --fix . && ruff format .` locally | | Unit tests fail | Code regression or missing import | Run failing test locally; check the PR diff | | Functional test (L0) fails | Integration breakage | Check GPU runner logs | | DCO sign-off missing | `git commit` run without `-s` | Amend: `git commit --amend -s` | | Multi-GPU tests fail silently | `CUDA_VISIBLE_DEVICES` not set | Set `CUDA_VISIBLE_DEVICES` explicitly | | `torchrun` port conflict | Multiple processes sharing a port | Pass `--master_port=<unused_port>` or set `MASTER_PORT` |
Intended uses
- Use Cicd when this documented workflow matches the task.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.