Cicd

When to use this skill

Use Cicd when an AI agent needs a reusable SKILL.md workflow for this job: CI/CD reference for NeMo AutoModel — pipeline structure, commit and PR workflow, CI failure investigation, and common failure patterns.

When not to use it

Skip Cicd when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

1. Personal install: create ~/.claude/skills/cicd/SKILL.md (and any bundled scripts) so Claude Code, Claude Desktop, and compatible agents can load it in every project.
2. Project install: commit the same folder at .claude/skills/cicd/ so teammates get the skill with the repo.
3. Restart the agent session after copying files so it re-scans the skills directory, then ask for the task in words that match the skill description.

Full install guide for Claude, Cursor, and Codex

What this skill does

# CI/CD

## Commit and PR Workflow

### Commits

All commits require DCO sign-off:

```bash git commit -s -m "feat: add new recipe for Qwen2" ```

If sign-off is missing on a recent commit, amend it:

```bash git commit --amend -s ```

### Pull Requests

Follow the PR template in `.github/PULL_REQUEST_TEMPLATE.md`. Every PR should include:

1. **What**: concise description of the change. 2. **Changelog**: bullet list of user-visible changes. 3. **Pre-checks**: confirm linting, tests, and sign-off.

PR title format: `[{areas}] {type}: {description}` (e.g., `[model] feat: Add Qwen3 VLM support`)

See `CONTRIBUTING.md` for the full PR workflow, area/type labels, and DCO requirements.

### Branch Naming

Use descriptive branch names prefixed with your username or a category:

``` username/feat_add_qwen2_recipe fix/gradient_clip_nan ```

## How CI Is Triggered

CI is triggered on `push` — **not** on `pull_request`. A bot called `copy-pr-bot` controls when CI runs.

**Mechanism:** 1. When a PR is opened, `copy-pr-bot` watches for a trust signal. 2. Trust is established in one of two ways: - All commits on the PR branch are **GPG-signed** by a verified NVIDIA contributor → bot triggers automatically. - An NVIDIAN posts `/ok to test <commit-sha>` as a PR comment → bot triggers manually for that SHA. 3. Once trusted, `copy-pr-bot` copies the PR's code into the remote branch `pull-request/<number>` and pushes it, which fires CI.

**Consequences:** - CI never runs on untrusted pushes — external contributors always need `/ok to test`. - The running workflow branch is `pull-request/<number>`, not the author's feature branch. - Pushing a new commit does **not** automatically re-trigger CI unless the commit is signed or `/ok to test <new-sha>` is posted.

## Pipeline Structure

``` lint-check └── cicd-container-build ├── unit-tests-core ├── unit-tests-diffusion └── functional-tests (L0 always; L1 with needs-more-tests label; L2 on schedule) ```

CI test scripts live in `tests/ci_tests/`. These are executed in the CI pipeline and should not be run locally unless reproducing a CI failure.

## CI Failure Investigation

### Locating the PR from a CI Branch

```bash # Extract PR number from branch name (e.g. pull-request/1234) PR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?<=pull-request/)\d+')

gh pr view "$PR_NUMBER" --repo NVIDIA-NeMo/Automodel gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Automodel --name-only gh pr checks "$PR_NUMBER" --repo NVIDIA-NeMo/Automodel ```

### Investigating a Failing Job

1. **Review the changeset**: `gh pr diff "$PR_NUMBER" --repo NVIDIA-NeMo/Automodel` 2. **Identify the failing job** from `gh pr checks` output. 3. **Fetch job logs**: ```bash gh run list --repo NVIDIA-NeMo/Automodel --branch "pull-request/$PR_NUMBER" gh run view <run_id> --repo NVIDIA-NeMo/Automodel --log-failed > run.log ``` 4. **Scan logs in chunks** — log files can be large, never load them whole: ```bash wc -l run.log tail -200 run.log sed -n '1,200p' run.log ``` 5. **Cross-reference the changeset** against the failing step.

## Common Failure Patterns

| Symptom | Likely Cause | Action | |---|---|---| | CI never started | Commits not GPG-signed and no `/ok to test` | Post `/ok to test <full-sha>` on the PR | | Lint job fails | `ruff` violation | Run `ruff check --fix . && ruff format .` locally | | Unit tests fail | Code regression or missing import | Run failing test locally; check the PR diff | | Functional test (L0) fails | Integration breakage | Check GPU runner logs | | DCO sign-off missing | `git commit` run without `-s` | Amend: `git commit --amend -s` | | Multi-GPU tests fail silently | `CUDA_VISIBLE_DEVICES` not set | Set `CUDA_VISIBLE_DEVICES` explicitly | | `torchrun` port conflict | Multiple processes sharing a port | Pass `--master_port=<unused_port>` or set `MASTER_PORT` |

Intended uses

• Use Cicd when this documented workflow matches the task.