Skip to main content
Agent skill

Video Generation

Generate videos from text prompts (and optional reference or frame images) using Letta's asynchronous video generation API. Use when the user asks to create, generate, or make a video or animation from a description, animate an existing image, or turn a prompt into a short video clip.

Get started

Get the complete skill folder, including its scripts and reference files.

When to use this skill

Use Video Generation when an AI agent needs a reusable SKILL.md workflow for this job: Generate videos from text prompts (and optional reference or frame images) using Letta's asynchronous video generation API. Use when the user asks to create, generate, or make a video or animation from a description, animate an existing image, or turn a prompt into a short video clip.

When not to use it

Skip Video Generation when the task is outside the creativity category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.

How to install

Where will you use it?

Install scope
Claude Code folder setup
mkdir -p ~/.claude/skills/video-generation
# Save SKILL.md to ~/.claude/skills/video-generation/SKILL.md
  1. Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/video-generation/SKILL.md. For a shared project, commit the folder at .claude/skills/video-generation/ instead.
  2. Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
  3. For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.

Full install guide for Claude, Cursor, and Codex →

Try a starter prompt

After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.

Starter prompt
Use the Video Generation skill for this task: Use Video Generation when this documented workflow matches the task. Confirm any required inputs or files before making changes.

What supports this recommendation?

71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.

Documentation
77/100
Practical detail
68/100
Source evidence
63/100
Source signals
72/100
Model compatibility evidence
  • claudeuntested

    No model-specific signal or recorded compatibility test was found.

  • chatgptinferred

    The skill text mentions ChatGPT or a closely associated term. This is keyword-derived compatibility; no hands-on model test is recorded.

  • geminiuntested

    No model-specific signal or recorded compatibility test was found.

  • copilotuntested

    No model-specific signal or recorded compatibility test was found.

  • llamauntested

    No model-specific signal or recorded compatibility test was found.

  • perplexityuntested

    No model-specific signal or recorded compatibility test was found.

  • mistraluntested

    No model-specific signal or recorded compatibility test was found.

  • grokuntested

    No model-specific signal or recorded compatibility test was found.

Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.

What this skill does

# Video Generation

Generate videos via Letta's async `POST /v1/videos` using `curl` + `jq`. Requires `LETTA_BASE_URL` and `LETTA_API_KEY`, which the runtime provides. If either is unset, stop and ask the user to authenticate with Letta Cloud; do not ask for a video provider key.

This skill is adapted from OpenRouter's `openrouter-video` skill ([OpenRouterTeam/skills](https://github.com/OpenRouterTeam/skills/blob/83827c3f3b1200bc3b609f4ab4ec3241c0cfb7ad/skills/openrouter-video/SKILL.md)). Letta's endpoint forwards the request with Letta's key and charges the organization's Letta credits for the cost of each completed video.

Build every URL relative to `${LETTA_BASE_URL%/}` and send `Authorization: Bearer $LETTA_API_KEY`. Do not hardcode `https://api.letta.com`: Desktop and remote runtimes may provide a proxy base URL, and the credential may only be valid through that URL.

## The three steps

Video generation is async: submit → poll → download. A single request can't return the video because generation takes 30s–a few minutes. Tell the user the job was submitted so they know the delay is expected.

1. `POST /v1/videos` → `{ id, polling_url, status: "pending" }` (`polling_url` is the absolute URL of step 2) 2. `GET /v1/videos/{id}` every ~30s until `status` is `completed` (terminal failures: `failed`, `cancelled`, `expired` — surface the `error` field verbatim) 3. `GET /v1/videos/{id}/content?index=0` **with the auth header** → MP4 bytes

A completed job also has `urls`: signed downloads that need no API key until `expires_at` (Unix seconds); anyone holding one can download the video, so prefer step 3.

## Pick parameters from the models endpoint, don't guess

`resolution`, `aspect_ratio`, `duration`, and `frame_images[].frame_type` are per-model. Before the first submit for a new model (or whenever the user asks for something specific), fetch the model's capabilities and only send values from the returned sets:

```bash curl -sS "${LETTA_BASE_URL%/}/v1/videos/models" \ -H "Authorization: Bearer $LETTA_API_KEY" \ | jq '.data[] | select(.id == "MODEL_ID")' ```

Fields on each model worth knowing: `supported_resolutions`, `supported_aspect_ratios`, `supported_sizes`, `supported_durations` (often discrete like `[4,6,8]`, not a range), `supported_frame_images` (which `frame_type` values are accepted), `generate_audio` and `seed` (capability bools), `pricing_skus`, and `allowed_passthrough_parameters`. An out-of-set value returns a 400, so validate client-side.

## Choosing a model

Recommendation as of 2026-10-29. Video models move fast, so this will go stale; public leaderboards such as [Artificial Analysis](https://artificialanalysis.ai/video/leaderboard/text-to-video) are one rough signal of what's newer, not a verdict. Only pick models that `/v1/videos/models` lists.

- **Default, when quality matters and there's time:** `bytedance/seedance-2.5`. Top tier on the leaderboard, native audio, clips up to 30s. Renders take about 4–6 minutes. - **Fast:** `minimax/hailuo-3-max` (MiniMax H3). Also top tier; renders in well under a minute. Use it for quick replies in a live thread or for iterating on a prompt, then rerun the final cut on Seedance. - Other top-tier models on the endpoint include `alibaba/wan-3.0`, `black-forest-labs/flux-3-video` and `heygen/heygen-video-1`. Some providers filter prompts more strictly than others; if one rejects a prompt, try another model rather than rewording around the filter. - `heygen/avatar-iv` makes a talking head from one photo.

## Writing the prompt

For a single clip, read [references/cinematography.md](references/cinematography.md). For multiple shots or a sequence, read [references/storytelling.md](references/storytelling.md).

For worked examples at the right level of detail, see [references/example-prompts.md](references/example-prompts.md).

A timestamped shot list ending on the payoff helps multi-beat clips.

## Full workflow (drop-in)

```bash #!/usr/bin/env bash set -euo pipefail

PROMPT="A golden retriever bounds across wet sand at golden hour to catch a tennis ball mid-air, spray flying off its fur. Low tracking shot at dog height, 35mm, shallow depth of field, warm backlight and lens flare, waves and the ball's thud in the audio. Ends on the dog trotting back toward camera, ball in mouth." MODEL="bytedance/seedance-2.5" OUTPUT="video-$(date +%Y%m%d-%H%M%S).mp4" base_url="${LETTA_BASE_URL%/}"

# Build payload — extend with resolution/aspect_ratio/duration/etc. as needed. payload=$(jq -n --arg model "$MODEL" --arg prompt "$PROMPT" \ '{model: $model, prompt: $prompt}')

submit=$(curl -sS -X POST "$base_url/v1/videos" \ -H "Authorization: Bearer $LETTA_API_KEY" \ -H "Content-Type: application/json" \ -d "$payload")

job_id=$(echo "$submit" | jq -r '.id // empty') if [ -z "$job_id" ]; then echo "Submit failed: $(echo "$submit" | jq -r '.message // .')" >&2 exit 1 fi echo "Submitted $job_id" >&2

while :; do sleep 30 resp=$(curl -sS "$base_url/v1/videos/$job_id" -H "Authorization: Bearer $LETTA_API_KEY") # Avoid the name `status` — zsh treats it as read-only. st=$(echo "$resp" | jq -r '.status') echo "Status: $st" >&2 case "$st" in completed) break ;; failed|cancelled|expired) echo "Generation $st: $(echo "$resp" | jq -r '.error // "unknown"')" >&2 exit 1 ;; esac done

curl -sS -f "$base_url/v1/videos/$job_id/content?index=0" \ -H "Authorization: Bearer $LETTA_API_KEY" \ --output "$OUTPUT"

echo "$resp" | jq --arg out "$(realpath "$OUTPUT")" \ '{job_id: .id, video_saved: $out, billing}' ```

Share the saved file the way your current surface supports: attach or upload it to the conversation or channel you are replying in. Keep credit amounts out of replies unless the user asks about cost; `billing.credits_charged` on the completed job has them.

## Parameters

Required: `model`. `prompt` is required by every model except those that can generate from image input alone. Common optional fields:

- `duration` (int) — must be one of the model's `supported_durations`. - `resolution` (string) / `aspect_ratio` (string) / `size` (string, `"WxH"`) — `size` is interchangeable with resolution + aspect_ratio. - `generate_audio` (bool) — only meaningful if the model's `generate_audio` capability is true. - `seed` (int) — honored only if the model's `seed` capability is true. - `frame_images[]` — image-to-video; each entry is `{ type: "image_url", image_url: { url }, frame_type: "first_frame" | "last_frame" }`. - `input_references[]` — reference-to-video (style guidance); image entries have the same shape, no `frame_type`. Audio and video references are `{ type: "audio_url", audio_url: { url } }` and `{ type: "video_url", video_url: { url } }`; only some models (such as `bytedance/seedance-2.5`) use them, and the rest ignore them. If both arrays are present, `frame_images` wins. - `provider.options.<slug>.parameters.<key>` — provider passthrough, see below.

Image `url` can be a public `https://` URL or a local-file data URL: `MIME=image/png; B64=$(base64 < file.png | tr -d '\n'); url="data:${MIME};base64,${B64}"`. Audio and video references take URLs the same way.

## Provider passthrough

Provider-specific params go under `provider.options.<slug>.parameters`. The allowed keys for a given model are listed (flat) in `allowed_passthrough_parameters` on the models endpoint — but the meaning, value range, and required combinations come from the *upstream provider's* API docs (Google Vertex, Alibaba Dashscope, Kwai, ByteDance Volc Engine, MiniMax, OpenAI, etc.). Read the upstream docs before using an unfamiliar key; casing conventions differ between providers (Google/OpenAI use camelCase, most others use snake_case).

Example:

```json { "model": "google/veo-3.1", "prompt": "a time-lapse of a flower blooming", "provider": { "options": { "google-vertex": { "parameters": { "personGeneration": "allow", "negativePrompt": "blurry, low quality" } } } } } ```

`heygen/avatar-iv` animates one photo into a lip-synced talking head: pass the photo as an image in `input_references` and describe the video in `prompt`. Its passthrough keys (slug `heygen`) include `voice_id`, `motion_prompt`, and `expressiveness`; take their values from HeyGen's API docs.

```json { "model": "heygen/avatar-iv", "prompt": "A friendly presenter welcomes viewers to the product tour", "aspect_ratio": "9:16", "input_references": [ { "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } } ] } ```

## Chaining with other skills

- **Keyframes:** generate stills with the `image-generation` skill, then pass them as `frame_images` (`first_frame` / `last_frame`) or image `input_references` as data URLs. - **Sound:** make a voiceover with the `speech-generation` skill or a soundtrack with the `music-generation` skill, then pass the audio as an `audio_url` input reference to a model that uses audio, or mix it onto the finished MP4 (for example with `ffmpeg`).

## Notes

- **Billing**: the organization needs credits for an estimate before a job is submitted; once it completes, it is charged for the video's cost. Don't resubmit needlessly. - **Errors**: `402` = insufficient credits (`credits_required` in body); `400`/`404`/`500` return `{ "message": "..." }` — surface it to the user.

Video generation is not ZDR-eligible because the provider must temporarily retain the output for the async download step.

Intended uses

  • Use Video Generation when this documented workflow matches the task.

Implementation notes

  • Review the source instructions and adapt inputs before running the workflow.

Try it, then tell us how it went

A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.